Has anybody seen this behavior? Batch jobs failing silently on CPU timeout except in certain circumstances

Question

There seems to be an issue with batch jobs failing silently when they exceed the CPU limit, except in certain circumstances. We have confirmed that this isn't just happening on my org but in one or two other orgs as well.

This is concerning because there is no way for us to know that our processes are actually working properly when installed in client orgs with many triggers, workflows, and processes on an object. In addition, any updates that happen prior to the failure do NOT get rolled back.

Here is an example class that illustrates the issue:

public class TestCpuTimeout implements Database.Batchable<sObject> {
    public Database.QueryLocator start(Database.BatchableContext bc) {
        return Database.getQueryLocator('SELECT Id FROM Account LIMIT 200');
    }

    public void execute(Database.BatchableContext bc, List<sObject> scope) {
        scope[0].put('NumberOfEmployees', 100);
        update scope[0];
        System.debug('Account updated: ' + scope[0].Id);

        List<String> largeData = new List<String>();
        for (Integer i = 0; i< 10000; i++) largeData.add('Test');
        for (Integer i = 0; i < 1000; i++) {
            // Not calling Limits.getCpuTime() seems to cause this batch job to silently fail.
            // When there is a silent failure, any record updates do not get rolled back.
            //System.debug(Limits.getCpuTime()); // This causes a CPU timeout.
            //long l = Limits.getCpuTime(); // This causes a CPU timeout.
            //System.debug('test'); // This does not cause a CPU timeout. This silently aborts. Record updates are not rolled back.
            String data = JSON.serialize(largeData);
        }
    }

    public void finish(Database.BatchableContext bc) { }
}

The class simply updates a single account and then serializes a lot of data asynchronously in order to simulate a CPU timeout.

Can anybody else replicate this issue, and/or explain to me what's going on? And yes, this may seem more suited for a case, but Salesforce closed my case (twice - #15549753 & #16942751) because developer support is not offered to non-premier partners.

Test Setup:

Setup debugging for the current user - levels set to Debug for Apex Code and Fine for Apex Profiling.

Test #1:

• Run Database.executeBatch(new TestCpuTimeout());

After a minute, the Debug Logs and Apex Jobs will show nothing out of the ordinary. It looks like it is successful, but it actually doesn't finish it's entire processing. The account is updated with NumberOfEmployees = 100. (Can be confirmed by grabbing the ID from the debug log.)

Test #2:

• Update the TestCpuTimeout class and uncomment line 16 - System.debug(Limits.getCpuTime());

• Run Database.executeBatch(new TestCpuTimeout());

After a minute, the Debug Logs and Apex Jobs will show that there was a CPU timeout exception. The account is NOT updated with NumberOfEmployees = 100. (It may be the same account from Test #1 - confirm that the Last Modified Date does not change.)

Test #3:

• Update the TestCpuTimeout class and comment out line 16. Uncomment line 17.

• Run Database.executeBatch(new TestCpuTimeout());

After a minute, the Debug Logs and Apex Jobs will show that there was a CPU timeout exception. The account is NOT updated with NumberOfEmployees = 100. (It may be the same account from Test #1 - confirm that the Last Modified Date does not change.)

Test #4:

• Update the TestCpuTimeout class and comment out line 17. Uncomment line 18.

• Run Database.executeBatch(new TestCpuTimeout());

After a minute, the Debug Logs and Apex Jobs will show nothing out of the ordinary. It looks like it is successful, but it actually doesn't finish it's entire processing. The account is updated with NumberOfEmployees = 100.

Tests #1 and #4 do not work as expected. This leads me to believe that for some unknown reason, having the Limits.getCpuTime() statement in the code triggers the CPU timeout exception, but nothing else is. This is extremely concerning and confusing.

This may be related to this Known Issue, but if so, this issue has gone unfixed for over three months.

Update: This issue seems to still reproduce in Summer '17 sandboxes, so it is still not fixed as of 7/13/17.

Answer 1

Yes, at least two other people have asked about it here on SFSE:

• batch class doesn't execute all the batches
• Total number of Batch jobs decreases during execution. Why that?

It seems related to this Known Issue:

Batch Apex job finishes unexpectedly in Spring'17 without processing all specified records

In Spring'17 apex batch class (Database.Batchable) doesn't process all the batches if batch execute method takes long time to run but within limits.

It stops processing batches after some number of processed records and finishes unexpectedly. No error is reported, no exception is thrown

It looks like the issue is fixed in SPRING '17 PATCH 5.2, and a number of sandbox instances (and a few production instances) have the issue marked as fixed. I can't confirm if there is any connection to the Known Issue you linked to.