7

There seems to be an issue with batch jobs failing silently when they exceed the CPU limit, except in certain circumstances. We have confirmed that this isn't just happening on my org but in one or two other orgs as well.

This is concerning because there is no way for us to know that our processes are actually working properly when installed in client orgs with many triggers, workflows, and processes on an object. In addition, any updates that happen prior to the failure do NOT get rolled back.

Here is an example class that illustrates the issue:

public class TestCpuTimeout implements Database.Batchable<sObject> {
    public Database.QueryLocator start(Database.BatchableContext bc) {
        return Database.getQueryLocator('SELECT Id FROM Account LIMIT 200');
    }

    public void execute(Database.BatchableContext bc, List<sObject> scope) {
        scope[0].put('NumberOfEmployees', 100);
        update scope[0];
        System.debug('Account updated: ' + scope[0].Id);

        List<String> largeData = new List<String>();
        for (Integer i = 0; i< 10000; i++) largeData.add('Test');
        for (Integer i = 0; i < 1000; i++) {
            // Not calling Limits.getCpuTime() seems to cause this batch job to silently fail.
            // When there is a silent failure, any record updates do not get rolled back.
            //System.debug(Limits.getCpuTime()); // This causes a CPU timeout.
            //long l = Limits.getCpuTime(); // This causes a CPU timeout.
            //System.debug('test'); // This does not cause a CPU timeout. This silently aborts. Record updates are not rolled back.
            String data = JSON.serialize(largeData);
        }
    }

    public void finish(Database.BatchableContext bc) { }
}

The class simply updates a single account and then serializes a lot of data asynchronously in order to simulate a CPU timeout.

Can anybody else replicate this issue, and/or explain to me what's going on? And yes, this may seem more suited for a case, but Salesforce closed my case (twice - #15549753 & #16942751) because developer support is not offered to non-premier partners.

Test Setup:

Setup debugging for the current user - levels set to Debug for Apex Code and Fine for Apex Profiling.

Test #1:

  • Run Database.executeBatch(new TestCpuTimeout());

After a minute, the Debug Logs and Apex Jobs will show nothing out of the ordinary. It looks like it is successful, but it actually doesn't finish it's entire processing. The account is updated with NumberOfEmployees = 100. (Can be confirmed by grabbing the ID from the debug log.)

Test #2:

  • Update the TestCpuTimeout class and uncomment line 16 - System.debug(Limits.getCpuTime());

  • Run Database.executeBatch(new TestCpuTimeout());

After a minute, the Debug Logs and Apex Jobs will show that there was a CPU timeout exception. The account is NOT updated with NumberOfEmployees = 100. (It may be the same account from Test #1 - confirm that the Last Modified Date does not change.)

Test #3:

  • Update the TestCpuTimeout class and comment out line 16. Uncomment line 17.

  • Run Database.executeBatch(new TestCpuTimeout());

After a minute, the Debug Logs and Apex Jobs will show that there was a CPU timeout exception. The account is NOT updated with NumberOfEmployees = 100. (It may be the same account from Test #1 - confirm that the Last Modified Date does not change.)

Test #4:

  • Update the TestCpuTimeout class and comment out line 17. Uncomment line 18.

  • Run Database.executeBatch(new TestCpuTimeout());

After a minute, the Debug Logs and Apex Jobs will show nothing out of the ordinary. It looks like it is successful, but it actually doesn't finish it's entire processing. The account is updated with NumberOfEmployees = 100.

Tests #1 and #4 do not work as expected. This leads me to believe that for some unknown reason, having the Limits.getCpuTime() statement in the code triggers the CPU timeout exception, but nothing else is. This is extremely concerning and confusing.

This may be related to this Known Issue, but if so, this issue has gone unfixed for over three months.

Update: This issue seems to still reproduce in Summer '17 sandboxes, so it is still not fixed as of 7/13/17.

  • I've seen this error in cs7 instance. Looks like limits were shared across batches while docs says each batch has its own. For me it was heap size. – Ilya Lepesh Jan 21 '17 at 13:10
  • salesforce.stackexchange.com/… - Been going on for a while. Were were told once it was due to recycle bin record (I do not believe it) and we wrote a batch to hard delete them daily. After about 14 days nothing changed – Eric May 15 '17 at 18:24
  • @Eric Yeah, I first reproduced this about 5 months ago. The link you posted seems to refer to an issue with the start method, though? The issue I'm experiencing occurs in the execute of the batch job. – Robert Watson May 15 '17 at 19:35
5

Yes, at least two other people have asked about it here on SFSE:

It seems related to this Known Issue:

Batch Apex job finishes unexpectedly in Spring'17 without processing all specified records

In Spring'17 apex batch class (Database.Batchable) doesn't process all the batches if batch execute method takes long time to run but within limits.

It stops processing batches after some number of processed records and finishes unexpectedly. No error is reported, no exception is thrown

It looks like the issue is fixed in SPRING '17 PATCH 5.2, and a number of sandbox instances (and a few production instances) have the issue marked as fixed. I can't confirm if there is any connection to the Known Issue you linked to.

  • Thanks - I actually saw this earlier in the week but thought maybe this was a different issue. It might be worth noting that my org is on Winter '17, not Spring '17. I assumed the issue was different as it says "doesn't process all the batches if batch execute method takes long time to run but within limits." I'm clearly not within the limits here - the loop maybe executes ~350 out of the 500 times before it reaches the 60 second threshold. – Robert Watson Jan 21 '17 at 1:16
  • Hmm, it will be interesting to see if the issue can be reproduced in an updated org. I'll take a look at repro. – Adrian Larson Jan 21 '17 at 1:18
  • I'll test with a smaller batch scope and see if the other batches are processed at all or ignored if the first batch crosses the CPU threshold. – Robert Watson Jan 21 '17 at 1:19
  • FYI I tested this last week with a smaller batch size - all batches were processed, it just appeared as though everything was successful when it wasn't. – Robert Watson Jan 25 '17 at 17:59
  • 1
    I've been meaning to circle back on this and finally did - the known issue you linked has been rolled out to all pods, but I'm still able to reproduce this issue. The known issue I linked to originally is still not fixed, so perhaps this is related to that one. – Robert Watson Mar 4 '17 at 2:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.