Batch jobs taking much longer than normal

Question

In my sandbox environment, I have a batch job that invokes itself when it's done - it watches for records in a "actions" table, and executes them, so when the table is empty, it basically does no work. Normally, this batch job 20-30 times a minute, but in the last few days, it's closer to once a minute, and has a 5-10 minute delay a time or two an hour. Usually, when one execution of the job scheduled another, it'd go to 'Queued' quickly, but in the last few days, it spends a while in 'Holding' first.

Is this delay in processing batch jobs just an indication that the Salesforce platform is overloaded and jobs from my sandbox are de-prioritized? A misconfiguration in my sandbox? Is there a way I can see something from Salesforce to indicate why these jobs are running so much more slowly?

If this is just a temporary-overload issue, I can live with it, but I'm worried that there's a fundamental issue with this approach and I may need to look at other options, as in production, I'd like this job to be firing frequently enough that I can process many actions a minute (I can only process one action per batch, as I need almost all of the DML limit since I have to upsert many tables).

Edit: Further investigation indicated some of our jobs would take 5-10 minutes between CreatedDate and CompletedDate in ApexAsyncJob during the time we saw few jobs executed, even though the processed 0 records.

Answer 1

Executing a batch 20-30 times a minute is a really bad idea; every batch execution consumes a minimum of 3 asynchronous operations (one for start, one for execute and one for finish, with more if the data must be processed in multiple chunks - an extra async operation per additional call to the execute method).

Now if you consider you only have a baseline of 250000 async operations per day (as covered in the governor limits documentation), and then realize you have 1440 minutes in a day, this one batch is consuming up to around 129600 (1440 * 30 * 3) of your async operations for your org. Then consider you may have future methods, other batches and queueables running too, your could easily hit you limits before 24 hours is up.

I am wondering if Salesforce is throttling your org because you are making too many async calls.

Given your description, I wonder if you should be using a Queueable to try to reduce the number of async executions, though you will have to occasionally switch to using a future method to break the chaining since Salesforce limit the "depth" of chaining for queueables. An alternative is to implement something like what we call an "adaptive batch" - take a look at this answer for more details.

UPDATE: As per sfdcfox's handy comments, you can view the current limits using one of these approaches:

• sfdx force:limits:api:display

• workbench REST explorer with /services/data/v48.0/limits/

In both cases look for the daily async apex executions detail.