1

I have a batch job that processes quite large amounts of data, in edge cases it can take up to 40 minutes to execute. I would like to have it run as often as possible. I have no way of estimating for sure what would be the maximum time for the batch job execution, with more users in the future it can take more time. I have looked through the documentation but I cannot find the answer for a simple question - what will happen if the frequency of the batch executions scheduled is consistently not enough for the previously scheduled job to finish? Is that a path to the queue being filled and eventually overflowing? I don't care about every single of the scheduled batch jobs being executed, because they're all supposed to react to real time changes in the data. It's the period between the executions that matters. I also want to have it running reliably, so it should keep getting executed even if one of executions fails.

Improve this question
3
  • 3
    Jobs will pile up they will not wait each other since there is no batch job queue. You should consider adding a re-schedule logic to finish method of you batch class so that whenever last chunk of your batch job gets completed it can schedule the job to whenever you want it to run.
    – Mert YALTI
    Commented Jun 5, 2020 at 15:08
  • 2
    This article talks about the behavior batch run overlaps with another batch run of same Apex class : foobarforce.wordpress.com/2015/10/25/batch-apex-query-behaviour
    – javanoob
    Commented Jun 5, 2020 at 16:28
  • 1
    An interesting point to note: you cannot use FOR UPDATE in a query locator to try to lock the records. You can, of course, have execute re-query the records to apply a lock for the duration of the execute execution, but that's not ideal. Writing some infrastructure, like I outlined in my answer, that guarantees single threaded processing nicely avoids race conditions, at least between instances of the batch (if not from other, external actors).
    – Phil W
    Commented Jun 5, 2020 at 16:50

1 Answer 1

Reset to default
4

It is not possible to schedule two batches with the same name via System.scheduleBatch concurrently. However, a given batch can be queued as many times as you like (based on flex queue size limits, of course) and have up to 5 instances executing "concurrently".

Additionally, that same batch can be scheduled even when there is another instance executing (since once the scheduled time for System.scheduleBatch is reached the schedule entry disappears, releasing the name, and the batch is executed).

NB: If you try to queue too many batchables, and the flex queue is filled, an exception will be thrown. You cannot queue any more until space is made in the queue.

You can build mechanisms on top of System.scheduleBatch and Database.Batchable that:

  • Allow multiple instances of the batch to be executed as long as they are looking at different "islands" of data from the total data set
  • Ensure that only one batch instance for a given batch is ever scheduled, queued or executing and have this automatically re-schedule itself if (when finish is reached) there is now more data to process.

The former we called "data domain processing" and this relies on maintaining details of the "data domain" in a way we can query and detect clashes.

The latter we called "adaptive batch" and this relies on:

  • The fact that only a single scheduled batch can be created with a given name
  • The fact that only a single Batchable.start is ever processed on your org at any one time and that the Async Apex Jobs details can be checked in a "thread safe" way because of this, to detect another instance of the batch that is already executing
  • Structuring the SOQL query locator generation in a way that allows use of an equivalent COUNT query in finish before re-scheduling the batch.
  • Having the application code request processing when it knows inserted/updated data meets the batch's query criteria. Such requests are ignored if the batch is already scheduled or queued/running since the batch will re-schedule itself on completion, if required.

Note that we always ensure there is at least a 1 minute gap between one execution of the "adaptive batch" and the other, to ensure that in error cases we don't accidentally consume all daily async executions in such cases.

You can read a bit more on this latter solution here but fundamentally you have to implement it yourself - Salesforce gives you the building blocks but no direct solution here.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .