I have a batch job that processes quite large amounts of data, in edge cases it can take up to 40 minutes to execute. I would like to have it run as often as possible. I have no way of estimating for sure what would be the maximum time for the batch job execution, with more users in the future it can take more time. I have looked through the documentation but I cannot find the answer for a simple question - what will happen if the frequency of the batch executions scheduled is consistently not enough for the previously scheduled job to finish? Is that a path to the queue being filled and eventually overflowing? I don't care about every single of the scheduled batch jobs being executed, because they're all supposed to react to real time changes in the data. It's the period between the executions that matters. I also want to have it running reliably, so it should keep getting executed even if one of executions fails.
It is not possible to schedule two batches with the same name via System.scheduleBatch concurrently. However, a given batch can be queued as many times as you like (based on flex queue size limits, of course) and have up to 5 instances executing "concurrently".
Additionally, that same batch can be scheduled even when there is another instance executing (since once the scheduled time for System.scheduleBatch is reached the schedule entry disappears, releasing the name, and the batch is executed).
NB: If you try to queue too many batchables, and the flex queue is filled, an exception will be thrown. You cannot queue any more until space is made in the queue.
You can build mechanisms on top of System.scheduleBatch and Database.Batchable that:
- Allow multiple instances of the batch to be executed as long as they are looking at different "islands" of data from the total data set
- Ensure that only one batch instance for a given batch is ever scheduled, queued or executing and have this automatically re-schedule itself if (when finish is reached) there is now more data to process.
The former we called "data domain processing" and this relies on maintaining details of the "data domain" in a way we can query and detect clashes.
The latter we called "adaptive batch" and this relies on:
- The fact that only a single scheduled batch can be created with a given name
- The fact that only a single Batchable.start is ever processed on your org at any one time and that the Async Apex Jobs details can be checked in a "thread safe" way because of this, to detect another instance of the batch that is already executing
- Structuring the SOQL query locator generation in a way that allows use of an equivalent COUNT query in finish before re-scheduling the batch.
- Having the application code request processing when it knows inserted/updated data meets the batch's query criteria. Such requests are ignored if the batch is already scheduled or queued/running since the batch will re-schedule itself on completion, if required.
Note that we always ensure there is at least a 1 minute gap between one execution of the "adaptive batch" and the other, to ensure that in error cases we don't accidentally consume all daily async executions in such cases.
You can read a bit more on this latter solution here but fundamentally you have to implement it yourself - Salesforce gives you the building blocks but no direct solution here.