I've recently run across multiple needs to execute multiple different batch jobs at the same time, and almost immediately hit the (somewhat inexplicable) SFDC limit of 5 batch jobs queued at once.
I've worked around this limit in a number of ways, none of which seem ideal to me.
- returning a
Iterable<SObject>from the start method, which contains a mix of all the SObjects I need to act upon, and then use dynamic inspection to figure out what I'm supposed to be doing on each row in the
executemethod. This seems fine for small jobs, but wouldn't presumably scale to millions of records due to heap size etc.
- using a scheduled apex job that runs once, executes a batch, and then that batch in its
finishmethod schedules another apex job (to run in say 1 minute) that executes the next batch in the chain, and so on.
Is there a more viable/flexible pattern that I could be using? What I really need is a flexible work queue where on-demand, not-very-time-sensitive batch jobs can be queued for future (serial) execution based on user activity. And that doesn't violate platform limits of batch jobs or scheduled jobs - really needs to contribute a max of 1 scheduled or batch job at any given point.
Not sure it will help, but the characteristics of the job set I'm envisioning are this:
- each task executes on a single SObject type and does one job: e.g. setting unset fields, doing fake M-D rollups, etc.
- some tasks are operating on a small subset of a table, others need to walk over every Contact/Lead/etc. with potentially millions of rows per batch job.
- while some tasks are known to be needed every N hours / days, most of them are needed only after some user-driven operation: e.g. click a big red button in an admin console, do something to more than N records, or a new/updated record of a certain type requires its downstream records to be operated upon.