Queueable design patterns to avoid "Too many queueable jobs added"?

Question

When my CustomObject1 is inserted or updated, I need to update a field on the Account object potentially traversing all the CustomObject1 who have a lookup-relationship to given accounts.

In principle, this was done using in-transaction update, but this quickly resulted in a violation of the limits, so the approach I took was to use QueueableApex. When a transaction of CustomObject1 insert or update occurs: - I collect all the Accounts who are impacted - I perform a count query for CustomObject1 instances on accounts.

Then I queue separately the jobs for accounts with many CustomObject1 (one job for account) from the one that have less than 5 CustomObject1 per account (single job). However this approach doesn't work, for example, when there are other jobs scheduled or other transaction on the same objects executed within short time interval.

I imagine a better approach would be to use all the limits in the current job and then when close to the limits queue a new one, but it is not obvious to me how to implement such a pattern.

What are the queueable design patterns available?

Answer 1

The "too many queueable jobs" error is a per transaction error. You can have an "unlimited" number of queueable and future items in the queue of jobs to execute, unlike scheduled and batchable jobs. The only rule is you can only enqueue 50 at once, or 1 when you're already in asynchronous context. You need to change your Queueable design so that you only add as few jobs as necessary to get your business logic done. Also remember that a Queueable job can "chain" itself, so a better design is to use obe job, start executing your logic, and when you reach a certain threshold of governor limits, chain to another job.

public void execute(QueueableContext context) {
  while(freeLimitsAvailable() && !recordsToProcess.isEmpty()) {
    proessMore(recordsToProcess);
  }
  if(!recordsToProcess.isEmpty()) { // Still more work to do... //
    System.enqueueJob(this);
  }
}
Boolean freeLimitsLeft() {
  return Limits.getDmlRows()/(Decimal)Limits.getLimitDmlRows() < 0.8 &&
         Limits.getDmlStatements()/(Decimal)LImits.getLimitDmlStatements() < 0.8 &&
         ...; // check all the limits you care about... //