Run more than one async jobs from Future/Quable context

Question

We have created a process which must work asynchronously (limits and architecture reasons) and it was working fine.

Some time later we have started to work with BigObject. And as long as writing to BigObject is callout, this operation must be done before any DML operation or in async process.

That where we have faced a problem. And its name is - System.LimitException: Too many queueable jobs added to the queue: 2

The problem process is described in diagram:

Step by step description:

1. Some process calls future method
2. update opportunity inside of future method
3. Inside of Opportunity trigger Queueable1 is called (old process)
4. Inside future method Queueable2 is called (save to BigObect)
5. System.LimitException: Too many queueable jobs added to the queue: 2 is thrown...

P.S.: Old and new processes must process successfully.

Is there any best practice, process or workaround to avoid System.LimitException: Too many queueable jobs added to the queue: 2 ?

We was thinking about creating Queueable class (name it like OpportunityTriggerQueueable) where all async work can be done. But in that case not clear how to make checks in trigger. Doubling code also is not a good idea.

Thank you.

Answer 1

I won't try to go in Future/Queueable chains, it can get messy.,

What I would do is to break the chain.

Your requirement of save to BigObect can be done by just firing a platform event. Inside the platform event trigger you can save it to the big objects,

We use platform events and big objects to log things, so I am sure about this approach. What platform event gives you is the flexibility to retry.

Answer 2

To add on to Phil's answer, what I do before ever invoking a future or a queuable is to reference these utility properties:

public static Boolean   isFutureable {
        get {
            if (isFutureable != null)   {return isFutureable;}  // for testing, permits coercion of value
            if (System.isFuture()) {return false;}  // no future permitted from future
            if (System.isBatch()) {return false;}   // no future permitted from batch
            return Limits.getLimitFutureCalls() - Limits.getFutureCalls() > 0;  // adequate headroom
        }
        set;
    }

public static Boolean   isEnqueueable {

    get {
        return isEnqueueable == null ? Limits.getLimitQueueableJobs() - Limits.getQueueableJobs() > 0 : isEnqueueable;}
    set;
}

Why properties and not methods? So testmethods can coerce a value in order to test out various paths

So,

if (Util.isFutureable) {..call @future method..}
else {
  if (Util.isEnqueueable) { .. System.enqueueJob(..); ...}
  else {
     System.schedule(...) // some fallback job  to get us going again 
                          // (although should also check for limits here too)
  }

Answer 3

This limitation specifically applies to future methods:

No more than 0 in batch and future contexts; 1 in queueable context method calls per Apex invocation

The solution is to make the initial invocation use a Queueable rather than a future method and to break chains of queuables/futures by alternating the approach if at all possible.

In terms of checking things twice, that should be OK as long as you have a common piece of code used in both contexts.

Answer 4

Future chaining is something tricky. Specially for more modular processes where you don't have control of the whole chain of execution.

In the past we didn't have an option to chain it (@future or batch methods didn't allow new future calls), so it was simple: make everything synchronous (or make a check: if in future -> call synchrounous / else -> call future).

Today we have more options, so things get a little more complicated.

The simple answer is to rework the chain of events so that you have control over what happens.

Depending on your process that means one of two things:

1. Make the first update on the opportunity Synchronous
2. Bring the old process and the Big Object together into a single Queuable. Do the checks inside the queuable class (remember it's a class, so you can store any kind of method or states inside it. The only System.enqueueJob method does is to call the "execute" method sometime later, but the whole class state is kept).

As mentioned in other answers, there are possible workarounds:

1. Use Platform Events and their triggers. Currently there's no limits on how much Platform Events can be called in a transaction, future or not.
2. Use a Schedulable class, schedule it for 1 second in the future (this breaks the "future chain" and creates a whole new context, but it's not really a good practice.
3. Have you heard something about Unit of Work? It's a concept that you store all DML operations to be performed at the end, no matter how many services needed those. You can apply the same concept.
   1. Create a interface called FutureAction or something like that. It should have a method called execute
   2. Create a Queuable class that has a list of FutureAction.
   3. In the opportunity trigger, you create an Instance of the queueable and for each check, keep adding FutureActions to the list.
   4. In the end, check if there's any FutureActions in the list and then call (or don't) the System.enqueueJob