Parallel callouts in apex

Question

On some user action, I am calling several different third party services in order to update data in salesforce. As these callouts are time consuming, I would like to execute them in parallel, and when they all finish continue processing with data coming from different sources. This all needs to be done by the backend, and when looking at solutions for this in apex, I thought about using future methods and the Queueable interface, but none of those really do the job (there is always some kind of sleep function missing).

This seems as a really common situation but it doesn't seem to be easily done in apex. Does anyone know what would be the best solution for this situation?

Thanks, Nikola

Answer 1

A Queueable chain is probably going to be the best solution pattern for you here, understanding that (as you say) there is not a particularly fluent Salesforce idiom for this kind of processing.

The nice thing about the Queueable chain is that it can essentially parallelize all of the different invocations of this functionality, even though each one individually runs serially. Hence, you won't run into issues with callout limits due to data volume, since you spin off each sequence of callouts into a separate Queueable. You can chain multiple different Queueable classes to get each callout and final processing completed.

Basically, the way the Queueable chain works is that each "step" is one Queueable class, which completes its work and when finished enqueues the next "step", which could be a different class. When you get to the last callout in the sequence, that Queueable can then enqueue your final, post-callout processing. There's no polling or sleep() functionality required - each job just kicks off the next as it finishes, and you don't monitor the chain externally.

Yes, it's far from ideal, but it's probably the best and definitely the most stable option relative to trying to simulate a sleep() call or mucking around with the Tooling API.

If it's most critical that the callouts be run in parallel, you could stick with the Queueable chain, but use a pattern like the one Dan Appleman develops in his Advanced Apex book. Basically, you'd serialize the request to a custom object and kick off N queueable chains, each of which has the job of running one of your N callouts on the data stored in that custom object (however many records there happen to be across your org). Those chains would all run in parallel.

Each Queueable, as it processes each custom object, would write its results back to the object in a different custom field. You could use a workflow rule or Process Builder to kick off the final processing job, contingent on the condition that all N custom fields are populated, meaning that all N jobs have completed.

This is all assuming that you aren't working within a Visualforce page, since you mention doing this on the backend. If you are working in Visualforce, the Continuation object might just suit you.

Answer 2

Conceptually this is the use case of Promise.all(), i.e. a promise that is only considered resolved when all the Promises originally passed into it are resolved. Unfortunately the library above does not support such a pattern.

I suppose in theory you could have, say, a Master object tracking groups of parallel callouts and a Detail object tracking the individual ones. The Master would have a roll-up summary for count of callouts expected, and another for count of callouts completed. A trigger on the Master object could monitor for the count completed field to go up to the count expected field, and when it does, fire off some Apex to complete the final steps. I don't know how durable this approach would be though.

Answer 3

This does not seem to be possible to do in any clean way :(. The fact that you cannot do any callouts after a DML query does not help neither. The only two ways seem to be:

1. Perform a for/while loop that will simulate the sleep function (bad usage of CPU).
2. Solution described on this link sounds like it would work, but is a dirty workaround :)

It's a bummer....

Answer 4

You specified that you want the callout to occur from the backend, so direct use of a Continuation from Visualforce isn't directly possible.

Even if you aren't directly in a Visualforce context, you can fake it by having an intermediate Visualforce page that you make a single GET call to. This was demonstrated by Reggie Nair in a Dreamforce 2015 talk. This intermediate VF page essentially orchestrates the parallel callouts and returns a combined result once they all complete.

The intermediate continuation would allow you to Make Multiple Asynchronous Callouts to both SOAP and REST web services.

It is currently limited to three parallel requests, but it will wait to call the callback method until all the responses are returned.