8

On some user action, I am calling several different third party services in order to update data in salesforce. As these callouts are time consuming, I would like to execute them in parallel, and when they all finish continue processing with data coming from different sources. This all needs to be done by the backend, and when looking at solutions for this in apex, I thought about using future methods and the Queueable interface, but none of those really do the job (there is always some kind of sleep function missing).

This seems as a really common situation but it doesn't seem to be easily done in apex. Does anyone know what would be the best solution for this situation?

Thanks, Nikola

1

4 Answers 4

7

A Queueable chain is probably going to be the best solution pattern for you here, understanding that (as you say) there is not a particularly fluent Salesforce idiom for this kind of processing.

The nice thing about the Queueable chain is that it can essentially parallelize all of the different invocations of this functionality, even though each one individually runs serially. Hence, you won't run into issues with callout limits due to data volume, since you spin off each sequence of callouts into a separate Queueable. You can chain multiple different Queueable classes to get each callout and final processing completed.

Basically, the way the Queueable chain works is that each "step" is one Queueable class, which completes its work and when finished enqueues the next "step", which could be a different class. When you get to the last callout in the sequence, that Queueable can then enqueue your final, post-callout processing. There's no polling or sleep() functionality required - each job just kicks off the next as it finishes, and you don't monitor the chain externally.

Yes, it's far from ideal, but it's probably the best and definitely the most stable option relative to trying to simulate a sleep() call or mucking around with the Tooling API.

If it's most critical that the callouts be run in parallel, you could stick with the Queueable chain, but use a pattern like the one Dan Appleman develops in his Advanced Apex book. Basically, you'd serialize the request to a custom object and kick off N queueable chains, each of which has the job of running one of your N callouts on the data stored in that custom object (however many records there happen to be across your org). Those chains would all run in parallel.

Each Queueable, as it processes each custom object, would write its results back to the object in a different custom field. You could use a workflow rule or Process Builder to kick off the final processing job, contingent on the condition that all N custom fields are populated, meaning that all N jobs have completed.

This is all assuming that you aren't working within a Visualforce page, since you mention doing this on the backend. If you are working in Visualforce, the Continuation object might just suit you.

6
  • Yes, it's in the backend so unfortunately the Continuation is not possible. Concerning the Queueable chain: that is something I've tried, but there doesn't seem to be such a thing as final callout. When a Queueable job is executed, we get the job ID and the only way to know that it's finished is to query the AsyncApexJob to see the status. As there is no such thing as sleep(), we would need to keep querying until the job is finished, which is not a good idea. Please correct me if I'm wrong... Commented Feb 7, 2018 at 9:36
  • With the Queueable chain, you don't have to poll - each step is one Queueable invocation that enqueues the next job. So each callout's job would fire the next callout job, and the last one would enqueue your final processing.
    – David Reed
    Commented Feb 7, 2018 at 12:59
  • Hum yes, but the problem still remains in the last enqueued job: it would need to ensure that all the jobs that were up in the chain finished their execution. That means that we would still need some kind of sleep() to be able to wait for the AsyncApexJob to tell us that all the jobs are finished. Commented Feb 7, 2018 at 13:25
  • Nikola, I added some more detail - in a Queueable chain, each queued job fires the next only upon its own completion. There's no polling or sleep() required, because the jobs run in serial, not parallel.
    – David Reed
    Commented Feb 7, 2018 at 13:49
  • But in that case, what's the point of chaining the jobs? The idea is to do the callouts in parallel and get a shorter global response time... Commented Feb 8, 2018 at 12:57
1

Conceptually this is the use case of Promise.all(), i.e. a promise that is only considered resolved when all the Promises originally passed into it are resolved. Unfortunately the library above does not support such a pattern.

I suppose in theory you could have, say, a Master object tracking groups of parallel callouts and a Detail object tracking the individual ones. The Master would have a roll-up summary for count of callouts expected, and another for count of callouts completed. A trigger on the Master object could monitor for the count completed field to go up to the count expected field, and when it does, fire off some Apex to complete the final steps. I don't know how durable this approach would be though.

1
  • Yes, it could be possible to setup a schema using custom objects that would trigger different processing, but it just seems as an overkill :( Commented Feb 7, 2018 at 9:41
0

This does not seem to be possible to do in any clean way :(. The fact that you cannot do any callouts after a DML query does not help neither. The only two ways seem to be:

  1. Perform a for/while loop that will simulate the sleep function (bad usage of CPU).
  2. Solution described on this link sounds like it would work, but is a dirty workaround :)

It's a bummer....

1
  • 1
    Please don't use #1. It'll come back to bite you - I just finished refactoring some code that did that and was causing highly unpredictable CPU limit exceptions.
    – David Reed
    Commented Feb 6, 2018 at 17:58
-1

You specified that you want the callout to occur from the backend, so direct use of a Continuation from Visualforce isn't directly possible.

Even if you aren't directly in a Visualforce context, you can fake it by having an intermediate Visualforce page that you make a single GET call to. This was demonstrated by Reggie Nair in a Dreamforce 2015 talk. This intermediate VF page essentially orchestrates the parallel callouts and returns a combined result once they all complete.

enter image description here

The intermediate continuation would allow you to Make Multiple Asynchronous Callouts to both SOAP and REST web services.

It is currently limited to three parallel requests, but it will wait to call the callback method until all the responses are returned.

4
  • It's not Visualforce, the processing is done in the backend as it's triggered by a webservice call from the front-office system. The workaround you propose is possible (I think it's the same thing as the link I've sent in my answer), but it just doesn't seem as a good solution, but more as a workaround which is not really viable. The downside is that you would use a callout to trigger another step of the process instead of doing everything internally... Commented Feb 7, 2018 at 9:39
  • Unfortunately the link to the talk doesn't load in any browser I can try it in.
    – Charles T
    Commented Feb 7, 2018 at 12:55
  • Looks like the Salesforce hosted version is broken. I've fixed the link to use the YouTube version of the talk. Commented Feb 7, 2018 at 18:36
  • @NikolaOkiljevic This would give you a single callout to a Visualforce page that would then make up to three parallel callouts and coordinate the response back to you. It is a bit of a work around to use continuations outside of Visualforce. Commented Feb 7, 2018 at 18:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .