Poll for long-running API responses - Solution patterns

Question

Many REST APIs that people want to use from within Salesforce have asynchronous, long-running operation. For example

1. An endpoint POST /api/resource to create a resource returning a handle Id to check
2. Another endpoint GET /api/resources with that Id to get the status/readiness of the creation

As many of such APIs don't send events that could be listened to in Salesforce, polling the second endpoint is all we can do in Salesforce.

The classical options to implement this are:

• Batch jobs that loop until a results is available
• Scheduled Apex to restart a check after a while (often not usable when the result is available after seconds or minutes)
• Queuable Apex
• A combination of all

Are there any solution patterns or even proven Open Source implementations on how to build such a polling in an elegant and scalable way?

Answer 1

Apex-poller is the open source framework that simplifies the polling for long-running processes in Salesforce.

Behind the scenes, it has a polling mechanism with schedulable & queueable classes that gather the best options on polling. However as a consumer you don't need to worry about that, you just need to provide the run/check/callback logic as Callable classes.

It's the alternative to a custom implementation like the mentioned in this post, besides it is highly customisable/flexible (i.e multiple delay options, including incremental delay or presets)

Answer 2

Some options we have come across. Not sure which one is elegant as that depends on individual usecase.

Simulate delay If call can take just few seconds, you can simulate delay in apex and retry the second call.

    public void sleep(Integer milliSeconds) {
        Long startingTime = System.now().getTime();
        while (System.now().getTime() - startingTime < milliSeconds)  {
            Crypto.getRandomLong();
        }
    }

Queue chaining Submit a Queueable job, which checks if results are available. If result is not available, it will kick off itself which will get run after a while. Note that you might run into daily async job limit.

Webhook Notification Create a public webhook (using site and anonymous vf page) which receives request from your external server.

When you send initial request, include this url which your external system must reach back with status updates.

Upon receiving the webhook request, validate the request and kick off another queueable job to process the request.

External system updates sfdc Have external system update sfdc when work is complete. You could then listen to those events via platform events or trigger and do rest of the work.