7

I have a batch job which processes millions of records through an HTTP callout to a third-party API. The API doesn’t handle batching, so each record needs its own callout. It takes well over 24 hours to process these records serially in the batch execute method, and it’s a nightly job that needs to finish before the next run starts. I have achieved this by moving the processing into a queueable spawned from the execute method. The batchable creates thousands of queueables, which are executed in parallel (up to about 50 simultaneously).

The problem is that this approach overwhelms the third-party API with more requests than it can handle. A single run of the job may result in tens or even hundreds of thousands of exceptions—mostly timeout exceptions on the Salesforce side, and all sorts of different exception responses from the API. The API is totally outside of my control and increasing throughput on their end does not seem to be an option.

I have mostly solved this problem by introducing a small delay between callouts—essentially just burning CPU time. With this approach, I have eliminated the exceptions almost entirely and all records finish processing in about 2 hours—even faster than the 4-5 hours it took without the delay, due to the exception handling. I guess it could be argued that it's more efficient with the delay than without because it avoids the heavy load from all the exception handling, but I'd really rather not have something like this in production if there's a better alternative.

How can I do this more efficiently? I can think of two possibilities:

  1. Restrict the number of instances of this queueable that can be processed in parallel. For instance, a cap of 25 simultaneous executions should be more than enough to complete the job in the allotted time without overwhelming the API.
  2. Slow down execution through some other janky workaround that is less wasteful than what I’ve got currently.

Is either of these possible? Other suggestions are also welcomed.

My current implementation looks roughly like this:

Batchable:

public class SomeBatchable implements Database.Batchable<sObject> {

    public Database.QueryLocator start(Database.BatchableContext bc){ 
        return Database.getQueryLocator('SELECT Id FROM Account'); 
    }

    public void execute(Database.BatchableContext bc, List<Account> scope){
        System.enqueueJob(new SomeQueueable(new List<Id>(new Map<Id,Account>(scope).keySet())));
    }    

    public void finish(Database.BatchableContext bc){}
}

Queueable:

public class SomeQueueable implements Queueable, Database.AllowsCallouts {

    private List<Id> recordIds {get; set;}

    public SomeQueueable(List<Id> recordIds){
        this.recordIds = recordIds;
    }

    public void execute(QueueableContext context){

        Map<Id,Account> records = new Map<Id,Account>([SELECT Id, Name FROM Account WHERE Id IN :recordIds]);

        Integer calloutLimit = Limits.getLimitCallouts();  
        Decimal cpuLimit = Limits.getLimitCpuTime() * 0.8;
        while(recordIds.size() > 0 && Limits.getCallouts() < calloutLimit && Limits.getCpuTime() < cpuLimit){            
            wait(50);
            process(records.get(recordIds.remove(0)));
        }

        if(recordIds.size() > 0){            
            System.enqueueJob(new SomeQueueable(recordIds));
        }

    }        

    // may god and /r/badcode have mercy on my soul
    private void wait(Integer milliseconds){
        Long start = DateTime.now().getTime();
        while(start + milliseconds > DateTime.now().getTime()){}
    }

    private void process(Account a){
        // http callout
    }

}
  • Do you need to do something with the response from the callout (beyond worked/didn't work handling)? – Keith C Aug 26 '17 at 9:59
  • Maybe you should route your requests through a service like CloudFlare. – Adrian Larson Aug 26 '17 at 12:54
  • 1
    How about creating an object to track your requests. Execute actual queueable class via trigger on that object. Upon execution, query additional 25 unfinished requests and update their status to in-progress. Continue processing / chain requests until finished. – dzh Aug 26 '17 at 18:37
  • I like @dzh's idea. Make the throttling of the API callouts explicit and controllable. You might not need a dedicated queue object and could instead use a dedicated sync field on the Account Object. Say a DateTime last synced and another field for current sync status. Scheduled job could just more the target sync data forward and then start chained job to process all the records in small batches. – Daniel Ballinger Aug 27 '17 at 20:34
1

"Restrict the number of instances of this queueable that can be processed in parallel. For instance, a cap of 25 simultaneous executions should be more than enough to complete the job in the allotted time without overwhelming the API."

Ideally if you could separate the records into 25 buckets this would help solve the problem. So let's say you have a Batchable implementing Database.Stateful which runs through the entire list of records and collects 25 List<Id> buckets (assuming the ID list itself would not bust the stack limit).

Now in the batch's finish() method you launch 25 instances of a Queueable, passing the List<Id> into it. Each Queueable execution pops one ID off the list at a time, makes the callout, waits for the response, and then continues to the next record until it's done. In principle this could also be 25 Batchable instances, each churning through its list, although I'm not sure if that's worse in terms of platform limits. In either case, you are guaranteed no more than 25 simultaneous executions.

  • Couple issues here: the governor allows only 1 queueable to be launched from finish(), and the heap limit is too low to store all 25 buckets simultaneously anyway. But I could do something similar, where each execute() adds the batch to a single bucket, and then empties the bucket and sends it to a queueable if it’s over a certain size. Downside is that each queueable is responsible for over 100k records, so even a single unhandled exception could result in tens of thousands of records not being processed. Great idea overall though. I’ll try this out when I get a chance. – Sequoyah Oct 3 '17 at 22:11
  • 1
    Interesting, I had not considered the possibility that passing just the IDs could still hit the heap limit. I like your idea though for generating one bucket at a time and then passing it to be emptied. I suppose you'd want to use some sort of logging to catch any exceptions in the chained Queueable so that it protects the ability to pass control to the next one. Maybe also some kind of failsafe where if you flip off a certain custom setting it stops chaining the Queueable and shuts down. – Charles T Oct 4 '17 at 1:22
0

Which is the chunk size that you are using for the batch? I suggest to put a small number of the chunk. You can make some tests and see which chunk size would be more suitable.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.