How to work around the 100 callout limitation?

Question

We need to have the ability to make more than 2999 callouts to a REST service. Currently, we are getting a raised exception when the number of callouts is more than current limit for callouts.

Thus, is there a solution for doing async REST callouts within Apex class to support the large amount required to support our requirement.

Answer 1

Create a batchable class, and do as many callouts as you need. The 100 callout limit applies only to a single transaction, but Batchable classes can create millions of transactions. There's plenty of examples on the forum here already about how batchable works, but for sake of argument, say you wanted a quick-and-dirty way to get started, you could do this:

global class Do2999Callouts implements Database.Batchable<Integer> {
    global Iterable<Integer> start(Database.BatchableContext context) {
        Integer[] values = new Integer[0];
        while(values.size() < 2999) values.add(values.size());
        return values;
    }
    global void execute(Database.BatchableContext context, Integer[] values) {
        // Do one callout for each integer in values
    }
    global void finish(Database.BatchableContext context) {
    }
}

You would call this code using:

Database.executeBatch(new Do2999Callouts(), 100);

Of course, this is only an arbitrary example. You could use smaller batch sizes, base them off of records, or do whatever else you need to. There's plenty of ways of doing this, you just need to split your transactions up into multiple parts, as demonstrated here.

Answer 2

2999 is a lot of callouts to be making in one transaction. Is the user going to sit there and wait for them to finish?

Brian's suggestion to use Batch Apex is a common way to deal will bulk callouts. It's definitely a good way to scale up the number of callouts that can be made. Being asynchronous is nature, the user won't have much feedback on how the batch is progressing unless you monitor the batch job status.

I see the current tags for the question include Visualforce. One option here is to use a Continuation with chaining or multiple callouts. The problem here will be the Asynchronous Callout Limits. As at Spring '16, you are limited to a maximum of 3 parallel Apex callouts in a single continuation and 3 chained callouts. Those limits will prevent you reaching the 2999 callouts required. Update May 2017. Jitendra Zaa published Limitless Chaining of Continuation object in Salesforce. It combines JavaScript remoting and Continuation objects to work around the limit.

You could also consider Queueable with Database.AllowsCallouts. This would require chaining the jobs together or using a staging custom object to queue the required calls. See Queueable Apex: More Than an @future.

Whatever approach you take, consider using Limits.getCallouts() and Limits.getLimitCallouts() to monitor the usage within a transaction.