29

What are the workarounds for the 10 web service callout limit in apex. If you need to make, say 50 callouts, how do you batch them so that you can execute them all asynchronously? Is there a good design pattern for it?

I'm frequently in this situation and there are different solutions for it that I'm aware of, but I thought it would be good to get this documented on the stackexchange, because it can be tricky and the documentation elsewhere is not great.

Note: The limit was raised to 100 callouts in the Winter '15 release, so this question is somewhat outdated now: http://docs.releasenotes.salesforce.com/en-us/winter15/release-notes/rn_apex_limits_callout_future.htm

  • 3
    If you're gonna try to do them all asynchronously, there is also the concurrent call out to the same url limit. – Samuel De Rycke Nov 30 '12 at 22:04
34

(community wiki)

@future (valid for triggers and visualforce)

From trigger or VF context you can have up to 10 calls to @future & each of them has a separate context that allows 10 callouts. So you can get up to 100 callouts from a trigger (that itself you're guaranteed will not contain more than 200 records if it's saved with API > 20.0, capped at 100 otherwise) or 10+100 in VF. Not bad. Before you decide to abuse it - consider the fact that there's a 24 hour rolling limit on @future invocations and that multiple callouts to same endpoint might also throw an error. From governor limits page:

A callout request is limited to a maximum of 20 simultaneous requests to URLs with the same host. The host is defined by the unique subdomain for the URL, for example, www.mysite.com and extra.mysite.com are two different hosts. This limit is calculated across all organizations that access the same host. If this limit is exceeded, a CalloutException will be thrown.

Batch Jobs

Batch jobs let you process many records (up to 50 M) with whatever granularity you wish (if you don't specify it, granularity will be 200). Essentially you decide which records you want to work on, you get a fresh context & governor limits for each chunk of data you're processing and then at the end of the batch job you can send an email, kick-off another batch etc. They're great for background processing tasks.

In batch jobs you can specify an optional scope parameter. If you really need this, the batch job may even be instructed to process 1 record at a time (and again - each execute() will let you make 10 callouts). That is - if you'll make sure that the batch class implements Database.AllowsCallouts ;) Read more about Database.executeBatch if you want to take this path.

Essentially:

public class MyBatchableClass implements Database.Batchable<sObject>, Database.AllowsCallouts{
// definitions of start(), execute() and finish() as in any batch
}

// and then in code that fires it (scheduled class? something that happens 
// after user clicks a button? you use 
Database.executeBatch(new MyBatchableClass(), 1);

// instead of 
// Database.executeBatch(new MyBatchableClass());

Javascript-related solutions

You can jump back and forth in the context. Build a Visualforce page that would process N records at a time, return to the browser, issue a next call that processes another N...

You know your data best - N can be a fixed number or maybe you'll just want to compare output of Limits methods:

if(Limits.getCallouts() == Limits.getLimitCallouts()){
    return 'I\'m not done yet';
} 

This is best suited in environment where user expects some kind of progress report like "X out of Y records processed". If you're after a fully automated background solution that can't be interrupted by user closing the tab for example - go for one of previous two.

"The call" can be Javascript that hits Apex code in form of webservice call (in ajax toolkit - for example a button on list view that processes selected records), @RemoteAction , actionFunction etc.

  • You would need to specify more detail for this to be a complete answer. You can easily run into the 10 callout limit using option (1). Option (2) is not explained in enough detail for me to use it. Option (3) seems like a hack that could result in a strange user experience, not sure if this should be considered a best practice. – paul Dec 3 '12 at 21:33
  • Hi @paul - I confess I don't see how I can run into 10 callouts limit with #1 - you mean if trigger has > 100 records probably? Yeah, use batch jobs then, almost always a better answer than future. #2 - have you ever written an apex batch job? I've expanded it a bit. I'd say a batch job + email / task notification at the end is best option of all these, #1 is hacky and #3 can be too easily interrupted by user navigating away. You can achieve same "progress bar" feeling by querying batch status in AsyncApexJob table. – eyescream Dec 3 '12 at 22:21
  • 1
    Thanks for updating your answer - this is very good and what I was looking for. I have done this stuff before, I just thought it would be good to have it documented here for others, by someone who has done it a lot, because it comes up a lot and in my opinion the documentation elsewhere is poor. – paul Dec 4 '12 at 4:08
  • Can someone provide a more in-depth example of the solution #2 please? I have the exact problem right now, thinking about the limits on the Web service. I don't have control on it and cannot make a batch request. I was thinking about (maybe) creating a middleware to achieve this (if there're no other solution). – jpmonette Mar 26 '13 at 14:57
  • 1
    @eyescream I think I got it. I was missing the Database.AllowsCallouts part. Seems to be working now! Thanks for taking some time to come back her :)! – jpmonette Mar 27 '13 at 15:16
10

The existing answers are all pretty good. I've had to work with all of these situations before. Ultimately if you need more than even those workarounds will allow you'll have to look at redoing the web service so you don't have to make as many callouts. Options for that? How about:

If you have control over the webservice you could alter it to be able to take a batch set of data and process them all individually on the server.

If you don't have control over the webservice you could look at writing an intermediary service that took the batch of data, and then individually called the real webservice individually.

That last one in particular would be a last resort sort of attempt as it's really messy but sometimes that's what it takes to get SFDC to work the way you need it to.

  • Actually, I could see doing it with an intermediary service as much, much less messy. The other solutions, which are "correct" if you're doing it with Apex, are inherently messy, because Salesforce has designed it very poorly. With an intermediary service you could send out a single clean JSON string and have an intermediate server do the callouts in a clean way. I'm strongly considering this option now, as I have been fighting with some messy triggers + callouts + scheduled apex. – paul Dec 3 '12 at 21:30
  • The messy part can come in to play when you have to handle responses and such. It's not so much messy as it can be a bit more work, but it should get it done. – Ryan Elkins Dec 5 '12 at 21:16
5

Setup an APEX Batch Job: http://www.salesforce.com/us/developer/docs/apexcode/Content/apex_batch_interface.htm

Than use an object iterator to keep track of how many total callouts you have to make through a simple association.

If you have 50 callouts to make, than you might need 5 records, each of which can represent a total of 10 possible callouts.

Hopefully whatever service you need to call out to get results from allows you to index results so you don't always have to start at the begging.

There will be some delay because Apex Batch jobs run when system resources are available, and there can be a delay.

Another way might be to use a page reference that points back to the same page, and increments an iterator until no more results are needed:

    public PageReference continue(){
        PageReference pageRef = ApexPages.currentPage();
        sObject result = [SELECT IndexId__c FROM sObject WHERE Id ...];
        if(result < 1000){
           methodThatMakesCalloutsAndIncrementIndex();
          pageRef.setRedirect(true);
       }
       else{
          pageRef = Page.FinishedPage;
       }
    }
2

Current limits have been increased to 100 if anyone needs it. Check out total number of callouts allowed here - https://www.salesforce.com/us/developer/docs/apexcode/Content/apex_gov_limits.htm

1

http://sforcetips.blogspot.fi/2013/06/saleforce-http-callouts-limit-workaround.html

The key is that the limit is "per execution", and an AJAX call (done through a standard Visualforce ) is actually a single execution. I can "loop" in Javascript to do requests as long as I need it.

Let's have a look at the code.

The controller:

public with sharing class testJavascript {
 public String requestStatus { get; set; }
 public String requestMore { get; set; }
 public Integer requestNumber { get; set; }
 public Integer requestCurrent { get; set; }
 public testJavascript () {
  requestStatus = '';
  requestMore = '';
  requestNumber = 10000;
  requestCurrent = 0;  
 }

 public void doRequest () {
  if (requestCurrent <= requestNumber) {
   Http http = new Http();
   for (Integer startPoint = requestCurrent; (requestCurrent <= requestNumber) && ((requestCurrent - startPoint) < 10); requestCurrent++) {
    HttpRequest req = new HttpRequest();
    req.setEndpoint('http://www.google.com');
        req.setMethod('GET');
    HTTPResponse res = http.send(req);
   }
   requestStatus = requestCurrent + ' out of ' + requestNumber + ' processed';
  }
  if (requestCurrent >= requestNumber)
   requestMore = 'false';
  else requestMore = 'true';
 }
}

The Page:

<apex:page controller="testJavascript">
<apex:form>
 <apex:outputpanel id="status">
  Status:

  {!requestStatus}
  <script>requestContinue = {!requestMore};</script>
 </apex:outputpanel>
 <script>
  function requestMore() {
   if (requestContinue)
    doRequest();
  } 
 </script>
 <button onclick="if (doRequest()) return false; else return false;">Start</button>
 <apex:actionfunction action="{!doRequest}" name="doRequest" oncomplete="requestMore()" rerender="status">
</apex:actionfunction>
</apex:form>
</apex:page>

Some explication:

In the controller I have a request counter (requestCurrent, set to 0 at beginning) and a request amount (requestNumber, set to 10000 at beginning) and a method, doRequest(), that, given those two variables, executes the next 10 HTTP requests (increasing the counter) and updates the status String (requestStatus) and the requestMore String/Boolean, setting it to true if other requests are needed (if the counter is lower than the request amount we need) or false if not. In the page I have an apex:actionFunction pointing to this method, that rerenders the status panel. In the panel I show the status string and I update a Javascript global variable (requestContinue) with the String/Boolean given in requestMore by the doRequest() APEX method. The actionFunction has an oncomplete attribute that calls another Javascript function, requestMore(), that is in the page code. This function checks for requestContinue value (updated after each rerender): if true calls again the doRequest() actionFunction and 10 more requests are executed, if false it does nothing. The user has just to push the button that does the first actionFunction call to start the loop.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.