9

I have a batch job that needs to callout to an external web service and retrieve a variable number of records. Seems like a perfect use-case to have an iterator which performs the callouts until there are no more results.

However, if the webservice returns many records (in separate paginated callouts) the batch job will hit the maximum callout timeout limit while it is being prepared. This leads me to conclude that the start method is querying for all results instead of lazy-loading them 100,200 at a time when it hits the execute method.

Is there any way to do these callouts in the iterator or do I just need to move them inside the execute method?

This is my current code

global with sharing class myBatch implements Database.Batchable<wrapperClass>, Database.AllowsCallouts, Database.Stateful{

    global Iterable<wrapperClass> start(Database.BatchableContext BC) {
        return new myIterator();
    }

    global void execute(Database.BatchableContext BC, List<wrapperClass> scope) {
        //do some stuff. never gets here.
    }
}

global with sharing class myIterator implements Iterator<wrapperClass>,  iterable<wrapperClass> {

    public List<wrapperClass> myData;
    private Integer index = 0;
    private Boolean APIHasMore = true;

    global myIterator() {
        //do initial callout for records, 100 at a time
        //deserialize json into myData
    }

    global boolean hasNext() {
        return  index<myData.size()-1 || APIHasMore;
    } 

    global wrapperClass next() { 
        if(index<myData.size()-1){
            index++;
            return myData[index-1];
        }
        else{
            //callout for next 100 records and deserialize into myData, if there are no more set APIHasMore to false
            index = 0;
            index++;
            return myData[index-1];
        }
    }

    global Iterator<wrapperClass> Iterator() {
            return this;
    }
}

Exception: First error: Exceeded maximum time allotted for callout (120000 ms)
14

You're absolutely correct. The system basically gathers everything upfront. This allows the system to display the total number of batches before it starts processing the data.

The "why" has to do with general limits: you can't query more than 50,000,000 rows, so it needs to know if you'll break that limit and abort early, and there's a daily limit, so the system needs to know if you'll break that limit and abort early if your batch couldn't possibly finish before the limits are reached.

Here's how the process looks in pseudo-code:

Iterator<Object> iter = BatchInstance.start(context).iterator();
Object[] batchData = new Object[0];
Batch[] batches = new Batch[0];
while(iter.hasNext()) {
    Object value = iter.next();
    if(value != null) {
      batchData.add(value);
    }
    if(batchData.size() == batchSize) {
        batches.add(new Batch(batchData));
        batchData = new Object[0];
    }
}
if(!batchData.isEmpty()) {
    batches.add(new Batch(batchData));
}
while(!batches.isEmpty()) {
    BatchInstance.execute(context, batches.remove(0).data);
}
BatchInstance.finish(context);

The actual implementation is abstracted away from us, but it illustrates how the system implements the batching logic.

Instead, you'll want to build an iterator that can determine how many records there are, if your endpoint can provide a total number of records. For example, the Salesforce SOAP API includes a "size" attribute that tells the client how many records will be returned across all calls to the same QueryLocator via pagination.

You can then return a number of placeholders for each record, and then you'll want to call the pagination API and buffer the results in your Batchable class. You'll also probably need to use some state data using Database.Stateful so you can maintain your position across execute calls.

Alternatively, I've used designs where I just count to some reasonable value, then perform a single callout each execute method, process those records, and repeat; if I reach the end early, I can System.abortJob, and if I don't, I can chain to another batch call to keep going.

Here's a basic Iterator:

public class CounterIterator implements Iterator<Integer>, Iterable<Integer> {
    Integer max, current;
    public CounterIterator(Integer maxValue) {
        current = 0;
        max = maxValue;
    }
    public Iterator<Integer> iterator() {
        return this;
    }
    public Integer next() {
        return current++;
    }
    public Boolean hasNext() {
        return max > current;
    }
}

Your batchable's start method can simply be:

public Iterator<Integer> start(Database.BatchableContext context) {
    Integer expectedRecords = WebService.getResultTotal();
    return new CounterIterator(expectedRecords);
}

If you can't get a total upfront, then just pick an arbitrary value, like 10000 or so; you can always abort early.

Your execute method would be:

public void execute(Database.BatchableContext context, Integer[] values) {
    // Do your callout, then...
    if(noMoreRecords) {
        System.abortJob(context.getJobId());
    }
}

And you can finish up with:

public void finish(Database.BatchableContext context) {
    if(!noMoreRecords) {
        Database.executeBatch(new BatchableClass(), 100);
    }
}

Keep in mind that there are governor limits about daily usage.

| improve this answer | |
  • man, there really needs to be a post to ideaexchange button next to the accept answer checkbox. Lazy-loading is one of the main benefits of using iterables instead of collections and it seems like batch jobs largely nullify that. I'll have to use the latter approach since I don't know the full amount of data returned. Thanks for the quick and authoritative answer! – Greg Grinberg Nov 5 '15 at 23:46
  • If queueable allowed chaining and callouts, it would have been the better approach. I sincerely hope they fix that some day. – sfdcfox Nov 5 '15 at 23:56
  • 1
    Released in Winter '15: Chain More Jobs with Queueable Apex – Scott Pelak Jan 2 '19 at 21:01
  • 1
    Released in Spring '17: Make Web Service Callouts from Chained Queueable Jobs – Scott Pelak Jan 2 '19 at 21:03
  • @ScottPelak Yeah, a modern implementation should use Queueable. This answer more focuses on the original question "why aren't iterators in batch jobs lazy-loaded." This answer should probably be preserved as is, but I can add a separate answer that incorporates a modern approach. – sfdcfox Jan 2 '19 at 21:08
3

With the introduction of chaining Queueable, and the ability to chain Queueable with callouts, we can daisy-chain multiple callouts efficiently without having to "guess" how many batches we need upfront from the Batchable interface. A simple implementation would look like:

 public class ChainedCallout implements Queueable, Database.AllowsCallouts {
   Integer pageNumber = 0;
   SObject[] records;
   Boolean hasMore = false;
   public void execute(Database.QueuableContext context) {
     getRecordsFromServer();
     persistData();
     if(hasMore) {
       System.enqueueJob(this);
     }
   }
   void getRecordsFromServer() {
     // do something to get records, set hasMore flag, update pageNumber
   }
   void persistData() {
     insert records; // Or some other logic
   }
 }
| improve this answer | |
  • 1
    This is a great point but I've seen some pretty massive performance differences in how quickly running through hundreds (much less thousands) of batch execute methods is vs chaining that many queuable methods. Queueables end up being rate-limited significantly and the throughput is much lower. – Greg Grinberg Aug 8 '19 at 17:23
  • My approach here now is just to have the batch job iterate over a list<integer> of 10,000 elements and abort within execute method when I reach the end. – Greg Grinberg Aug 8 '19 at 17:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.