Why aren't iterators in batch jobs lazy-loaded?

Question

I have a batch job that needs to callout to an external web service and retrieve a variable number of records. Seems like a perfect use-case to have an iterator which performs the callouts until there are no more results.

However, if the webservice returns many records (in separate paginated callouts) the batch job will hit the maximum callout timeout limit while it is being prepared. This leads me to conclude that the start method is querying for all results instead of lazy-loading them 100,200 at a time when it hits the execute method.

Is there any way to do these callouts in the iterator or do I just need to move them inside the execute method?

This is my current code

global with sharing class myBatch implements Database.Batchable<wrapperClass>, Database.AllowsCallouts, Database.Stateful{

    global Iterable<wrapperClass> start(Database.BatchableContext BC) {
        return new myIterator();
    }

    global void execute(Database.BatchableContext BC, List<wrapperClass> scope) {
        //do some stuff. never gets here.
    }
}

global with sharing class myIterator implements Iterator<wrapperClass>,  iterable<wrapperClass> {

    public List<wrapperClass> myData;
    private Integer index = 0;
    private Boolean APIHasMore = true;

    global myIterator() {
        //do initial callout for records, 100 at a time
        //deserialize json into myData
    }

    global boolean hasNext() {
        return  index<myData.size()-1 || APIHasMore;
    } 

    global wrapperClass next() { 
        if(index<myData.size()-1){
            index++;
            return myData[index-1];
        }
        else{
            //callout for next 100 records and deserialize into myData, if there are no more set APIHasMore to false
            index = 0;
            index++;
            return myData[index-1];
        }
    }

    global Iterator<wrapperClass> Iterator() {
            return this;
    }
}

Exception: First error: Exceeded maximum time allotted for callout (120000 ms)

Answer 1

You're absolutely correct. The system basically gathers everything upfront. This allows the system to display the total number of batches before it starts processing the data.

The "why" has to do with general limits: you can't query more than 50,000,000 rows, so it needs to know if you'll break that limit and abort early, and there's a daily limit, so the system needs to know if you'll break that limit and abort early if your batch couldn't possibly finish before the limits are reached.

Here's how the process looks in pseudo-code:

Iterator<Object> iter = BatchInstance.start(context).iterator();
Object[] batchData = new Object[0];
Batch[] batches = new Batch[0];
while(iter.hasNext()) {
    Object value = iter.next();
    if(value != null) {
      batchData.add(value);
    }
    if(batchData.size() == batchSize) {
        batches.add(new Batch(batchData));
        batchData = new Object[0];
    }
}
if(!batchData.isEmpty()) {
    batches.add(new Batch(batchData));
}
while(!batches.isEmpty()) {
    BatchInstance.execute(context, batches.remove(0).data);
}
BatchInstance.finish(context);

The actual implementation is abstracted away from us, but it illustrates how the system implements the batching logic.

Instead, you'll want to build an iterator that can determine how many records there are, if your endpoint can provide a total number of records. For example, the Salesforce SOAP API includes a "size" attribute that tells the client how many records will be returned across all calls to the same QueryLocator via pagination.

You can then return a number of placeholders for each record, and then you'll want to call the pagination API and buffer the results in your Batchable class. You'll also probably need to use some state data using Database.Stateful so you can maintain your position across execute calls.

Alternatively, I've used designs where I just count to some reasonable value, then perform a single callout each execute method, process those records, and repeat; if I reach the end early, I can System.abortJob, and if I don't, I can chain to another batch call to keep going.

Here's a basic Iterator:

public class CounterIterator implements Iterator<Integer>, Iterable<Integer> {
    Integer max, current;
    public CounterIterator(Integer maxValue) {
        current = 0;
        max = maxValue;
    }
    public Iterator<Integer> iterator() {
        return this;
    }
    public Integer next() {
        return current++;
    }
    public Boolean hasNext() {
        return max > current;
    }
}

Your batchable's start method can simply be:

public Iterator<Integer> start(Database.BatchableContext context) {
    Integer expectedRecords = WebService.getResultTotal();
    return new CounterIterator(expectedRecords);
}

If you can't get a total upfront, then just pick an arbitrary value, like 10000 or so; you can always abort early.

Your execute method would be:

public void execute(Database.BatchableContext context, Integer[] values) {
    // Do your callout, then...
    if(noMoreRecords) {
        System.abortJob(context.getJobId());
    }
}

And you can finish up with:

public void finish(Database.BatchableContext context) {
    if(!noMoreRecords) {
        Database.executeBatch(new BatchableClass(), 100);
    }
}

Keep in mind that there are governor limits about daily usage.

Answer 2

With the introduction of chaining Queueable, and the ability to chain Queueable with callouts, we can daisy-chain multiple callouts efficiently without having to "guess" how many batches we need upfront from the Batchable interface. A simple implementation would look like:

 public class ChainedCallout implements Queueable, Database.AllowsCallouts {
   Integer pageNumber = 0;
   SObject[] records;
   Boolean hasMore = false;
   public void execute(Database.QueuableContext context) {
     getRecordsFromServer();
     persistData();
     if(hasMore) {
       System.enqueueJob(this);
     }
   }
   void getRecordsFromServer() {
     // do something to get records, set hasMore flag, update pageNumber
   }
   void persistData() {
     insert records; // Or some other logic
   }
 }