Is it possible to use a iterable batch job to page through a web services result set?
I need to import a potentially large number of records into Salesforce via a third party web service. They currently have a method that returns all the possible records, but it runs into callout timeout and SOQL limits when the number of records exceeds a certain threshold.
Instead, I'd like to use a batch apex job to process a finite number of records from the web service at a time. I'm proposing that the web service be extended with two new methods. One that returns the total number of records to be processed and another that returns a subset of records within a range (basically paging support). A iterable batch job could then be created to pull back each page of results and process them within the timeout and SOQL limits.
My concern is the following warning in the Using Batch Apex docs:
If you use an iterable, the governor limit for the total number of records retrieved by SOQL queries is still enforced.
Is this limit only applied within each batch execution or does it apply for the entire batch job?
I suspect the former, but I'd like some confirmation before heading down this path.
In the same doc page there is also:
Each execution of a batch Apex job is considered a discrete transaction.
Which to me implies the governor limits are per execution as well rather than across the entire batch job.