4

I need to consume an API from an external source and insert it in my Enterprise org. I have to retrieve 200 at most in every transaction. I implemented a Queueable Apex class that chains itself in the execute method if there is more data to be retrieved (I get a JSON with a 'has_more_data' boolean.)

The thing is there are over 30k contacts and I'm only able to retrieve the first 1000 (5 executions with 200 contacts) before I hit the stack depth limit.

I tried to create a batch but cannot retrieve every contact in the start method, so I changed it to Apex Queueable. How can I avoid this limitation?

The execute method looks like this:

global with sharing class ContactHistoryQueueable implements Queueable, Database.AllowsCallouts {
// .......

public void execute(QueueableContext context) {
    String contactHistoryJSON = ContactApiHandler.getContactHistory(paginationCursor);
    ContactWrapper contactHistoryresult = ContactWrapper.parse(contactHistoryJSON);

    // Process the contacts
    ....

    if (contactHistoryresult.has_more_data) {
        // Chain next job with next cursor
        System.enqueueJob(new ContactHistoryQueueable(contactHistoryresult.next_paginationCursor));
    }
}
}
2
6

This limitation is specifically only enforced in Sandboxes and Developer Edition orgs. In production orgs, you are allowed to chain indefinitely (but, be aware of the backoff mechanism as described in this answer).

If you want to avoid this problem, consider using Batchable instead. See this question of mine where I wanted to implement a way to get a particular pattern of behavior. The trick is to use some arbitrary data instead of a query, such as returning a list containing many thousands of values. This could be done with a custom Iterable<Object> interface, or another of your choosing.

For example, if your API can tell you how many pages there will be, you can calculate how many execute methods you'll need, and plug that in to an iterator to set the appropriate batch size.

Also, keep in mind that even after doing this, you could use the finish method to make sure there are no more records to retrieve, and if there are, you can then Database.executeBatch again to continue processing.

You could even just return an arbitrarily large number of items, and if you run out of data early, you can call System.abortJob(context.getJobId()) to abort your Batchable partway through.

13
  • Thank you for this. I reviewed it and I think it will solve my issue or at least get me in track Oct 14 at 21:00
  • While I completely agree with the first paragraph, for me the use of a batchable is simply not the right solution to the problem. You have to create enough fake data to have enough iterations and have to use System.abortJob to terminate early - which cannot be used without appropriate permissions that are typically only held by admins. It will likely also use 2 more async calls than you really need.
    – Phil W
    Oct 14 at 21:47
  • @PhilW Fake "data"? Are you not aware that you can use non-record objects in a start method? That aside, most batches should be running in system mode, as it would be inconvenient to run in user mode most of the time.
    – sfdcfox
    Oct 14 at 22:06
  • I am aware you can return an iterator. But this needs to return "fake data" just to provide something to be iterated.
    – Phil W
    Oct 15 at 6:16
  • 1
    @MarcosCarri If you're going with batchable, that'd be the general idea. Just generate a list of whatevers, and you can abort if you run out of work early (but note PhilW's comment about the System.abortJob only working as an admin user).
    – sfdcfox
    Oct 15 at 16:40
4

The approach we take is to:

  1. Ensure that the queueable state can be represented easily using simple types or collections of simple types.
  2. Add a counter as queueable state.
  3. When the queueable is constructed set the counter to 1.
  4. Each time the queueable is about to chain, at the end of execute, check the counter.
  5. If it is less than 5 then increment it and chain itself.
  6. If it is 5 then invoke a future method passing the queueable state (but not passing the counter).
  7. Have the future method construct a new instance of the queueable from the required state parameters (which sets the initial counter automatically) and call execute (with null context).

Job done. The switching to future breaks the chain and avoids any backing off/throttling for this async processing, and having the future call the same execute avoids wasting an async call.

And even better, as long as your state can be represented with simple types/collections of simple types, there is very little new code to write to convert your existing queueable to this pattern.

2
  • 1
    I did think about suggesting this approach, but the answer I provided is based on actual code I wrote for a continuous integration project (millions of API calls a month), and the actual approach you suggested here was my first iterative attempt at implementing what we needed to do. It tended to be far more stable as a batchable (but, tbh, our org was... complicated). This method is almost as a reliable, though, but might require a bit more babysitting.
    – sfdcfox
    Oct 14 at 22:10
  • 2
    Well, the queueable approach benefits from the transaction finaliser, making recovery quite easy to achieve (though not in the future call step). We have used this pattern numerous times and not suffered from a lack of reliability.
    – Phil W
    Oct 15 at 6:14

Not the answer you're looking for? Browse other questions tagged or ask your own question.