I have an apex class that is going to make more than 100 api calls to an external resource to get more than 10000 records for insertion into Salesforce.

What should I use to get around this?

I cannot use batch apex because I am not iterating over Salesforce records.

The purpose of this class is to run nightly to gather records from an external system for insertion into Salesforce.

  • Sounds to me like you should take a step back, and re-evaluate what you're trying to do, and why you want to do it. What makes you so certain that you'll use more than 100 callouts? Why do you need to integrate that much data on a daily basis (are you sure that all of the records are required)? I can think of a few potential workarounds (Scheduled Apex + chained Queueable, Direct external objects using OData, Indirect external objects using OData + heroku + postgres), but they're all pretty heavy-handed approaches, and I'd really put in some effort to try to avoid them if practical. – Derek F Oct 25 '17 at 20:18
  • @DerekF There is a good chance there will be more than 100 callouts, the system only returns 1000 records per page, and I can only request one page at a time :( The records are all required per the business unit. I was hoping there is a way to do this async so limits do not apply. We are talking about inserting up to 200k records a night. Divided by 1000 records per page, thats a lot of API calls. – Steel Reserve 211 Oct 25 '17 at 20:20
  • @DerekF Perhaps if I created a custom object containing a record for each "page" and did a batch apex interation over the number of pages expected? – Steel Reserve 211 Oct 25 '17 at 20:26
  • Could really use more details here. What is this 'system' you're grabbing data from? Is it something you/your company controls? Can you send data to Salesforce from this system throughout the day to reduce the load you'd need to handle at any one point in time? Does this system not have a proper API (sounds like you're scraping pages)? – Derek F Oct 25 '17 at 20:26

You can still use batch Apex - even if you're not iterating over record. For example, this would require you to create a start method returning a list of strings. The execute method would then take a List<String> as the scope parameter:

global class MyBatchClass implements Database.Batchable<String>, Database.Stateful

In short you just need something to iterate on. If your external service gives a count, you could use that. The Database.Stateful allows you to store the information (in non-static variables) in the MyBatchClass that a serialized/deserialized for each execution on the execute method.

| improve this answer | |

It wont let me comment because I have less than 50 reputation, so this isn't really an answer - just something to also be aware of. Each record in salesforce is 2kb in size and if you're inserting 200k records per night that's 400mb of storage you're taking up per night. That might easily put you over your data limits:


Just something to keep in mind as you develop this

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.