Best Practice for Bulk Lead Uploads with Triggers

Question

I have 5 different triggers calling out to different API services (I am using @future callouts) every time a new lead is added to Salesforce.

My concern is that when I bulk add say 500 leads to Salesforce with DataLoader (haven't tried yet) all those triggers will end up shooting a request out at the same time and I will exceed the rate limits (or at least seriously clog them) for the APIs that I am using ending up in 100s of failed triggers.

Is there a way to ensure that those triggers don't get fired all at once for each lead that is being added but instead respect the rate limits/proceed at a slower pace?

Answer 1

How quickly do you need those 5 callouts per Lead to occur?

If that answer isn't immediately and you could wait a few minutes before doing the callout then a custom queue object could be a useful solution.

It would look roughly like this:

• New Custom Object to represent the Leads that need the callouts to occur
• Replace existing triggers that call the future methods with one trigger that upserts a record into the new custom object with the Lead Id.
• Create Scheduled Job (or jobs) that start a batch job.
• Create a batch job that runs for a query over the custom object. It will find all the Leads that still need to callouts to occur.
• In the batch job execution it performs the callouts. Has the option to flag Leads that are failing certain callouts.

Now you no longer have to worry about causing a massive spike in callouts. Or at the very least you have a mechanism where you can control the callout rate and handle any potential errors.

There will be a number of variations on this depending on how fast you need to records to be processed and how many there will be. For instance, you could try the new Platform Events to publish events when the leads are modified. It might be possible to have some of the external systems subscriber to those events.

Answer 2

Well, like @Kurunve said, on option certainly is to rewrite the triggers, although depending on the org, it can be a real challenge...

Another option is to set your batch size when you upload records. Both in the Apex Dataloader and Dataloader.io you can configure it. For the first one, it's in the settings as you can see here:

I'm not sure you have any other options, but maybe someone else knows.

Answer 3

If you have a concern about how often your triggers will cause a call out you could create a new object. Something like Service Transactions, the responsbility of this object would be to track your Leads, Contacts, Accounts.. Really anything. So rather than making a call out everytime your Lead record gets proccessed you could create a new service transaction record where you would store the ID, then you could create a Batch that would process these Service transaction records to your liking.. This is easier said than done, but this is just a thought if you have a real concern about how often your triggers are firing..