Understanding Batch Apex / Callout performance

Question

I am looking for some advice regarding the optimization of an Apex Batch which is behaving quite differently in UAT and PROD environments.

In our ORGs we have a Batchable process that is responsible for fetching data from different sObjects (all of them using a filter on a Lookup field, thus being indexed), sending that data to an external endpoint, and ultimately delete records.

While testing it on a UAT environment the callout times are quite fast (40/50ms), the batch scope can be set as 200 and the whole process works as expected.

However, while trying to run it on PROD the performance is heavily degraded (callout time increases to 9000ms), no more than 10 records can be modified on each scope and overall each transaction takes around 3 minutes to be completed.

There are some differences between the environments, the most important ones IMO being:

• The data volume is pretty much bigger in PROD (We're fetching around 20K records on the QueryLocator but there are roughly 3 million records in total)
• Around 10 future calls are being made per minute (while the batch is also processing);

I tested invoking the PROD endpoint outside of Salesforce (using Postman) with the same number of records on the payload which are being sent from Salesforce (10) and the response time varies between 12, 30, and 120 seconds (leading to a timeout, thus I am ruling that the issue is only related to Salesforce).

I used Query Planner to test the query used in the batch Start method, which I have attached to this post.

Although the issue might also be related to infrastructure on the endpoint side and we haven't run against any Non-selective query exception I would like to assure optimal performance on the Salesforce side. Would the data volume impact the overall callout time to an external endpoint?

I am running out of ideas to improve the batch. Would you advise any particular action here? Looking forward to discussing some ideas and let me know if it wasn't fully clear.

Best regards and thank you in advance!

PS.: Edited to include the Execution Overview Timeline after the initial comment:

Answer 1

Having a LDV scenario won't affect execute run time significantly, but it may affect the total execution time in start; the larger the potential query pool, the more time it takes to retrieve all the results. Since you're using indexes, this probably would be an insignificant amount of time.

If you are in a very "busy" org, with lots of Queueable and/or future calls, there may be a longer delay between each call to execute, since other jobs are competing for resources. However, again, this shouldn't largely impact the runtime within a single execute call.

---

Anecdotal Evidence; skip if boring...

In fact, I once wrote a simulated multitasking, throttled endpoint service that was responsible for sending dependent, backlogged API calls to six different services, giving priority to real-time updates and queueing API/batch updates for later execution, with the ability to enable each endpoint independently, as well as set custom rate limits for each endpoint, and move queued API updates to real-time updates if a user manually changed a record that was previously queued.

All of this was needed to handle about 4 million records that needed to be integrated to systems that could not handle the speed at which Salesforce was able to blast out records (it rendered the servers on the far end unusable for real-time users that needed to use its front-end).

The org itself had probably tens of millions of accounts, much less all the other data they had stored in that org. The queue was estimated to be backed up by about three to four weeks; we kept tinkering with the settings to keep the servers on the far end running at about 80% capacity so they wouldn't just decide to quit their day jobs.

Before the throttling implementation, Salesforce was able to gracefully throw out some odd 300 API calls per second (estimated, it was hard to tell with all the crashing that kept happening), bringing the remote services to their knees and begging for the sweet release of a crash, which happened frequently. A single Batchable class was able to bring down six different servers, at once, without even breaking a sweat. Honestly, it was beautiful to watch, but was costing a ton of money for the business, so it had to be fixed.

The reason why this was possible was that the endpoints were a middleware that queued up actions (so, approximately millisecond response times), which were then fed to the far end servers as fast as they could eat them up, some servers actually needing up to about 1.5 seconds per record. I never knew why, but it sounded like an unbearable system to work with. I was glad to be in Salesforce.

---

Anyways, that's all just a long-winded way of saying that it's almost certainly not Salesforce. The platform is an absolute beast of a system that is pretty much an engineering marvel. Now, I'm not saying Salesforce is perfect--when Salesforce has an outage, it tends to be a major showstopper--but I am saying that under normal circumstances, Salesforce can dish out more than many small endpoints can handle. Salesforce does scale well, and is a force to be reckoned with.

So, it's almost certainly the far end. You might need to write some sort of rate-limiting code, or you might need to optimize some indexes on the far end, etc. I suppose you might be running into some weird bug, and of course, for that, you should probably at least see if you can get a hold of R&D (via Support) to see if they can diagnose any problems on the Salesforce end, but most likely, you have a configuration issue and/or index problem on the remote server.