Working with large data volume

Question

I have a design question and I want an expert opinion from the group.

Here is the use case:

1. Whenever there is any update on the Booking record (let's say change the status to "Cancelled"), I want those records to be updated in another system. They have their table to store these updates.
2. It has to be real-time. So as soon as a record(s) are updated we will update their system.
3. It should work for the single record as well as multiple records (let's say 100K records)

Solution 1: I can introduce streaming API and let the other system subscribe to it and on any update, they can get the payload and work on it. But I am not sure how efficient this would be when we work on 100K booking records at the same time.

Solution 2:

We can call a future(callout = true) method from the trigger. I am not sure how efficient that solution would be when I am working with 100K records simultaneiusly. We have a limitation of 100 callouts per transaction using the asynchronous code. I know we can solve this by using EnqueJob Interface too, but I am not sure how can we fit EnqueJob here.

What would be the best approach to work with a large number of data in Salesforce? I am more interested to solve it with Solution 2.

I would really appreciate any example with a pseudo-code that would help me understand the concept quickly.

Answer 1

In Salesforce, you can only update 10,000 records in a transaction, and that's via Apex. Otherwise, using the API and other tools, you're limited to just 200 records per transaction. So, at no point will you be updating 100k records "all at once."

CDC (Change Data Capture) can stream these updates out efficiently, up to 2,000 at a time, so somewhere between 50 and 500 streaming events depending on efficiency. If you write a trigger and use future methods, you just need up to 2 future calls per 200 records, the maximum you'll see in a trigger context, although this leaves a theoretical maximum of 100 future calls in some weird cases (e.g. an Apex class that updates 10,000 rows at once). Ideally, you should limit any code you write to handle no more than about 2,000 rows at once.

In summary, I think it's safe to say that any solution you engineer will probably work as far as Salesforce is concerned. The possible exceptions to this revolve around if you might need to call this class from a batchable class, but that's a fairly easy problem to work around with some coordination in your code. We have plenty of examples of that here on SFSE if you do a quick search.