How to send a large number of records to an external system in multiple callouts?

Question

Object A has multiple child records of object B (about 10 000).

The requirement is to send the whole object A including all records of object B when a checkbox field is set to TRUE on object A. This means a maximum of 10 001 records to be sent to an external system.

Now the problem is that 10 001 records will exceed the heap size limit and I'd like to split the records to be sent for example in batches of 2 000, but if any of those batches fail to be sent (the external service returns 400 Bad Request for example as a response to one of the calls), I need to terminate the transaction and not send anything more and somehow later identify that the object A with object B needs to be resent in a retry mechanism.

I was thinking I could make a custom staging object, where I would create a record for each integration to be fired, so the checkbox on object A would change and I would create a record with the ID of the object A and also the status of the integration.

However, I don't know if it possible in Salesforce to make a continuation of callouts where the records would be split by for example OFFSET in the query. The only continuation reference I could find in the documentation was for Lightning Web Components. I need this to be fired from a trigger.

What I can not do:

• Move this integration to a middleware (the integration has to be outbound from Salesforce to external system)
• Send only the changes (all the record A and records B have to be sent)

Kindly asking for some advice.

Answer 1

Here are a couple simple ideas for you

• add an A.Last_B_Sent__c text custom field
• when A.Checkbox__c field is changed from false to true, kick off your batch process, sending it A.Id. Create a done property on the batchable class and set it to false initially.
• in your batchable start method, query all B records for A.Id, WHERE B.Id > A.Last_B_Sent__c, ORDER BY B.Id
• in the batch execute method, if not done, do the callout and update A.Last_B_Sent__c with the last B.Id successfully sent (handle an update failure as well). If the callout fails, set done to true
• in the batch finish method, update A.Checkbox__c to false (handle an update failure as well)

A simple retry mechanism could be a periodic batch process that queries all B records where B.A__r.Checkbox__c is true and B.Id > B.A__r.Last_B_Sent__c. The execute and finish steps would be the same as the process above.