Prevent limits in a callout intensive process where client can only receive one request at a time?

Question

I have a callout intensive process where the client can only receive one request at a time and I'm having a hard time identifying what is the best practice for asynchronous solutions to handle this problem. One record per request, the client system can handle multiple requests, but only one record per request. These are changes are being invoked by a trigger when the record is updated.

Batch

If there is a large data update that results in 5000 requests needing to be sent to this system that can only accept one request at a time, I was thinking of batching 500 batches of 10 callouts each, but there may be other agents (ex: 200) also performing operations that would result in http callouts, and I ideally wanted to follow the same pattern of whether it was a mass update or a single record update. But if 200 agents are updating 200 records, resulting in 200 batch jobs, I could end up with the 100 flex queue limit exception.

Queueable

I cannot find any sort of limit on queueables in regards to how many queuables you can have scheduled other than 50 per queueables scheduled per transaction. I could do 100 http callouts in each queueable for a max of 50 queueables, processing 5000 total records.

However if it were 6000 records... it seems queueable or batchable leaves me vulnerable to potential exceptions.

Scheduled Batch + Custom Object / Indicator Field on the Object (Checkbox)

I could place them in a temporary storage custom object or place an indicator on the record and schedule a job that runs batches on the dataset to send only what I can every X minutes that would allow for the batch jobs to go down in the flex queue to clear. However, if my requirement is to send updates at least every 10 minutes, there would be 144 (6 x 10 minutes) * 24 hours a day schedule jobs which I would think would put me over the 100 scheduled apex jobs at one time (unless this literally means at one time like 100 jobs at 6:05pm).

Hybrid

I could also do different paths based on how many records are being updated, which is my most likely choice. Most use cases are single record updates by users, but there may situations that require the update of 1000s of records. If it's one record update I could identify this by looking at the size of the records passed from the trigger and synchronously make the request for that record. Where any update of more than one record would result in being placed as a custom object to be picked up as needed.

I'm leaning towards the hybrid approach, but do not like that I am following different patterns based on size leading to maintenance of two routes of outbound requests and their logic to manage the requests.

Answer 1

I have one solution in my mind maybe it will work

So what you can do is you can create a new field like Is_For_CALLOUT__c which will be us ed to mark the record to go for the callout.

Then in your Before update trigger make Is_For_CALLOUT__c = true for records that are going for the callouts. Then check your batch(which will do the callout) is running or not and if it's not running then you can run the batch otherwise don't run the batch.

Batch will check if any record which has Is_For_CALLOUT__c == true then make the callout and update Is_For_CALLOUT__c = false

last, you can call your same batch from your finish method

So from that, your batch is like polling on records. That way you will not find any limit error.

So if one agent update the record so it will run the batch then if another agent update the record then it will mark the record for callout. In finish method our batch again run if its find any record then it will make callout.

you can take boolean variable by default it will false. in execute method you can make it true. this variable will use to check we have to run batch again or not.

Hope that works for you.