I have a callout intensive process where the client can only receive one request at a time and I'm having a hard time identifying what is the best practice for asynchronous solutions to handle this problem. One record per request, the client system can handle multiple requests, but only one record per request. These are changes are being invoked by a trigger when the record is updated.
If there is a large data update that results in 5000 requests needing to be sent to this system that can only accept one request at a time, I was thinking of batching 500 batches of 10 callouts each, but there may be other agents (ex: 200) also performing operations that would result in http callouts, and I ideally wanted to follow the same pattern of whether it was a mass update or a single record update. But if 200 agents are updating 200 records, resulting in 200 batch jobs, I could end up with the 100 flex queue limit exception.
I cannot find any sort of limit on queueables in regards to how many queuables you can have scheduled other than 50 per queueables scheduled per transaction. I could do 100 http callouts in each queueable for a max of 50 queueables, processing 5000 total records.
However if it were 6000 records... it seems queueable or batchable leaves me vulnerable to potential exceptions.
Scheduled Batch + Custom Object / Indicator Field on the Object (Checkbox)
I could place them in a temporary storage custom object or place an indicator on the record and schedule a job that runs batches on the dataset to send only what I can every X minutes that would allow for the batch jobs to go down in the flex queue to clear. However, if my requirement is to send updates at least every 10 minutes, there would be 144 (6 x 10 minutes) * 24 hours a day schedule jobs which I would think would put me over the 100 scheduled apex jobs at one time (unless this literally means at one time like 100 jobs at 6:05pm).
I could also do different paths based on how many records are being updated, which is my most likely choice. Most use cases are single record updates by users, but there may situations that require the update of 1000s of records. If it's one record update I could identify this by looking at the size of the records passed from the trigger and synchronously make the request for that record. Where any update of more than one record would result in being placed as a custom object to be picked up as needed.
I'm leaning towards the hybrid approach, but do not like that I am following different patterns based on size leading to maintenance of two routes of outbound requests and their logic to manage the requests.