Given that there are no documented guarantees around order of execution for Queueable implementations (these do not go into a queue in the same way that Batchables do), I would consider creating a "single threaded" processing approach instead. By ensuring that all the outgoing messages are handled through a mechanism designed to ensure that there is no (or using specifically controlled) concurrency, and basing order of send on the created date for your custom records, you can guarantee the correct order of delivery for deltas applied to any given Case record.
It is easy enough to create a "single threaded"
Batchable (this is something we did for ourselves several years ago). I haven't, as yet, created a general purpose "single threaded"
Queueable. However, in this case I believe that is actually unnecessary.
I have created an example of an approach based on a "command object", Platform Events and a simple callout handling
Queueable - I'd have loved to have avoided the latter, but Salesforce does not (yet) support making a callout directly from a Platform Event subscriber.
The principles here are:
- To capture creation, update and delete events for Case (in this specific example) in a custom object, here called
Command__c. This includes a status to help control ordering of sending for any given record and a JSON string of the relevant fields and their values.
- To generate a Platform Event that causes the out-of-transaction processing of these "commands". This avoids any nasty governor limits for initiating async Apex within triggers, regardless of whether the trigger is handling bulk updates and/or is itself being called from an async context.
- To use a simple
Queueable to convert up to 100 commands per execution into callouts, while ensuring that the order of "commands" is retained for any given record.
You can find code demonstrating this in github - the only thing it doesn't do is make any callouts (but you can find where this should be done by searching for "TODO").
Some of this code is lifted from another of my repos, one supporting an article I wrote for ApexHours.
NB: While I've had a play with this with some individual and bulk DML operations (create, update and delete), I do not actually guarantee that it really is doing what I've said and ensuring that send order is maintained - I will do some specific analysis by adding debug in the near future and update the repo if required.
UPDATE: The following are Q&As related to the example solution based on questions added to this answer.
Where in the process should I capture the data to send?
This needs to be captured in the DML processing for the record (here Case) insert/update/delete.
To try to capture this asynchronously (e.g. in the platform event trigger handler) means you could end up querying the case(s) being created or updated after they have already been further updated (or even deleted), meaning you would not capture full history.
Additionally, to do this capture outside the DML context means you won't know which fields changed and cannot easily compute the delta to only send individual field changes when handling DML updates.
The example code captures the field values into the Command records within the context of the Case trigger.
Yes, this can introduce CPU limit issues if someone tries to bulk insert or bulk update a large number of Cases - with this code in place I got a CPU timeout when trying to bulk insert and update 1000 cases in a single transaction (in two DMLs), though I could successfully bulk insert and update 500 cases together.
Won't creating a "Command" per DML cause other governor limit issues?
If there is a lot of automation against Case (or other target object), or the calling code is trying to perform massive bulk insert/update/delete operations against the object, it is possible to hit various governor limits (e.g. DML row issues or CPU timeout as mentioned above). However, in the callout side of things, this sequential generation of "Commands" should not directly cause governor limit issues.
Once the Commands exist in the database, the platform event trigger invokes a processor that in turn queries Commands waiting to be processed (using oldest to newest and ensuring that no Commands are processed if there is already one on-going for a given record). This ensures that at most 100 Commands are processed at a time, thus staying within transaction callout limits, leaving others to the next time the platform event trigger is called (and ensuring this will happen by raising a new platform event).
If 100 is too many, e.g. because the automation for the callout is too complex and you would hit CPU limits or cumulative callout timeouts, you can simply reduce the number of Commands processed together by changing the LIMIT on the SOQL in the
This will send callouts when a change X is made that is then reversed out
Indeed, if you have a Case where the description (or some other field) with value A is changed to value B then back to A in two separate DML operations, this will generate callouts. However, this is the correct behaviour because for a short while the Case's description (or other field) did change value.
To try to "optimize" this makes little sense to me; you have no idea how long the time is between the first and second change.
The one optimization that is worthwhile (and that I didn't do) is to detect that no field values changed and avoid generating a Command in the first place.
What happens if the end point has downtime?
In this case we just need to keep the Commands and use them again for later retries.
I didn't show any error handling in the
CommandSender, but that is easy to add. The key is for the sender to capture the set of Command IDs for which sending failed into a new Set of IDs and at the end of its execute ensure it doesn't update these Commands to
Complete status. Instead it should update its own
commandIds set to reflect this content then explicitly re-enqueue itself via
System.enqueueJob, but this time explicitly passing a delay. This then easily implements a retry.
You could even implement a back-off process, lengthening the delay after each re-try, by also storing an attribute in the queueable that is the previous delay value (defaulting to 0 to indicate there was no delay before the first re-try) and using this to calculate the next delay value.
You may need to ensure that you don't enqueue further
CommandSender Queueables in the
CommandProcessor during this downtime (I leave this as an exercise for the reader, but I'd consider having a "Resending" status for a Command).
Your Command cannot cope with Long Text Area fields
True indeed. Here I would treat those fields specially, having specific additional "generic" Long Text Area fields in the Command that are explicitly used to store Long Text Area field values (for "known" fields of this type for a given object) and update the
UpdatedFieldsJSON__c generation to skip these fields; once the JSON is generated I would iterate the required Long Text Area fields and (conditionally, for update) process them through to these separate fields.
Obviously you may need to explicitly configure some mapping for these, or you may explicitly store the name of the long text area field being stored, alongside the value, using two separate fields... e.g.:
- LongTextAreaFieldName1__c - to store the field name.
- LongTextAreaFieldValue1__c - to store the field value.
You could have n of these field pairs you simply dynamically iterate to store insert/update for these Long Text Areas. Obviously if there are more than the n fields in your Case (or other object) you will need to add extra pairs into the Command.
Can we store history of the updates?
Yes, in fact that is exactly what you get with the Command. These describe the changes applied (and could capture who did it too, if you wanted). It's just that the Command also includes a Status that controls how they are converted to callouts.
Now, directly storing all history in the org is not going to scale. Leaving the Commands in the database will use a lot of storage and, worse, cause SOQL scalability/selectivity issues down the line.
You should only keep history for the recent past in the org, optionally archiving and explicitly deleting Commands when they are old enough (and in
Couldn't this be easier to implement?
Well, yes, it could, if Salesforce supported callouts from Platform Event trigger subscribers.
If this was supported, we get a simple way to implement single-threaded handling for all the callouts. The down-sides would be:
- You cannot recover from governor limit issues (which you can using a Finalizer with a Queueable).
- It would be literally single-threaded. There would be no sensible way to do parallel sending of independent Commands (i.e. relating to different records) without re-introducing the Queueable.
Feel free to upvote this idea for direct callout support.