Queueable Apex consideratoins

Question

This is more of a theoretical question and probably goes more to design and I hope I can do this justice.

I have successfully coded in apex multiple triggers which make callouts using the @future (callout=true). These work just fine and I do realize when using the @future these are asynchrous calls and thus I do not have control when salesforce will process although it is usually pretty fast. All that being said.

We have a new community and the issue is it is making multiple insert/updates in a single transaction. My triggers fire and the @future callouts are made which are intended to put data into an external system. The issue as you may imagine is sometimes children get to the external system prior to the parent and I get 500 error codes which is probably due to that scenario but cant guarantee it. A retry feature would be nice as I believe it is an order of when the the external system processes the data at the endpoint.

I mention Queueable apex as I have loosely encounter discussions that the solution could possibly be found here. I am not necessarily tied to solving this issue via Queueable apex but maybe someone has encountered a similiar sceneario.

The real question is has anyone solved an issue like this and how did you do it?

Thanks!

Answer 1

Having built a "rate-limited multiplexing API engine" before, I can tell you that a robust solution can quickly get very large. We had a complicated system that had the following features:

• Throttled API calls per minute, configurable per endpoint.
• Priority queue for near-real-time updates, secondary queue for less sensitive updates.
• Automatic API dependency check; a parent API call would always go before the child.
• Enable/disable individual endpoints, per API call.
• Enable/disable transaction creation, per API call.
• Disable/restart scheduled class responsible for kicking off batchable class.

This solution ended up having:

• Utility class for creating (possibly dependent) transactions.
• Scheduled class for calling the batchable class on the hour.
• Batchable class that iterated over oldest transactions and performed callouts for almost an hour then self-aborted to allow the scheduler to restart the process.
• One class per API call to allow for dynamically adding new services.
• One custom object for logging pending API calls.
• One list custom setting for enabling/disabling/throttling API calls.
• One Visualforce page and controller for configuring all this data.

Of course, you don't need to make it this complicated, but the key points here are to have a persistent object that can log requests and even enable you to retry failed callouts, and come up with a dependency system so you can specifically order objects in a way that parent records will never be sent before their children. You could tweak this a little further to use Queueable instead of Batchable, schedule jobs more frequently than every hour, etc.

Unfortunately, about the best we can say for sure is that you need to plan ahead instead of just coding. Consider all the possibilities. Maybe this won't be "real-time" enough for you, or maybe it will overwhelm the external system resources (we had that happen on our first pass with this project), or something else entirely. Make sure you have a decent error logging framework, perhaps with some sort of threshold where it'll email someone if the errors stack up too high.

Overall, it's possible to do what you're trying to do, but the path there might be a little more complicated than you originally expected.

Answer 2

Queueable won't fully solve your problem. Queueable can ensure that the asyc stuff starts in order, but doesn't take into account how the order may change as the external systems communicate over the internet e.g. if you start to sync a parent and the external system is being slow, then you start to sync the child and it fails because the parent hasn't finished yet. SF has done its part - the operations started in order but, depending on the external system, they may not finish in order.

I've built something like the system sfdcfox mentions myself, and it's a really great multi-purpose solution to have.

If you can't justify spending all that effort, and you don't care about the order of records that aren't related to each other in SF, then the alternative is to coordinate things via status fields in the SF records. You don't say what objects you're syncing, so I'm just going to pick standard objects with a parent/child relationship.

Let's suppose you needed to sync Accounts and Contacts, and that the Account must be sent before the Contact. The simple way would be to fire an @future any time it became apparent that the object needed to be sent. But, that is where you are right now: no guarantee that Account exists on the external system before Contact is sent.

So:

• add a field called something like Sync_Status__c to both Account and Contact
• when an Account needs to be sent, set Sync_Status__c to Pending and fire the @future method
• when the Account @future returns success, update the Sync_Status__c to Sent
• when a Contact needs to be sent, make sure the parent Account has Sync_Status__c = Sent before you start the @future
• add a trigger to Account for when Sync_Status__c ticks over to Sent, so that any Contacts which were waiting for the Account get sent at that point

If you expect to sync other objects in the future, then a multi-purpose system like sfdcfox's will be better in the long-run. If not, this will do the job with much less effort.