10

This is a bit of an odd one. We are interacting with an external system that does not support concurrency due to a bug which they are struggling to fix. We perform a HTTP callout, the callout takes between 15 - 45 seconds to run.

During this time no other request to this web callout can occur or the system it is calling to will essentially crash. Because of how long this operation is, it must be an asynchronous job of some sort

NOTE: To clarify, if different transactions invoke this same callout within the same period it will cause the same issue, this problem is separate to Salesforce Transactions. A single Salesforce Transaction could require multiple callouts though

What we've thought of so far:

  • Use scheduled batches (size of 1) and remove any transactional interactions with the API: This is an okay solution, it does allow us to run each call one at a time relatively safely, but we would like something that is closer to real time if possible
  • Chained Jobs using Flex Queue + Queueable Interface: This sounds okay on a transactional level, but if two transactions occur at the same time / similar time it will create multiple chains
  • Future jobs: This is what we're using right now, Future jobs will run whenever regardless of the status of other jobs and definitely allow the issue to occur

Is there another type of asynchronous processing interface we could not think of? Or a Salesforce mechanism to guarantee only one of an Interface will run at any given time

  • 2
    You could create a queue object and instead of running the future write a record in the queue object. Then have a batch work through that one at a time – Eric Apr 30 '18 at 4:39
  • That's the direction I was thinking with the Scheduled Batches. Not ideal, but it will work. I was hoping there was an alternative that might work – Mattisdada Apr 30 '18 at 5:14
  • Not sure its a perfect way but can d one thing. Create a custom setting and check it when you are making callout and uncheck it after transaction complete. All callout if this checkbox is checked show user some information to try again. Not a best approach but will work. – Tushar Sharma Apr 30 '18 at 5:17
  • @Mattisdada the batch could run all the time continuously. – Eric Apr 30 '18 at 11:19
  • @Eric An infinitely chained Queueable job? Otherwise I don't know how you could run a batch all the time – Mattisdada May 1 '18 at 6:31
10

I've implemented the following in numerous orgs and it works pretty well. It's similar to Keith's suggestion, but a bit more detailed and it does usually do near-realtime processing:

  1. Create a custom object to queue records that need to be sent via callout. Each time a callout is required add a record, use an auto-number to preserve ordering
  2. Do all of your callouts in a Queueable which processes one record at a time
  3. Create another custom object for mutual exclusion (let's call it Mutex__c). This has a external id field on it, referring to the process you want to run (I often use this to manage multiple integrations in one org) and a checkbox field on it called something like Run_Queueable__c
  4. Create a trigger on Mutex__c object which starts a Queueable when Run_Queueable__c turns from false to true
  5. Have your Queuable set Run_Queueable__c to false when it has nothing left to process, otherwise keep re-queueing itself until everything is done
  6. Have a trigger on the queue items which does an upsert with the queueable name and Run_Queueable__c = true

This ensures that only one Queuable is running at once. Even if two transactions start at once, only one of them gets to set the mutual exclusion record from false to true. The other one just writes over the true value with true again, so it doesn't start another Queueable.

So, I'd have something like this as a trigger on the queue object:

Set<String> doCalloutStatuses = new Set<String> {
        'Pending',
        'Retry'
};

for(Integer i=0; i < newList.size(); i++) {
    My_Queue_Object__c newQ = newList[i];

    if(doUpsertStatuses.contains(newQ.Callout_Status__c)
            && (oldList == null
            || !doUpsertStatuses.contains(oldList[i].Callout_Status__c))) {
        upsert new Mutex__c(Queueable_Name = MyQueueable.class.getName(), Run_Queueable__c = true);
        return;
    }
}

I can't really post all the code as it's integrated into a load of internal libraries that we have. But, hopefully, you get the idea.

Generally, it works well for me. The one major complication has been that if the Mutex__c object gets out of sync with what's actually running, then you're in trouble. This can happen during an org-split or SF maintenance where they kill your job before it has chance to set Run_Queueable__c=false. They you get what they call a zombie process in Unix, so you need a scheduled job to go reap them.

  • Is there a FOR UPDATE somewhere in the process that upserts the Mutex__c records? Couldn't there be a race condition with multiple transactions where the trigger sees Run_Queueable__c toggle to true? – Daniel Ballinger May 1 '18 at 1:23
  • 1
    I don't have a FOR UPDATE in there. I don't think that there is a race condition because there is only one process allowed to set Run_Queueable__c to false (the Queueable itself). Multiple transactions trying to set it to true isn't a true race condition because it doesn't matter what the eventual order is. The DB makes one of them win, so it toggles exactly once and the rest of the order is irrelevant. This has been running in prod for years for many customers, including some very busy systems. I would love to write a blog about this one day, but it would be LONG. – Aidan May 1 '18 at 10:48
  • Just for fun I tried to break it by introducing a delay in the trigger execution to lengthen the transaction time. Seems to handle it just fine. You either get the latest/current value for the field via Trigger.oldMap or the parallel transaction fails out with a UNABLE_TO_LOCK_ROW. ACID database++ – Daniel Ballinger May 2 '18 at 0:55
  • That's one of the details in the real system. The mutex record become a global that everyone wants access to, and this can cause row lock errors. So, rather than do an upsert, I actually query it first (without FOR UPDATE or that would defeat the point) and only set if I'm actually going to change the value. And I catch row lock errors to do a few retries. In a B2C org where the customers can trigger callouts, I found this was necessary. It was working fine for most (B2B) orgs before I started doing that, though – Aidan May 2 '18 at 8:33
5

The only locking mechanism I know of is the SOQL for update. (But appears broken at the moment for this scenario - see Webservice Callouts within a Select For Update statement are not blocked per Daniel's comment.)

So I suggest say a custom setting field that is queried/set/unset by the Queueable and when the QueryException results the Queueable re-enqueues itself as this means a callout is already in progress.

  • 3
    Interesting side note on FOR UPDATE - there is a known issue around callouts that drops the lock. – Daniel Ballinger May 1 '18 at 0:41
  • Hey Keith, that's a really interesting approach, have you ran this in production before? It sounds a little risky. I can imagine infinite loops or deadlocks possibly occurring as a result. The issue that @DanielBallinger brought up sounds like it may cause issues – Mattisdada May 1 '18 at 6:28
  • @Mattisdada I do not have this in production. Agreed that it looks like its broken at the moment or worse still perhaps never worked. I suppose the only good news is that "known issues" generally do get fixed so keep a watch on this issue. – Keith C May 1 '18 at 8:01
  • @DanielBallinger Thanks for that link; more like a blocker at the moment for this approach than a side note though... – Keith C May 1 '18 at 8:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.