Spin locks with variable time retry backoffs

Question

Use Case

1. After insert triggers start Queueable
2. Queueable execute() contends for same resource, uses spin lock to wait for resource availability. Many queueables could be running at same time.
3. Exception thrown when number of retries exceeded in spin lock

Spin lock (hat tip to @sfdcfox for this)

public  class LocksServiceImpl implements ILocksService {    
    /**
     * acquire - Obtain a lock that prevents other transactions from executing
     */

    public void acquire(String lockItem) {

        Integer retryCount = 0;
        Exception error;

        while (retryCount < LocksService.MAX_RETRIES) {
            try {
                //  FOR UPDATE will pause this Txn until lock is freed by other transaction
                Mutex__c[] mutexes = MutexesSelector.newInstance().selectForUpdateByItem(new Set<String> {lockItem});
                if (mutexes.isEmpty()) {
                    insert new Mutex__c (Lockable_Item__c = lockItem);
                }
                return;     // this Txn now has the lock on lockItem
            }
            catch (Exception e) {
                if (Util.isTransientException(e)) { // UNABLE_TO_LOCK_ROW
                    error = e;
                    retryCount++;
                }
                else {
                    error = e;
                    break;
                }
            }
        }
        throw new LocksService.MutexException('Unable to obtain a transaction lock on ' + lockItem +
                ' retries:' + retryCount + ' v. maxRetries:' + LocksService.MAX_RETRIES + ' exception:' + Util.showException(error));
    }

}

Challenge

• This works OK most of the time but sometimes the spin exceeds MAX_RETRIES and throws an exception. I'd like to implement some form of backoff logic to delay the spin lock retry rather than immediately repeat the SELECT ..FOR UPDATE
• In fact, the higher the concurrency of queueables, the more likely this occurs
• I looked at some promising backoff algos like exponential and jitter

But how to variable delay in the most effective way and still get best performance?

Options

1. CPU sleep for n secs - not great as limit of 10 CPU secs per Txn and I don't want to use that limit up while spin locking
2. SOQL querying n records (say, the oldest n Contacts - we have 10E6+ of these) - doesn't use CPU but burns SOQL rows that the transaction might need
3. Spin lock more than 10 times (current max retry) - doesn't help if multiple queueables are concurrently running - they all spin lock at the same time and all run out of retries at same time.
4. Delay (using #1 or #2) the execution of System.enqueueJob in the after insert trigger. Maybe; the trigger is lightweight but I have no control over when SFDC starts the queueables so contention between queueables can still happen.
5. Single thread all the queueables through a custom object pattern like Dan Appleman's Async Pattern (this is my fallback as that pattern is implemented at our org).

How can I do something of variable expensiveness (in elapsed time) that consumes the fewest SFDC limits?

Answer 1

I'd just requeue the job if you hit max retries:

try {
  LocksServiceImpl.acquire(lock);
} catch(LocksService.MutexException e) {
  System.enqueueJob(this);
  return;
}

At this point, your job will retry later. There's supposed to be a built-in delay for chained jobs, so this should provide the delay you're looking for.

Answer 2

Although not an answer per se, I wanted to post some performance stats based on a stress test using the spin lock and spin lock fallback strategy as answered by @sfdcfox. This is not a critique of the answer which was good but more of a cautionary tale should one pursue this route. YMMV

Test:

• 50 queueable jobs (System.enqueueJob(..)) each launched in one stress testing transaction.
• Each job enqueues on the same spin lock - that is, they all compete with each other.
• Spin lock does a Select for Update and tries 10 times before abandoning
• Each job does the same amount of work: (deserializes 1000 custom Json and then constructs/inserts 1000 Assets). That is, the work is "substantive".

Result:

• 4 of the 50 queueables completed quickly and the rest were requeued by the spin lock backoff solution to another queueable. SFDC then reschedules these queueables (delaying up to 1 minute) and they (46) compete with each other again. This repeats itself over time until they all complete
• Total elapsed time to get through 50 initial threads each trying to insert 1000 assets = 26 minutes. Three hundred nine (309) queueable retry jobs (!)

Analysis

• YMMV
• A batch job would almost certainly be faster
• Introducing some variable jitter delay between each spin lock retry would probably improve things but as the OP stated, it is not clear what a good delay would be that doesn't consume a scarce SFDC resource