How to fix two categories of UNABLE_TO_LOCK_ROW errors in Apex tests running in parallel?

Question

I'm taking a look at running 300+ test classes in parallel again today. The main error - in about 5% of the test classes - was:

UNABLE_TO_LOCK_ROW, unable to obtain exclusive access to this record

which makes sense given the parallel execution. But the question is how to fix these. There seem to be 2 root causes...

Contact locks

The tests have Contact record inserts that had no Account specified and there is some underlying Account defaulting trigger logic. I am assuming that the lock is on the parent Account so that specifying a separate Account every time will fix. Does that seem right? It appears to work.

Custom setting locks (and User locks)

Some tests modify hierarchical custom settings to exercise some of our configuration options and so update the org-level custom settings at the same time, hence the locking problem. (The current User is also modified in a smll number off cases.) No easy solution comes to mind for these: is there a good pattern for this case? There is this known issue on the subject Document known issue with UNABLE TO LOCK ROW when testing Custom Settings.

Presently trying this ugly code for the custom setting upserts in the tests:

/**
 * When tests are run in parallel, UNABLE_TO_LOCK_ROW errors occur where tests update the same custom setting.
 * This class aims to get around that by retrying over a longish time.
 */
public class Retry {

    public static void upsertOnUnableToLockRow(SObject sob) {

        // Only ever seen one retry needed this should be lots
        Integer n = 20;
        Integer sleepMs = 100;

        Exception lastException;
        for (Integer i = 0; i < n; i++) {
            try {
                upsert sob;
                return;
            } catch (DmlException e) {
                if (!e.getMessage().contains('UNABLE_TO_LOCK_ROW')) {
                    throw e;
                }
                lastException = e;
                sleep(sleepMs);
            }
        }

        throw lastException;
    }

    private static void sleep(Integer ms) {

        Long start = System.currentTimeMillis();
        while (System.currentTimeMillis() < start + ms) {
            // Throw away CPU cycles
        }
    }
}

Answer 1

With the caveat that a mocking solution is the better way to go, below is the class I ended up using to workaround the custom setting UNABLE_TO_LOCK_ROW errors. A bit more info here too https://force201.wordpress.com/2019/05/14/embracing-apex-parallel-testing/.

/**
 * When tests are run in parallel, UNABLE_TO_LOCK_ROW errors occur where tests update the same custom setting.
 * This class aims to get around that by retrying.
 * Can also be applied to ordinary SObjects.
 */
public class Retry {

    // Typically zero or one retry so this should be plenty unless there is some kind of deadlock
    private static final Integer TRIES = 50;
    private static final Integer SLEEP_MS = 100;

    public class RetryException extends Exception {
    }

    public static void upsertOnUnableToLockRow(SObject sob) {

        upsertOnUnableToLockRow(new SObject[] {sob});
    }

    public static void upsertOnUnableToLockRow(SObject[] sobs) {

        if (sobs.size() == 0) return;

        Long start = System.currentTimeMillis();
        Exception lastException;

        for (Integer i = 0; i < TRIES; i++) {
            try {
                SObject[] inserts = new SObject[] {};
                SObject[] updates = new SObject[] {};
                for (SObject sob : sobs) {
                    if (sob.Id != null) updates.add(sob);
                    else inserts.add(sob);
                }
                insert inserts;
                update updates;
                return;
            } catch (DmlException e) {
                // Immediately throw if an unrelated problem
                if (!e.getMessage().contains('UNABLE_TO_LOCK_ROW')) throw e;
                lastException = e;
                sleep(SLEEP_MS);
            }
        }

        Long finish = System.currentTimeMillis();
        throw new RetryException(''
            + 'Retry.upsertOnUnableToLockRow failed first id='
            + sobs[0].Id
            + ' of '
            + sobs.size()
            + ' records after '
            + TRIES
            + ' tries taking '
            + (finish - start)
            + ' ms with exception '
            + lastException
        );
    }

    private static void sleep(Integer ms) {

        Long start = System.currentTimeMillis();
        while (System.currentTimeMillis() < start + ms) {
            // Throw away CPU cycles
        }
    }
}