Why do running tests in parallel makes APEX CPU Timeout possible likely?

Question

We have designed tests to work on collection of records so we can ensure Governor Limits are not violated. These tests have a surprising behaviour:

• They always succeed on Scratch orgs
• They fail on Sandboxes only when all the tests are run in parallel with an APEX CPU Timeout Exceeded

Interesting enough, the exception is not raised inside the Test.startTest() / Test.stopTest() region but in the preceding code that sets up a complex requirement graph before the test is started.

This fact raises a number of questions:

1. Is this behaviour well known and should be taken into consideration when designing tests?
2. Should @testSetup always be used when possible, since a new set of governor limits is available inside the method?
3. When it is not possible to use @testSetup because different test methods inside the same class require different set-ups, is it better to
   1. Give up the idea of having multiple test scenarios for the same feature in the same class, and start breaking it down
   2. Use Test.startTest() and Test.stopTest() not to wrap only the "effectful" code whose outcome you wish to test, but the whole testmethod body

Answer 1

I've been struggling/battling with this since the Spring '13 release.

It tends to manifest in one of two ways when parallel testing is enabled:

1. As you found, Tests fail with APEX CPU Timeout Exceeded messages, or
2. Entire tests classes fail with Could not run tests on class 01p400000000001 because: connection was cancelled here messages.

The sad truth is that data created and used in Apex test cases isn't fully siloed (as at Summer '19). Not so much in that actual data will seep from one tests to another. But more that the indexes and unique constraints that are enforced in the database cause deadlocks between test cases.

There is even a section on this in the Developer Docs now under Best Practices for Parallel Test Execution: (my emphasis)

Tests that are started from the Salesforce user interface (including the Developer Console) run in parallel. Parallel test execution can speed up test run time. Sometimes, parallel test execution results in data contention issues, and you can turn off parallel execution in those cases. In particular, data contention issues and UNABLE_TO_LOCK_ROW errors can occur in the following cases:

• When tests update the same records at the same time. Updating the same records typically occurs when tests don’t create their own data and turn off data isolation to access the organization’s data.
• When a deadlock occurs in tests that are running in parallel and that try to create records with duplicate index field values. A deadlock occurs when two running tests are waiting for each other to roll back data. Such a wait can happen if two tests insert records with the same unique index field values in different orders.

What can you do?

1. Ensure that you aren't using @IsTest(SeeAllData=true) anywhere in your tests.

2. Turn off parallel test execution and take the hit in how long it takes to run the tests synchronously.

3. Attempt to completely bypass the DML operations via mocking. How practical this is will depend on the complexity of your tests.

4. If you detect failures during test execution, run those failing tests again until the results stabilize. So far this is the fastest way I've found to handle the non-deterministic nature of running parallel tests.

   This is more or less what I've been doing day to day. Since the SFDX CLI didn't exist when I needed it I did this directly against the Salesforce APIs. It's available in the FuseIT SFDC Explorer. (A free tool released by my current employer). On the "Apex Tests Queued" tab keep running the tests with "Run Selected Tests". It will only run the tests that failed in the last run. Gradually you will end up with just the tests that are actually failing.

5. Vote for the idea Control the degree of parallelism when running apex tests in parallel. Right now Salesforce will try and run up to 30 tests in parallel. It's kind of crazy when combined with the potential for deadlocks. A much lower degree of parallelism would still allow for faster overall test execution while reducing the chances of deadlocks.

6. Vote for the idea Parallel Tests Option (isParallel) on the @IsTest Annotation to exclude tests. Some tests are just never going to play well with others. But that doesn't mean we should give up on running the other tests in parallel.

Answer 2

At work we have been quite successful adopting tooling to work around failures with parallel testing on scratch and other orgs. The tooling first attempts to run all tests in parallel and then examines the failures to determine which to attempt to re-run sequentially before producing a report of anything still failing, similar to how you might do this manually.

The tooling is just a fairly simple custom sfdx cli that calls other sfdx cli commands to perform the test runs. Using this has enabled us to cut test suites that would normally take many hours sequentially down to 10-20 minutes. The failure rates when running in parallel are typically fairly small (10s of failures) but are obviously quite unpredictable. A small side benefit is that we can also handle some of the other common sfdx cli test command failures and so reduce some of the other random failures in the CI pipe.

In back to back testing I did not find using isParallel had any noticeable impact but reducing batch testing helped further improve execution time considerably, by about 50% in the case I tested.

Update: I written a short post with some more details about how this works and that links to an implementation you can have a play with, see https://nawforce.blog/2019/06/09/parallel-unit-testing-via-sfdx-cli/.