7

We have designed tests to work on collection of records so we can ensure Governor Limits are not violated. These tests have a surprising behaviour:

  • They always succeed on Scratch orgs
  • They fail on Sandboxes only when all the tests are run in parallel with an APEX CPU Timeout Exceeded

Interesting enough, the exception is not raised inside the Test.startTest() / Test.stopTest() region but in the preceding code that sets up a complex requirement graph before the test is started.

This fact raises a number of questions:

  1. Is this behaviour well known and should be taken into consideration when designing tests?
  2. Should @testSetup always be used when possible, since a new set of governor limits is available inside the method?
  3. When it is not possible to use @testSetup because different test methods inside the same class require different set-ups, is it better to
    1. Give up the idea of having multiple test scenarios for the same feature in the same class, and start breaking it down
    2. Use Test.startTest() and Test.stopTest() not to wrap only the "effectful" code whose outcome you wish to test, but the whole testmethod body
  • Each test method has its own limits and is a separate transaction so that is not your cause. I have and old idea on this where complex setup is required to do in a test what would be several individual transactions in real life. For example creating a grandchild that requires products, accounts, opp, etc before it can be inserted. Testsetup mostly solves this though. One thing you can do is play with where you first use test.starttest. Putting it sooner in the method will break up the transaction limits during the test – Eric Jun 2 at 22:11
  • so if you use: @testSetup + Test.startTest()/stopTest() you get a total of three separate limits? – Edmondo1984 Jun 3 at 7:40
  • 1
    It is unclear how the limits apply to testSetup (other than once during the whole class) as I do not believe it is documented, at least a quick review of the documentation did not reveal anything. You will have to benchmark it from your own logs. But you can move the startTest to earlier in your method to try and play with the limits. Also look into the possibility of optimizing your code as well which may help alleviate your issue – Eric Jun 3 at 7:46
  • My code is highly optimized, and the tests run on a batch of 100 records always work in a scratch org and in sandbox when run in stand-alone. It seems that excecuting 500 tests overload the CPU, so the limits are separated by the CPU is not? – Edmondo1984 Jun 3 at 9:27
  • When you say "They always succeed on Scratch orgs", you mean for both the non-parallel and parallel cases? We see problems with parallel testing in scratch orgs (connection was cancelled here errors) that we don't see in other types of orgs but have not hit unexpected CPU limit errors. But we did have to add some retry code that spins and so burns CPU: do you have any code of that nature already? – Keith C Jun 3 at 12:36
5

I've been struggling/battling with this since the Spring '13 release.

It tends to manifest in one of two ways when parallel testing is enabled:

  1. As you found, Tests fail with APEX CPU Timeout Exceeded messages, or
  2. Entire tests classes fail with Could not run tests on class 01p400000000001 because: connection was cancelled here messages.

The sad truth is that data created and used in Apex test cases isn't fully siloed (as at Summer '19). Not so much in that actual data will seep from one tests to another. But more that the indexes and unique constraints that are enforced in the database cause deadlocks between test cases.

There is even a section on this in the Developer Docs now under Best Practices for Parallel Test Execution: (my emphasis)

Tests that are started from the Salesforce user interface (including the Developer Console) run in parallel. Parallel test execution can speed up test run time. Sometimes, parallel test execution results in data contention issues, and you can turn off parallel execution in those cases. In particular, data contention issues and UNABLE_TO_LOCK_ROW errors can occur in the following cases:

  • When tests update the same records at the same time. Updating the same records typically occurs when tests don’t create their own data and turn off data isolation to access the organization’s data.
  • When a deadlock occurs in tests that are running in parallel and that try to create records with duplicate index field values. A deadlock occurs when two running tests are waiting for each other to roll back data. Such a wait can happen if two tests insert records with the same unique index field values in different orders.

What can you do?

  1. Ensure that you aren't using @IsTest(SeeAllData=true) anywhere in your tests.
  2. Turn off parallel test execution and take the hit in how long it takes to run the tests synchronously.
  3. Attempt to completely bypass the DML operations via mocking. How practical this is will depend on the complexity of your tests.
  4. If you detect failures during test execution, run those failing tests again until the results stabilize. So far this is the fastest way I've found to handle the non-deterministic nature of running parallel tests.

    This is more or less what I've been doing day to day. Since the SFDX CLI didn't exist when I needed it I did this directly against the Salesforce APIs. It's available in the FuseIT SFDC Explorer. (A free tool released by my current employer). On the "Apex Tests Queued" tab keep running the tests with "Run Selected Tests". It will only run the tests that failed in the last run. Gradually you will end up with just the tests that are actually failing.

  5. Vote for the idea Control the degree of parallelism when running apex tests in parallel. Right now Salesforce will try and run up to 30 tests in parallel. It's kind of crazy when combined with the potential for deadlocks. A much lower degree of parallelism would still allow for faster overall test execution while reducing the chances of deadlocks.

  6. Vote for the idea Parallel Tests Option (isParallel) on the @IsTest Annotation to exclude tests. Some tests are just never going to play well with others. But that doesn't mean we should give up on running the other tests in parallel.
2

At work we have been quite successful adopting tooling to work around failures with parallel testing on scratch and other orgs. The tooling first attempts to run all tests in parallel and then examines the failures to determine which to attempt to re-run sequentially before producing a report of anything still failing, similar to how you might do this manually.

The tooling is just a fairly simple custom sfdx cli that calls other sfdx cli commands to perform the test runs. Using this has enabled us to cut test suites that would normally take many hours sequentially down to 10-20 minutes. The failure rates when running in parallel are typically fairly small (10s of failures) but are obviously quite unpredictable. A small side benefit is that we can also handle some of the other common sfdx cli test command failures and so reduce some of the other random failures in the CI pipe.

In back to back testing I did not find using isParallel had any noticeable impact but reducing batch testing helped further improve execution time considerably, by about 50% in the case I tested.

Update: I written a short post with some more details about how this works and that links to an implementation you can have a play with, see https://nawforce.blog/2019/06/09/parallel-unit-testing-via-sfdx-cli/.

  • That sounds super useful, @KevinJones. Any chance you'd make that tooling open source? – David Reed Jun 6 at 18:26
  • Did you give up any non Sfdx deployment to prod ? – Edmondo1984 Jun 6 at 18:40
  • This is more or less what I've been doing. Since the SFDX CLI didn't exist I did this directly against the Salesforce APIs. I've expanded how I approached it in my answer. I could certainly see the appeal of doing this from the CLI. – Daniel Ballinger Jun 6 at 22:12
  • 1
    I have been working towards open sourcing a smarter variant of this to help with testing during refactoring work, but will try make a basic version available in next few days – Kevin Jones Jun 7 at 5:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.