Test cases failing with "Could not run tests on class 01pXXX because: connection was cancelled here"

Question

When running all the test classes in my dev org I'm getting intermittent failures on some of them with:

Could not run tests on class 01p400000000001 because: connection was cancelled here

The failed records can be found using SOQL:

Select Id,ApexClassId,Status,ExtendedStatus,ParentJobId,TestRunResultId 
from ApexTestQueueItem 
where ParentJobId = '7071W00004wd9gE' and Status = 'Failed'    

If I run just those test classes again the problem doesn't occur.

The number of classes affected ranges between 2 and 8 per run out of a total of 72 classes.

Checking "Disable Parallel Apex Testing" avoids the problem, but also makes the tests take considerably longer to execute.

I've considered using the @IsTest(isParallel=true) annotation to isolate the problem classes. However, there isn't a @IsTest(isParallel=false) option, so I'd need to modify the majority of my test classes. Also, the classes that fail vary from run to run.

Also raised as support case #19013567

---

Update #1 from support:

The root cause of this issue?
Answer: - Parallel test execution is the root cause of this issue.

Solution?
Answer: - Disable Parallel test execution.

Question: - Why do you need to Disable Parallel Apex Testing?
Answer: - As per the salesforce document, Tests that are started from the Salesforce user interface (including the Developer Console) run in parallel. Parallel test execution can speed up test run time. Sometimes, parallel test execution results in data contention issues, and you can turn off parallel execution in those cases.

---

Update #2

I went to great effort to isolate all the DML operations on a custom hierarchy setting in test cases using a mock. And the results was... a marginal improvement in performance and continued intermittent failures on some random test classes.

I went back to the drawing board and had a closer look at the debug log.

There is 20+ seconds on a DML_INSERT for 6 records. This particular insert happens in the test setup for nearly all my test classes.

I added some elapsed time measurement to that method and an assertion that killed it any time it took longer than a second. 35 of my 72 classes failed on it.

I'll keep trying to isolate the cause around this operation.