Moving to parallel testing has made one of our builds run 4x faster but the test result files returned from:

sfdx force:apex:test:report

after polling via:

sfdx force:apex:test:run

vary on every run. This is illustrated by the last section of this Jenkins trend chart:

Jenkins trend chart

or this individual Jenkins test summary (showing the difference from the last build):

Jenkins test summary

Is losing consistent test results the price you pay for parallel testing, or is there a way to get a consistent count?


The actual code that is running is this https://github.com/claimvantage/sfdx-jenkins-shared-library/blob/master/vars/runApexTests.groovy


Adding the query that Daniel Ballinger suggests in our build confirms that the variation is that entire classes don't get tested e.g.:

BenefitClaimedPaymentsControllerTest      Completed  (4/4)
BenefitReductionsTest                     Failed     Could not run tests on class 01p1k000000EEbF because: connection was cancelled here
BenefitReductionsWithLookbackTest         Completed  (22/22)
BenefitReductionGraphsControllerTest      Completed  (1/1)
  • Are you using the -w option? If so, it may be an issue with getting the last CometD message and/or outright dropping messages. Can you provide the full CLI command (sans any sensitive details)? May 23, 2019 at 22:11
  • Hi @DanielBallinger, Added a link to the actual code that runs and made the start of the question a bit clearer - sorry it was misleading as written. The results are grabbed in final single sfdx force:apex:test:report.
    – Keith C
    May 24, 2019 at 7:21
  • I encountered this same issue recently while trying to run parallel tests in continuous integration (this question was extremely helpful!). I've written logic to catch and retry this class of silent failures in a forthcoming version of CumulusCI.
    – David Reed
    Nov 15, 2019 at 18:44
  • Hi @DavidReed Nice! FYI a colleague in QA was looking at CumulusCI the other day and liked what they saw.
    – Keith C
    Nov 15, 2019 at 18:51
  • @KeithC Yes, he reached out to get connected!
    – David Reed
    Nov 15, 2019 at 18:57

2 Answers 2


I just completed a parallel test run. Note that all these tests currently pass when run individually or in the synchronous test mode. After the parallel run I then ran the same SOQL query you are:

select Status, MethodsEnqueued, MethodsCompleted, MethodsFailed 
from ApexTestRunResult 
where AsyncApexJobId = '7071W00006mnz7p'

enter image description here

That came back with:

  • Status: Completed
  • MethodsEnqueued: 811
  • MethodsCompleted: 797
  • MethodsFailed: 13

This aligns somewhat with your findings. E.g. Why doesn't MethodsCompleted + MethodsFailed equal MethodsEnqueued?

A query against ApexTestResult for the Apex Job showed only 797 records.

Id, TestTimestamp, Outcome, ApexClassId, MethodName, AsyncApexJobId, QueueItemId, RunTime 
from ApexTestResult 
where AsyncApexJobId = '7071W00006mnz7p'

And that included the 13 marked as failures. So the first thing we can conclude is that ApexTestRunResult.MethodsFailed is actually included in the ApexTestRunResult.MethodsCompleted count and there are 14 test methods unaccounted for.

I think the important details are in ApexTestQueueItem with the Status Failed.

Id,ApexClassId, ApexClass.Name,Status,ExtendedStatus,ParentJobId,TestRunResultId,ShouldSkipCodeCoverage
from ApexTestQueueItem 
where ParentJobId = '7071W00006mnz7p'
order by Status desc

There are three test classes that outright failed to run. They failed with the statuses:

  • Status: Failed
  • ExtendedStatus: "Could not run tests on class 01p40000000HDZA because: connection was cancelled here"

I checked the number of test methods in the classes that failed. They were 2, 1, and 11 respectively. So that explains my missing 14 test methods. Those test methods where neither Completed or Failed. They just didn't run.

For my scenario - There is a data isolation issue when running the tests in parallel that manifests as a "connection was cancelled here" error. As a result, the test methods aren't run and no results come back for them. As it is a concurrency error the number of failures will vary from test run to test run. I've gone into some of the painful details of this in Speeding up Salesforce unit testing performance.

Your scenario is likely similar. You could try adding some sort of guard statement that ApexTestRunResult.ClassesEnqueued equals ApexTestRunResult.ClassesCompleted to make sure entire classes aren't being excluded from the results.

With this being a concurrency issue it would also be useful to control the number of test cases that run at any one time. The current limit of 30 is way to high when there are issues siloing test data. Please consider voting for the idea - Control the degree of parallelism when running apex tests in parallel

I'll need to gather some more data to prove it, but my suspicion is that is caused by using the Steaming API to monitor and report on the results.

It will either be a timing issue between when the test job finishes and when the last CometD message is received or possibly an outright failure to receive a message. Either way, the SFDX CLI command is missing a number of ApexTestResult (07M) records.

I'd take your test ApexJobIds and directly query for the corresponding ApexTestResult records. E.g.

from ApexTestResult 
where AsyncApexJobId = '7071W000052hsrXQAQ'

The other possibility is that the force:apex:test:run command isn't queuing up all the existing test classes. If it is somehow not creating the ApexTestQueueItem records for a subset of Apex classes then those tests would never run. I'd confirm that the expected number of records are created for the ApexJobId.

select Id,ApexClassId,Status,ExtendedStatus,ParentJobId,TestRunResultId,ShouldSkipCodeCoverage 
from ApexTestQueueItem 
where ParentJobId = '7071W000052hsrXQAQ'

There should be one record per Apex test class.

  • 1
    Thanks Daniel; doing some queries after the build has finished sounds like the next step. If the full data is there - I suspect it isn't - the direct querying sounds like a good work-round though would require some formatting to pass on to Jenkins.
    – Keith C
    May 24, 2019 at 7:17
  • Thanks for your work looking into this (I haven't got back to this). It sounds like I should add the query that includes ExtendedStatus to the build automation to at least surface the problem in the build log? Would be good to get some sense of the failure pattern too. But for now, just something we have to live with.
    – Keith C
    May 28, 2019 at 7:32
  • I've added a PPS to the question that confirms that it is classes that get missed. Not good in a CI environment.
    – Keith C
    May 28, 2019 at 19:29
  • We have kind of the same issue, even using the serial model for apex test runs. Mar 2, 2022 at 13:53

The underlying bug that appears to be the culprit in this issue has been published as Known Issue W-7834236.

Some Apex unit tests fail in a way that only shows in specific test UIs. The ApexTestQueueItem records for their parent classes will show Status = Failed and ExtendedStatus = "Could not run tests on class <01p class id> because: connection was cancelled here".

For these tests, ApexTestResult records will usually not be created, although some may exist. It's these tests that show as in progress forever in tools like Salesforce DX and the Developer Console, which are looking for ApexTestResult records. One consequence is that many testing tools will report a different number of unit tests completed from run to run, due to monitoring ApexTestResult records.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .