ApexTestRunResult's status is set to complete and all tests pass, but the enqueued and completed count do not match

Question

It is possible to programatically run tests in the platform by creating ApexTestQueueItem records. They reference classes by their Id, and you can even optionally specify if you want the test to calculate coverage or not.

When inserting those records you'll automatically get an AsyncApexJob Id assigned to the ParentJobId on the previously mentioned queue record. We can use this Id to query for results, like:

SELECT Id,
       ClassesEnqueued,
       ClassesCompleted,
       EndTime,
       MethodsEnqueued,
       MethodsCompleted,
       MethodsFailed,
       Status,
       TestTime
FROM ApexTestRunResult
WHERE AsyncApexJobId = '7071I00002EvjEvQAJ'

Now, according to the documentation, this record holds the information for our tests alright. The amount of classes enqueued, passed, etc.

---

My problem is: some seconds after putting some test classes to run, this record is marked as Complete (on the Status field) by the platform, but the classes and methods values do not match what has been specified on the test.

If I specify that I want, for example, to run 12 tests, then when the record is updated to Complete I'll get:

ClassesEnqueued: 12
ClassesCompleted: 8

The same issue applies for the Method fields.

So 4 classes and some methods are missing, but the test result record is still set to Complete.

Am I missing something here?

---

Edit

One of those 12 classes was not a test, thank you @sfdcfox for pointing that out. But all other 11 classes are.

When querying the ApexTestRunResult:

SELECT Id, ClassesEnqueued, ClassesCompleted, EndTime, MethodsEnqueued, MethodsCompleted, MethodsFailed, Status, TestTime FROM ApexTestRunResult WHERE AsyncApexJobId = '7071I00002Evpx8QAB'

...I get a result like this:

Id: 05m1I000000XsLjQAK
ClassesEnqueued: 11
ClassesCompleted: 8
Status: Completed
MethodsEnqueued: 35
MethodsCompleted: 32
MethodsFailed: 0
EndTime: 2018-08-17T03:52:55.000Z
TestTime: 2417

And when querying the ApexTestResult records related to the same job:

SELECT Id, ApexClass.Name, MethodName, ApexTestRunResult.Status, Outcome FROM ApexTestResult WHERE AsyncApexJobId = '7071I00002Evpx8QAB'

I get 35 rows of records, all passing. Which also doesn't match the result from ApexTestRunResult (35 vs 32).

Querying the specific tests and retrieving the summary from the relationship also shows the mismatch between the enqueued and completed tests (although all are passing).

Answer 1

From the ApexTestRunResult docs:

MethodsCompleted The total number of methods completed during the test run. This value is updated after each class is run.

I'm seeing this discrepancy as well when I look back through previous ApexTestRunResult records. In my cases I had parallel testing enabled.

I'm assuming there is a race condition around multiple tests trying to update MethodsCompleted with parallel testing enabled. E.g. several finish around the same time, but the MethodsCompleted only gets the value from one of several tests that were running just before it completed.

I've always switched over to the ApexTestResult once the job appeared to be completed to show the actual results.