6

It is possible to programatically run tests in the platform by creating ApexTestQueueItem records. They reference classes by their Id, and you can even optionally specify if you want the test to calculate coverage or not.

When inserting those records you'll automatically get an AsyncApexJob Id assigned to the ParentJobId on the previously mentioned queue record. We can use this Id to query for results, like:

SELECT Id,
       ClassesEnqueued,
       ClassesCompleted,
       EndTime,
       MethodsEnqueued,
       MethodsCompleted,
       MethodsFailed,
       Status,
       TestTime
FROM ApexTestRunResult
WHERE AsyncApexJobId = '7071I00002EvjEvQAJ'

Now, according to the documentation, this record holds the information for our tests alright. The amount of classes enqueued, passed, etc.


My problem is: some seconds after putting some test classes to run, this record is marked as Complete (on the Status field) by the platform, but the classes and methods values do not match what has been specified on the test.

If I specify that I want, for example, to run 12 tests, then when the record is updated to Complete I'll get:

ClassesEnqueued: 12
ClassesCompleted: 8

The same issue applies for the Method fields.

So 4 classes and some methods are missing, but the test result record is still set to Complete.

Am I missing something here?


Edit

One of those 12 classes was not a test, thank you @sfdcfox for pointing that out. But all other 11 classes are.

When querying the ApexTestRunResult:

SELECT Id, ClassesEnqueued, ClassesCompleted, EndTime, MethodsEnqueued, MethodsCompleted, MethodsFailed, Status, TestTime FROM ApexTestRunResult WHERE AsyncApexJobId = '7071I00002Evpx8QAB'

...I get a result like this:

Id: 05m1I000000XsLjQAK
ClassesEnqueued: 11
ClassesCompleted: 8
Status: Completed
MethodsEnqueued: 35
MethodsCompleted: 32
MethodsFailed: 0
EndTime: 2018-08-17T03:52:55.000Z
TestTime: 2417

And when querying the ApexTestResult records related to the same job:

SELECT Id, ApexClass.Name, MethodName, ApexTestRunResult.Status, Outcome FROM ApexTestResult WHERE AsyncApexJobId = '7071I00002Evpx8QAB'

I get 35 rows of records, all passing. Which also doesn't match the result from ApexTestRunResult (35 vs 32).

Querying the specific tests and retrieving the summary from the relationship also shows the mismatch between the enqueued and completed tests (although all are passing).

7
  • Are you sure all 12 classes were test classes, and had at least one test method?
    – sfdcfox
    Aug 17, 2018 at 3:27
  • 1
    Can you post more complete query results? You specify many more useful fields in your query such as MethodsEnqueued, MethodsCompleted, etc. Do you have any failing tests? Also, have you tried querying the individual ApexTestResult records?
    – Adrian Larson
    Aug 17, 2018 at 3:27
  • @sfdcfox Now that you mentioned it, I'm sure at least 11 out of those classes are test classes. Aug 17, 2018 at 3:47
  • @AdrianLarson no failing tests in my org. After inserting the queue item records I get all twelve lines on the "Apex Test Execution" page (even though one of them isn't a test). Still, I get only 8 out of 11 actual tests. I haven't tried querying the individual test result records. Will do and edit my post. Aug 17, 2018 at 3:49
  • 1
    That's kind of odd. I wrote code similar to this that ran every sunday night to highlight failing tests, and I never had the values not equal each other in the end.
    – sfdcfox
    Aug 17, 2018 at 4:14

1 Answer 1

4

From the ApexTestRunResult docs:

MethodsCompleted The total number of methods completed during the test run. This value is updated after each class is run.

I'm seeing this discrepancy as well when I look back through previous ApexTestRunResult records. In my cases I had parallel testing enabled.

I'm assuming there is a race condition around multiple tests trying to update MethodsCompleted with parallel testing enabled. E.g. several finish around the same time, but the MethodsCompleted only gets the value from one of several tests that were running just before it completed.

I've always switched over to the ApexTestResult once the job appeared to be completed to show the actual results.

2
  • So a bug / WAD then. Nice detective work on this one....
    – Eric
    Aug 21, 2018 at 0:23
  • Did you test this with parallel testing disabled as well? Aug 21, 2018 at 14:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.