ApexTestResultLimits in Tooling API work for both synchronous and asynchronous runs?

Question

The documentation at ApexTestResultLimits states that:

The ApexTestResultLimits object is populated for each test method execution, and it captures the limits used between the Test.startTest() and Test.stopTest() methods. If startTest() and stopTest() aren’t called, limits usage is not captured. Note the following: The associated test method must be run asynchronously. Limits for asynchronous Apex operations (batch, scheduled, future, and queueable) that are called within test methods are not captured. Limits are captured only for the default namespace.

And then on the very same page ApexTestResultLimits object has the field LimitContext which is

Indicates whether the test run was synchronous or asynchronous.

So ApexTestResultLimits gets generated for both synchronous and asynchronous runs?

Answer 1

Through manual testing, I believe the ApexTestResultLimits.LimitContext field is not displaying what you would logically assume it is. It doesn't appear to be related to what the documentation claims. Or it simply isn't working at all.

I did a baseline SOQL query to measure the current state of the records:

Select Id, ApexTestResultId, CreatedDate, LimitContext from ApexTestResultLimits order by CreatedDate desc

All the existing records came back with LimitContext = 'SYNC'. Which was odd, as I typically run tests async. However, I'd just created a package version in that org, so it might have been related to that.

I then selected a subset of test cases in the org and ran them asynchronously. They definitely ran async, as I saw multiple in the running state at the same time. Afterwards, the number of ApexTestResultLimits records returned increased. Again, they were all LimitContext = 'SYNC'.

The Dev Console Test > Always Run Asynchronously setting didn't seem to make a difference to the LimitContext. Having that setting off and only running a single test method ran the test synchronously. I know this, as if you try and run another test while the first is still running you get a "Failed to run tests synchronously" error.

When they say that ApexTestResultLimits don't get generated for synchronous tests, that is correct (as at Spring 20 v48.0). If you run a single test case in the dev console you will briefly see the following appear:

Afterwards, there will be no change to the total number of ApexTestResultLimits records.

---

I suspect ApexTestResultLimits.LimitContext isn't aimed at recording the nature of how the test was run, but rather the "limits context" that was being applied. If you look at Execution Governors and Limits you'll note the split in several places between "Synchronous Limit" and "Asynchronous Limit". That doesn't apply to the nature of the test run (are the tests running in parallel) but instead how the limits are being enforced in the testing transaction.

And just to further confuse matters... I don't think it is currently possible to have any other limits context in Apex other than the synchronous limits. E.g. there is currently no way to access the 12 MB heap limit that aynsc would give you in a testing context.