Code Coverage Calculation - Seems to be including code in test methods

Question

According to the documentation:

Test classes (classes that are annotated with @isTest) are excluded from the code coverage calculation. This exclusion applies to all test classes regardless of what they contain—test methods or utility methods used for testing

However I am currently finding that to be false (not the case).

As you can see in the included image (not all classes included in image) all my non test classes have coverage of 100% with a few stragglers dus to overzealous catch blocks. In total, for non test methods I have a grand total of 27 lines uncovered.

The coverage indicates that 324 lines are uncovered.

In my classes annotated with @isTest there are 297 lines across all @isTest annotated classes where the methods are not marked as test methods (Utility methods).

I have checked the code coverage from the Dev console, IntelliJ, the Setup UI and they all say the same thing 84.xx% total coverage.

I have cleared the test history, recompiled all classes, all the usual stuff.

Is the documentation wrong?

An example method in an @isTest annotated class

private static PageReference setTestPageReference(){
    PageReference pr = Page.Example_Page;
    pr.getHeaders().put('Host','https://login.salesforce.com');

    test.setCurrentPageReference(pr);
    return pr;
}

The class that contains the above method has 0/5 line covered. When I add the testmethod to the method definition, the class changes to have 0 uncovered lines and the code coverage for the entire org goes up by .01%

Basically it seems that code in a @isTest annotated class IS being included in test coverage which is a problem since I have a large test Utility class annotated with @isTest and none of the methods are marked as test methods.

So the Questions:

1. Am I the only one observing this behavior?
2. Should I worry about it?
3. Any harm in adding the testmethod designation to the utility methods? (It feels wrong)

Answer 1

I've had similar conversations about this before with salesforce, because this was actually causing us issues in deployment. What we came away with was as follows.

The Documentation is Correct

According to Support, any class that has @isTest specified should be completely exempt from code coverage. It should display as 0/0 coverage.

The System is Wrong

They confirmed with R&D that someone internally has already logged a bug regarding this. There is a Known Issue that's tracking this bug.

We Should Test (For Now)

In our case, we had nearly 500 lines of code that were showing no coverage, and it was actually causing deployments to fail with a net total of 74% coverage. If you're getting deployment errors, then you need to unit test your unit tests. Otherwise, don't worry about it.

Answer 2

Am I the only one observing this behavior?

You are not. There is at least one relevant Known Issue: Non-test methods in a test class are counted as part of Apex code Coverage. It seems that if you create a class, then add the @IsTest definition after the fact, that can contribute to the issue.

Should I not worry about it?

I believe this bug does not affect deployments to production, which is generally the most critical time to have accurate code coverage numbers. I believe it would be unlikely to follow the reproduction steps listed in the Known Issue I linked to, since you would have to deploy a class as live and then convert it to the @IsTest annotation.

Any harm in adding the testmethod designation to the utility methods?

Security scanners will certainly frown upon test methods with no assertions. I'd avoid it.