Test.startTest() in testSetup method

Question

An year ago I asked a question regarding the advantages of using testSetUp annotation instead of having a method for common data and calling it in every test method. My main issue at the time with using testSetup method was if it issues 30 queries, every test method starts its life with only 70 queried remaining and on top of that the data created in it is unavailable in test methods. We have to explicitly query again.

I happened to notice that if I wrap the whole testSetup code between Test.startTest() and Test.stopTest(), the soql queries are not counted against the individual test methods and we can use Test.startTest() and Test.stopTest() again in the test methods.

Question:

Since this behavior is not documented, are there any negative effects from this approach?

Example:

@isTest
private class AvinashTest {

    @testSetup static void commonData(){
        Test.startTest();

        Account a = (Account) TestFactory.createSObject(new Account(), true);
        Contact c = (Contact) TestFactory.createSObject(new Contact(), true);
        Opportunity o = (Opportunity) TestFactory.createSObject(new Opportunity(AccountId=a.Id,StageName='Closed Lost', Channel_Type__c = 'Direct'), true);
        Product2 prod1 = (Product2) TestFactory.createSObject(new Product2(), true);
        PricebookEntry pbe = (PricebookEntry) TestFactory.createSObject(new PricebookEntry(Product2Id = prod1.Id), true);
        Asset ast = (Asset) TestFactory.createSObject(new Asset( Product2Id = prod1.Id, AccountId = a.Id), true);

        System.Debug('**commonData**'+Limits.getQueries()); //Returns 16

        Test.stopTest();
    }

    @isTest static void demoMethodOne() {

        System.Debug('**demoMethodOne**'+Limits.getQueries()); //Returns 0

        Test.startTest();

        Opportunity o = [SELECT Id FROM Opportunity LIMIT 1];

        Test.stopTest();

    }

    @isTest static void demoMethodTwo() {

        System.Debug('**demoMethodTwo**'+Limits.getQueries()); //Returns 0

        Test.startTest();

        Opportunity o = [SELECT Id FROM Opportunity LIMIT 1];

        Test.stopTest();

    }

}

Answer 1

This is a method you would have known about if you'd seen my previous answer on the subject. We use this internally because without using this technique, we continually run into governor limits on some of our larger tests. Several developers, including myself, have independently found this feature, and it's allowed us to continue building our internal applications without more aggressive optimization of our code. So far, we've observed no negative consequences from this technique, but have enjoyed more reliable tests and the ability to insert a lot more standardized test data than was previously possible.

Answer 2

As of five days ago, I have a test class that fails because it uses Test.startTest() in a @testSetup method, and the other methods then test classes that make callouts. Previously this test class was working fine for over a year.

When Test.starttest() is used it is throwing the error "System.CalloutException: You have uncommitted work pending. Please commit or rollback before calling out." When Test.starttest() is removed from the @TestSetup method, the error goes away.

Apparently they messed something up so that DML used within Test.startTest() in a @testSetup method is somehow being treated as the same transaction as the callouts which happen in the actual test methods, which of course is totally wrong. It is treating it as if DML statements are being made before a callout in a single transaction, which is not allowed. Definitely seems like a Salesforce bug, and I have a ticket in to them.