Unexpected "CalloutException" only when validating changeset

Question

I'm trying to deploy some code. I've tested everything in sandbox, verified unit tests pass and coverage is good. Created a changeset, uploaded to Prod. When I validate the changeset, I get 1 failure, with the following message:

Changeset "xxx" will not validate. One unit test fails with "An unexpected error occurred when trying to make a callout to the Box API. You have uncommitted work pending. Please commit or rollback before calling out

The exception is one I'm familiar with. You can't make an HTTP callout if you have DML pending (since the callout may fail or hang). However:

1. The code was specifically designed to avoid this issue. I've hand checked the code to ensure there's no DML occurring before the callout.
2. The same test passes in sandbox. I've double-checked that all relevant code is in the changeset.

The only thing I can think is that some of the unit test setup DML is "leaking" past the Test.startTest() call? Could this be a Spring 17 issue? I've opened a case, but client has Standard support; I expect it may be closed as "development issue". Has anyone else seen this variance between sandbox & validation?

Edit 1-Mar-17: At @DanielBallinger's suggestion I was able to capture a log of the failing test execution during the validation attempt. I see 3 pairs of DML_BEGIN/DML_END, and they are the three inserts I'm doing as part of my test setup; they all occur before Test.startTest(). I also scanned the log from test.startTest() to the error, looking for anything else (WF, etc) that could cause this; I see nothing. Further, none of the objects involved have any WF or Processes defined. It certainly looking like a platform issue.

And after 20 minutes on the phone with support, the case was closed because it's a "developer issue" and the client has Standard support.

Answer 1

Per two known issues, W-2979657 and W-2839839:

The workaround is to set the API version of the class which makes the callout to v.33 or lower. A class with an API version of 34 or higher will inappropriately throw the callout exception when the unit test executes.

This workaround allows my minimalist reproduction code to pass the test as well as the production test code which originally surfaced the problem.

I experienced this exception being thrown when running tests on some instances, but not all of them. I ran the same reproduction unit test in as many different instances as I had easy access to. Test results per instance:

• NA11: Pass (Tested in 1 org)
• NA22: Fail (Tested in 6 orgs)
• NA30: Fail (Tested in 5 orgs)
• NA34: Pass (Tested in 1 org)
• NA35: Pass (Tested in 2 orgs)

Reproduction Code

Static Resource: Mock_API_Response

{
   "total":1,
   "nextOffset":1,
   "data":[
      {
         "company_id":"a"
         , "product_name":"product"
         , "updated":"2017-01-01 02:51:01"
      }
   ]
}

Batch Class

public without sharing class Callout_Batch implements Database.Batchable<Integer>, Database.stateful, Database.allowsCallouts {

    public Callout_Batch() {

    }

    public List<Integer> start(Database.BatchableContext BC) {

        // execute a callout
        String url = 'https://api.example.com/v2/scores?limit=1000&offset=0&api_key=1234';

        HttpRequest request = new HttpRequest();
        request.setTimeout(60 * 1000);
        request.setEndpoint(url);
        request.setMethod('GET');

        HttpResponse httpResponse = (new Http()).send(request);

        String response = httpResponse.getBody();

        return new List<Integer>();
    }

    public void execute(Database.BatchableContext BC, List<Integer> scope) {

    }

    public void finish(Database.BatchableContext BC) {  

    }
}

Callout Batch Test

@isTest
private class Callout_Batch_Test {

    static testmethod void testCallout_Batch() {

         // get the standard user profile id 
        Id profileId = [SELECT Id FROM Profile WHERE Name = 'Standard User'].Id;

        String email = 'unit.test.user+345@' + UserInfo.getOrganizationId() + '.test.com';
        User u = new User(
            FirstName           = 'Test',
            LastName            = 'User ',
            Email               = email,
            Username            = email,
            Alias               = 'tuser345',
            Phone               = '503-555-1212',
            ProfileId           = profileId,
            TimeZoneSidKey      = 'America/Los_Angeles',
            LocaleSidKey        = 'en_US',
            EmailEncodingKey    = 'ISO-8859-1',
            LanguageLocaleKey   = 'en_US'
        );

        insert u;

        system.runas(u) {

            MultiStaticResourceCalloutMock multiMock = new MultiStaticResourceCalloutMock();
            multiMock.setHeader('Content-Type', 'application/json');
            multiMock.setStatusCode(200);
            multiMock.setStaticResource('https://api.example.com/v2/scores/?limit=1000&offset=0&api_key=1234', 'Mock_API_Response');

            Test.startTest();

            Test.setMock(HttpCalloutMock.class, multiMock);            

            Database.executeBatch(new Callout_Batch(), 1);

            Test.stopTest();
        }
    }
}

Exception Thrown

Class: Callout_Batch_Test
Method Name: testCallout_Batch
Pass/Fail: Fail
Error Message: System.CalloutException: You have uncommitted work pending. Please commit or rollback before calling out