Test error condition in batch class with callouts

Question

I have an Apex class with a callout (oregonDistricts.apxc) and I have two tests written that test both the success and error conditions. These tests are passing and the class has 100% code coverage. In the test for the error condition, I'm using a separate mock that returns a 'daily api limits exceeded' error and sets a variable in the class (apiLimitExceeded) if that error is returned.

Then I have a batch class (OregonDistrictsBatch.apxc) which is just running a batch of records through the original class (oregonDistricts.apxc).

I'm trying to write tests for the batch class, and I'm only getting 75% coverage and failing to test the error condition, and I can't figure out why. It seems like I'm failing to pass the apiLimitExceeded variable to the batch class so the error condition is never tested. The second test in this class (testErrorCondition) is failing (the System.assert at the end of the test is failing because the apiLimitExceeded variable is not getting set to TRUE).

Can anybody help me figure out why this is not working?

oregonDistricts.apxc:

public class oregonDistricts {

    public Integer congressionalDistrict {get;set;}
    public Integer statehouseDistrict {get;set;}
    public Integer statesenateDistrict {get;set;}
    public String gApiPassthroughErrorMessage {get;set;}
    public Boolean apiLimitExceeded {get;set;}

    public oregonDistricts(String address) {
            system.debug('Address =' + address);    
            Results(address);
    }

public void Results(String address) {

        Http httpProtocol = new Http();
        // Create HTTP request to send.
        HttpRequest request = new HttpRequest();
        // Set the endpoint URL.
         String endpoint = 'http://portal.opeuseiu.org/processes/google_civic_api_passthrough.php'
            +'?secretkey=XXXXX'
            + '&address=' + address;

        request.setEndPoint(endpoint);
        // Set the HTTP verb to GET.
        request.setMethod('GET');
        // Send the HTTP request and get the response.
        // The response is in JSON format.
        HttpResponse response = httpProtocol.send(request);
        System.debug(response.getBody());


        // Parse JSON response.
        JSONParser parser = JSON.createParser(response.getBody());
        congressionalDistrict = null;
        statehouseDistrict = null;
        statesenateDistrict = null;
        gApiPassthroughErrorMessage = '';

        String CD;
        String HD;
        String SD;
        while (parser.nextToken() != null && gApiPassthroughErrorMessage != 'dailyLimitExceeded') {
            if ((parser.getCurrentToken() == JSONToken.FIELD_NAME)) {
                // first check for over API limits error message
                // skip everything else if over limits
                if (parser.getText() == 'reason') {
                    system.debug(parser.getText());
                    parser.nextToken();
                    gApiPassthroughErrorMessage = parser.getText();
                    system.debug(gApiPassthroughErrorMessage);
                    if (gApiPassthroughErrorMessage == 'dailyLimitExceeded') {
                        apiLimitExceeded = true;
                        return;
                    }
                }
                if (parser.getText() == 'congressionaldistrictnum') {
                    // Get the value.
                    parser.nextToken();
                    CD = parser.getText();
                    If (CD == null) { congressionalDistrict = 0; }
                    If (CD != null) { congressionalDistrict = Integer.valueOf(CD); }

                }
             if (parser.getText() == 'statehousedistrictnum') {
                    // Get the value.
                    parser.nextToken();
                    HD = parser.getText();
                 If (HD == null) { statehouseDistrict = 0; }
                 If (HD != null)
                 {  statehouseDistrict = Integer.valueOf(HD); }

                }
               if (parser.getText() == 'statesenatedistrictnum') {
                    // Get the value.
                    parser.nextToken();
                     SD = parser.getText();
                    If (SD == null) { statesenateDistrict = 0; }
                    If (SD != null) { statesenateDistrict = Integer.valueOf(SD); }
                }
            }
        }
        system.debug('Address =' + address + ' Congressional  =' + congressionalDistrict + ' State House  =' + statehouseDistrict + ' State Senate  =' + statesenateDistrict);
    }
}

oregonDistrictsTest.apxc:

@isTest
public class oregonDistrictsTest_2019 {
    @isTest static void testGetCallout() {
        Test.setMock(HttpCalloutMock.class, new OregonDistrictsMock());
        String address = '3892+friar+ct+se%2csalem+or+97302';
        Test.startTest();
        oregonDistricts od = new oregonDistricts(address);
        new oregonDistricts(address).Results(address); 
        Test.stopTest();
        System.assert(od.congressionalDistrict  != null);
        System.assert(od.statehouseDistrict  != null);
        System.assert(od.statesenateDistrict  != null);
    }   

    @isTest static void testGetCalloutError() {
        Test.setMock(HttpCalloutMock.class, new OregonDistrictsErrorMock());
        String address = '3892+friar+ct+se%2csalem+or+97302';
        Test.startTest();
        oregonDistricts od = new oregonDistricts(address);
        new oregonDistricts(address).Results(address); 
        Test.stopTest();
        System.assertEquals(od.gApiPassthroughErrorMessage, 'dailyLimitExceeded');
        System.assertEquals(od.apiLimitExceeded, true);
        System.assertEquals(od.congressionalDistrict, null);
        System.assertEquals(od.statehouseDistrict, null);
        System.assertEquals(od.statesenateDistrict, null);
    }
}

oregonDistrictsBatch.apxc:

global class OregonDistrictsBatch implements Database.Batchable<sObject>, Database.AllowsCallouts, Database.Stateful{

   // instance member to retain state across transactions
    global Integer recordsProcessed = 0;

    // errors stored here for passing to user email in finish method
    Exception[] errors = new Exception[0];
    string stringErrors = '';
    global Boolean apiLimitExceeded = false;

    // query requires oregon street address, active wkr w/PIN or CW record,
    // and null in any one of the 4 fields populated by the callout.
    // for now also including records with 0s in all district fields
    // but can remove that once error handling for over API limits is implemented
    string q = 'SELECT  Id,MailingStreet,MailingCity,MailingState,MailingPostalCode FROM Contact WHERE MailingStreet != NULL AND MailingState=\'OR\' AND ((Active_Worker__c = TRUE AND L503_PIN__c != NULL) OR Has_CareWorks_Record__c = TRUE) AND (District_Update_Date__c = NULL OR Cong_District__c = NULL OR OR_HD__c = NULL OR OR_SD__c = NULL)';

    global Database.QueryLocator start(Database.BatchableContext bc) {
        return Database.getQueryLocator(q);
    }


   global void execute(Database.BatchableContext BC, List<Contact> scope){
       // process each batch of records
       try {
           for ( Contact c  : scope) { 

                String address = EncodingUtil.urlEncode(c.MailingStreet + ',' 
                                                              + c.MailingCity + ' ' 
                                                              + c.MailingState + ' ' 
                                                              + c.MailingPostalCode, 'UTF-8');
                system.Debug('Address =' + address);

               oregonDistricts od = new oregonDistricts(address);

               if (od.apiLimitExceeded != true) {
                    c.Cong_District__c = od.congressionalDistrict;
                    c.OR_HD__c = od.statehouseDistrict;
                    c.OR_SD__c = od.statesenateDistrict;
                    c.District_Update_Date__c = system.now();
               } else {
                   apiLimitExceeded = true;
                   stringErrors = 'API limit exceeded';
                   return;
               }

                recordsProcessed = recordsProcessed + 1;

            }
                update scope;
       } catch (Exception e) {
           errors.add(e);
       }

    }


   global void finish(Database.BatchableContext BC){
       System.debug(recordsProcessed + ' records were processed.');

   // Get the ID of the AsyncApexJob representing this batch job
   // from Database.BatchableContext.
   // Query the AsyncApexJob object to retrieve the current job's information.
   AsyncApexJob a = [SELECT Id, Status, NumberOfErrors, JobItemsProcessed,
      TotalJobItems, CreatedBy.Email
      FROM AsyncApexJob WHERE Id =
      :BC.getJobId()];
   // Send an email to the Apex job's submitter with list of errors, if any.
   String msg='';
   for (integer i =0; i<errors.size();i++) {
        msg +='\n' +'Error: ' + errors[i];
        }
   if(!errors.isEmpty() || stringErrors != '') {
            Messaging.SingleEmailMessage mail = new Messaging.SingleEmailMessage();
            mail.setSubject('Errors occurred during batch process.');
            String[] toAddresses = new String[] {a.CreatedBy.Email};
            mail.setToAddresses(toAddresses);
            mail.setPlainTextBody(msg);
            Messaging.sendEmail(new Messaging.Email[] { mail });
        }
       // Send an email to the Apex job's submitter notifying of job completion.
   Messaging.SingleEmailMessage mail = new Messaging.SingleEmailMessage();
   String[] toAddresses = new String[] {a.CreatedBy.Email};
   mail.setToAddresses(toAddresses);
   mail.setSubject('Apex Sharing Recalculation ' + a.Status);
   mail.setPlainTextBody
   ('The batch OregonDistricts job processed ' + recordsProcessed + ' in '  + a.TotalJobItems +
   ' batches with '+ a.NumberOfErrors + ' failures.');
   Messaging.sendEmail(new Messaging.SingleEmailMessage[] { mail });

    }

}

OregonDistrictsBatchTest.apxc:

@isTest
private class OregonDistrictsBatchTest {
    static testmethod void test() { 
        List<Contact> contacts = new List<Contact>();
        // insert 10 contacts with unknown employer and test address; these will be guaranteed to meet query criteria for queriedContacts List
        for (Integer i=0;i<10;i++) {
            contacts.add(new Contact(
                firstname='first' + i, 
                lastname='last' + i, 
                accountId='0016100001RyWhAAAV',
                Has_CareWorks_Record__c = TRUE,
                MailingStreet='3892 friar ct se',
                MailingCity='Salem',
                MailingState='OR',
                MailingPostalCode='97302'));
        }
        insert contacts;
        List<Contact> queriedContacts = [SELECT  Id,MailingStreet,MailingCity,MailingState,MailingPostalCode FROM  Contact WHERE MailingStreet != NULL AND (District_Update_Date__c = null OR Cong_District__c  = null OR OR_HD__c = null OR OR_SD__c = null)];
        System.debug(queriedContacts);
        System.assert(queriedContacts.size() >0);
        Test.setMock(HttpCalloutMock.class, new OregonDistrictsMock());
        Test.startTest();
        OregonDistrictsBatch odb = new OregonDistrictsBatch();
        Database.executeBatch(odb,10);
        Test.stopTest();
        // after the testing stops, assert records were updated properly
        // select first 10 records with a district update date populated
        List<Contact> listContact = [SELECT id, District_Update_Date__c, Cong_District__c, OR_HD__c, OR_SD__c FROM Contact WHERE District_Update_Date__c != NULL LIMIT 10];
        // assert that 10 records were found
        System.assertEquals(listContact.size(),10);   
        // assert that each of the targeted fields was populated in each of the 10 test records                                                                                                                                                            
        for(Contact contact : listContact){
          System.assert(contact.Cong_District__c  != null);
          System.assert(contact.OR_HD__c  != null);
          System.assert(contact.OR_SD__c  != null);
        }
    }
    static testmethod void testErrorCondition() { 
        Test.setMock(HttpCalloutMock.class, new OregonDistrictsErrorMock());
        Test.startTest();
        OregonDistrictsBatch odb = new OregonDistrictsBatch();
        Database.executeBatch(odb,10);
        Test.stopTest();
        // after the testing stops, assert that 'apiLimitExceeded' var is set to true
        // (this test simulates a callout after daily API limits are met)
        System.assertEquals(odb.apiLimitExceeded, true);
    }

}

OregonDistrictsErrorMock.apxc:

@isTest
global class OregonDistrictsErrorMock implements HttpCalloutMock {
    // Implement this interface method
    global HTTPResponse respond(HTTPRequest request) {
        // Create a fake error response to test error condition
        HttpResponse response = new HttpResponse();
        response.setHeader('Content-Type', 'application/json');
        response.setBody('{"returned_errors":[{"error":{"errors":[{"domain":"usageLimits","reason":"dailyLimitExceeded"}]}}]}');
        response.setStatusCode(200);
        return response; 
    }
    global HTTPResponse exceptionRespond(HTTPRequest request) {
        CalloutException e = (CalloutException)CalloutException.class.newInstance();
        e.setMessage('Unauthorized endpoint, please check Setup->Security->Remote site settings.');
        throw e;
    }
}

OregonDistrictsMock.apxc:

@isTest
global class OregonDistrictsMock implements HttpCalloutMock {
    // Implement this interface method
    global HTTPResponse respond(HTTPRequest request) {
        // Create a fake response
        HttpResponse response = new HttpResponse();
        response.setHeader('Content-Type', 'application/json');
        response.setBody('{"civic_data":{"congressionaldistrictnum":"5","statehousedistrictnum":"20","statesenatedistrictnum":"10"}}');
        response.setStatusCode(200);
        return response; 
    }
}

Answer 1

This is a tricky subtlety that has to do with how asynchronous Apex is enqueued and executed in unit test context. I think your code probably works, but in the way you're testing it right now, you cannot verify it.

static testmethod void testErrorCondition() { 
    Test.setMock(HttpCalloutMock.class, new OregonDistrictsErrorMock());
    Test.startTest();
    OregonDistrictsBatch odb = new OregonDistrictsBatch();

This is a bit like a shell game: we've got to watch the instance of OregonDistrictsBatch here.

    Database.executeBatch(odb,10);

At this point, what's happened is that Salesforce has serialized odb and stored it in the database for later execution. No changes have been made to your copy of odb.

    Test.stopTest();

Now, we ask Salesforce to require all asynchronous code between here and Test.startTest() to execute asynchronously. In normal Apex, of course, we wouldn't be able to do that, so we'd never assume that odb would be updated with batch results. But here, it sure looks like we can do...

    // after the testing stops, assert that 'apiLimitExceeded' var is set to true
    // (this test simulates a callout after daily API limits are met)
    System.assertEquals(odb.apiLimitExceeded, true);

But we can't: the OregonDistrictsBatch that just got executed synchronously is not odb. It's a new instance the platform deserialized from what it originally persisted when you called Database.executeBatch(), and it's not connected to odb at all.

In order to verify the error behavior here, you can call execute() directly, synchronously, passing it a List<Contact> that you choose and query yourself. This test is not very satisfying, because you can't do a true end-to-end; however, if you do this, and obviate the need for Database.executeBatch(), you'll be able to validate the internal state of the batch class after the execute() method finishes.

As an unrelated note, you have an issue with the other test method, test(). You're hard-coding an Account id that your test code can't see:

            accountId='0016100001RyWhAAAV',

You'll need to create the Account in test context.