Batch Class with Custom Iterator Not Using First Element?

Question

I have been hitting my head against the wall with this one.

I have a batch class that runs against Leads, Contacts, and Accounts as well as 2 external objects. I created a CustomIterator to accomplish it all in one class. This iterator returns List to the Database class. I am trying to test my Batch Class but am hitting an issue when I pass in a Lead. I am debugging my code and can see that my Lead record is present in the CustomIterable, but my execute never runs. What exactly could be the issue for this?

This is what my Batch Class looks like. I removed the lines of code that I do not believe are impacting this behavior.

global with sharing class BatchSyncClass implements Database.Batchable<Object>, Database.AllowCallouts {
    // The selected SObjects to perform processing against. Assigned via the Constructor
    private final List<String> SELECTED_OBJECT_TYPES;
    // How far back this process should look. Assigned via the Constructor. Constructor will update to the last time this process ran, otherwise it goes back to yesterday at midnight UTC
    private final Datetime START_DATE;
     // The list of Account record changes that need to be saved
     @TestVisible
     private List<Account> ACCOUNT_RECORDS = new List<Account>();
     // the List of Lead record changes that need to be saved
     @TestVisible
     private List<Lead> LEAD_RECORDS = new List<Lead>();
     // The List of Contact record changes that needs to be saved.
     @TestVisible
     private List<Contact> CONTACT_RECORDS = new List<Contact>();  

    // Constructor to specify how far back to look for data
    global BatchSyncClass(Datetime theStartDate) {
        this.START_DATE = theStartDate;
        this.SELECTED_OBJECT_TYPES = this.VALID_OBJECT_TYPES;
    }

    // Starting BatchableContext
    global Iterable<Object> start(Database.BatchableContext theBatchableContext) {
        System.debug('The Selected Objects => ' + this.SELECTED_OBJECT_TYPES);
        CustomIterable theCustomIterable = new CustomIterable(this.SELECTED_OBJECT_TYPES, this.START_DATE);
        System.debug('theCustomIterable => ' + theCustomIterable);
        System.debug('theCustomIterable.iterator => ' + theCustomIterable.iterator());
        return theCustomIterable;
    }

    // Main method for batch execution
    global void execute(Database.BatchableContext theBatchableContext, List<Object> theRecords) {
        System.debug('BatchSyncClass ::: execute');
        System.debug('theRecords => ' + theRecords);
        List<Lead> theLeads = new List<Lead>();
        List<Account> theAccounts = new List<Account>();
        List<Contact> theContacts = new List<Contact>();
        for (Object theRecord: theRecords) {
            try {
                System.debug('theRecord => ' + theRecord);
                Map<String,Object> theRecordMap = (Map<String,Object>)JSON.deserializeUntyped(JSON.serialize(theRecord));
                System.debug('theRecordMap => ' + theRecordMap);
                if (theRecordMap.get('attributes') != null) {
                    Map<String,Object> theRecordAttributes = (Map<String,Object>)theRecordMap.get('attributes');
                    if (theRecordAttributes.get('type').toString() == 'Account') {
                        theAccounts.add((Account)((SObject)JSON.deserialize(JSON.serialize(theRecord), SObject.class)));
                    } else if (theRecordAttributes.get('type').toString() == 'Contact') {
                        theContacts.add((Contact)((SObject)JSON.deserialize(JSON.serialize(theRecord), SObject.class)));
                    } else if (theRecordAttributes.get('type').toString() == 'Lead') {
                        theLeads.add((Lead)((SObject)JSON.deserialize(JSON.serialize(theRecord), SObject.class)));
                    }
                } 
            } catch(Exception e) {
                System.debug('e=>'+e);
            }
        }
        System.debug('Number Leads => ' + theLeads.size());
        System.debug('theLeads => ' + theLeads);
        System.debug('Number Accounts => ' + theAccounts.size());
        System.debug('theAccounts => ' + theAccounts);
        System.debug('Number Contacts => ' + theContacts.size());
        System.debug('theContacts => ' + theContacts);
        handleAccounts(theAccounts);
        handleContacts(theContacts);
        handleLeads(theLeads);
        handleSubmitChanges();
    }

    // Finish BatchableContext. Runs once all records/changes have been processed.
    global void finish(Database.BatchableContext theBatchableContext) {
        System.debug(LoggingLevel.INFO, 'Batch Sync complete!');
    }

    // Custom Iterable class
    public class CustomIterable implements Iterable<Object> {
        private List<String> objectTypes;
        private Datetime startDate;

        public CustomIterable() {}

        public CustomIterable(List<String> objectTypes) {
            this.objectTypes = objectTypes;
        }

        public CustomIterable(List<String> objectTypes, Datetime startDate) {
            this.objectTypes = objectTypes;
            this.startDate = startDate;
        }

        public Iterator<Object> iterator() {
            List<Object> theRecords = new List<Object>();
            for (String objectType: objectTypes) {
                if (objectType == 'Account' || objectType == 'Contact' || objectType == 'Lead') {
                    String query = getQuery(objectType);
                    if (query != null) {
                        theRecords.addAll(Database.query(query));
                    }
                } 
            }
            return new CustomIterator(theRecords);
        }

        private String getQuery(String sobjectType) {
            String queryString;
            String startDateString;
            if (this.startDate != null) {
                startDateString = this.startDate.formatGMT('yyyy-MM-dd\'T\'HH:mm:ss.SSS\'Z\'');
            }
            if (sobjectType == 'Contact') {
                queryString = 'SELECT Id, FirstName, LastName, Status__c, AccountId, Email, Account.Customer_Status__c, Account.Type, Contact_Role__c, ' + 
                    'Do_Not_Email__c, DoNotCall, Phone, Title, HasOptedOutOfEmail, LeadSource ' + 
                    'FROM Contact';
            } else if (sobjectType == 'Account') {
                queryString = 'SELECT Id, Name, Customer_Status__c, Type, Primary_Business__c, Industry, AccountSource, Associated_Content__c, ' + 
                    'Website, BillingStreet, BillingCity, BillingPostalCode, BillingState, BillingCountry ' + 
                    'FROM Account';
            } else if (sobjectType == 'Lead') {
                queryString = 'SELECT Id, FirstName, LastName, Status, LeadSource, Associated_Content__c, Company, DoNotCall, Do_Not_Email__c, Email, ' + 
                    'HasOptedOutOfEmail, HasOptedOutOfFax, Industry, Phone, Name, Rating, Title, Website, Secondary_Source__c, Description ' + 
                    'FROM Lead';
            } 
            if (queryString != null && startDateString != null) {
                queryString += ' WHERE LastModifiedDate >=' + startDateString + ' ';
            }
            System.debug('Query String => ' + queryString);
            return queryString;
        }
    }

    // Custom Iterator Class 
    public class CustomIterator implements Iterator<Object> {
        private List<Object> theRecords;
        private Integer currentIndex;

        public CustomIterator(List<Object> theRecords) {
            System.debug('The Records to Iterate Over => ' + theRecords); 
            this.theRecords = theRecords;
            this.currentIndex = 0;
        }

        public Boolean hasNext() {
            System.debug('currentIndex => ' + currentIndex);
            System.debug('theRecords.size() => ' + theRecords.size());
            Boolean hasNext = theRecords != null && !theRecords.isEmpty() && currentIndex < (theRecords.size() - 1);
            System.debug('hasNext => ' + hasNext);
            return hasNext;
        }

        public Object next() {
            if (hasNext()) {
                currentIndex += 1;
                System.debug('New currentIndex => ' + currentIndex);
                Object theRecordToReturn = theRecords[currentIndex];
                System.debug('theRecordToReturn => ' + theRecordToReturn);
                return theRecordToReturn;
            } else {
                throw new NoSuchElementException('Iterator has no more elements.');
            }
        }
    }
}

In my Debug Logs, I can see that the Lead record has been found but the "execute" is not processing. I am not sure exactly what the problem is.

Test Class/Method

@isTest 
private without sharing class BatchSyncClassTest {

    @isTest 
    private static void lead_batch_sync() {
        Test.setMock(HttpCalloutMock.class, new BatchSyncClassMockResponse.ContactSync());
        TestDataFactory.clearFeatureFlags();
        Lead l = TestDataFactory.insertLeads(1, false)[0];
        l.Email = '[email protected]';
        l.Status = 'Prospecting';
        insert l;
        Test.startTest();
        BatchSyncClass batchsync = new BatchSyncClass(System.now().addHours(-1));
        Database.executeBatch(batchsync, 5);
        Test.stopTest();
        Lead theUpdatedLead = [ 
            SELECT Id, FirstName, LastName, Status, LeadSource, Associated_Content__c, Company, DoNotCall, Do_Not_Email__c, Email, 
            HasOptedOutOfEmail, HasOptedOutOfFax, Industry, Phone, Name, Rating, Title, Website, Secondary_Source__c, Description 
            FROM Lead
            WHERE Id = :l.Id
            LIMIT 1
        ];
        System.assertEquals('Website', theUpdatedLead.LeadSource);
    }
}

I can find where the Lead record is added to my Iterable object but my execute method never runs

10:34:47.131 (1975882123)|USER_DEBUG|[734]|DEBUG|The Records to Iterate Over => (Lead:{Id=00Q7b00000GLzl7EAD, FirstName=New, LastName=Lead 1667835286065, Status=Prospecting, Company=Test Company 1667835286065, DoNotCall=false, Do_Not_Email__c=false, [email protected], HasOptedOutOfEmail=false, HasOptedOutOfFax=false, Name=New Lead 1667835286065, Title=Test Lead})

I also see it is present in the ".iterator()" method in the Batch start method...

10:34:47.131 (1952374249)|USER_DEBUG|[110]|DEBUG|theCustomIterable.iterator => CustomIterator:[currentIndex=0, theRecords=(Lead:{Id=00Q7b00000GLzl7EAD, FirstName=New, LastName=Lead 1667835286065, Status=Prospecting, Company=Test Company 1667835286065, DoNotCall=false, Do_Not_Email__c=false, [email protected], HasOptedOutOfEmail=false, HasOptedOutOfFax=false, Name=New Lead 1667835286065, Title=Test Lead})]

I am at a loss here, maybe I just have tunnel vision at this point. Is anyone able to spot the issue here?

Thanks!

Answer 1

To answer your question directly, you incremented the index before you used it:

public Object next() {
    if (hasNext()) {
        currentIndex += 1;
        System.debug('New currentIndex => ' + currentIndex);
        Object theRecordToReturn = theRecords[currentIndex];
        System.debug('theRecordToReturn => ' + theRecordToReturn);
        return theRecordToReturn;
    } else {
        throw new NoSuchElementException('Iterator has no more elements.');
    }
}

It should be:

public Object next() {
    if (hasNext()) {
        Object theRecordToReturn = theRecords[currentIndex];
        currentIndex += 1;
        return theRecordToReturn;
    } else {
        throw new NoSuchElementException('Iterator has no more elements.');
    }
}

That said, I feel like your batchable is overly complicated. Here's my implementation of your custom record iterator:

public class CustomIterator implements Iterator<Object> {
    Object[] theRecords;
    public CustomIterator(Object[] theRecords) {
        this.theRecords = theRecords.clone();
    }
    public Boolean hasNext() {
        return this.theRecords.size() > 0;
    }
    public Object next() {
        return this.theRecords.remove(0);
    }
}

Your code doesn't need to be nearly as defensive as you made it; if the callee falls off the stack because they didn't call hasNext first, or they passed in a null list, or whatever else, that's on them.

I sometimes quote Say No to Null and, to add to advice I'd link, I'd recommend to learn how to Code Without Exceptions.

Your code has a lot of weird edge cases that "can't" happen. For example:

if (objectType == 'Account' || objectType == 'Contact' || objectType == 'Lead') {
    String query = getQuery(objectType);
    if (query != null) {
        theRecords.addAll(Database.query(query));
    }
} 

Calls getQuery, if, and only if, it's one of three values. But then, in getQuery, you do this:

if (queryString != null && startDateString != null) {
    queryString += ' WHERE LastModifiedDate >=' + startDateString + ' ';
}

queryString cannot be null, because only three valid objects are allowed to be passed in. That leads to:

if (query != null) {

Which is always true, because query will not be null. We can assure ourselves of these facts by thinking through the logic carefully. Consider how many lines of code I'm using in the custom iterator to yours. This isn't just less code, it will run faster.

Some further optimization is likely possible as well, but I'll end here for now.