Working around callout limit in Batch Apex

Question

I have a batch Apex class that makes a callout to an API that returns records in pages of 25. It also tells me how many pages there are so essentially I do - so I have a loop that then goes through those and makes any subsequent calls.

However, if there are over 2500 results... then there are more than 100 callouts being made.

Is there a structure that would help me get around this limit?

    Public static void getAddlLeads(set<string> companyIds) {

       for(string ziCompanyId : companyIds) {
            //Make the call
            Http http = new Http();
            HttpRequest req = new HttpRequest();
            req.setendpoint(fullURL);
            req.setmethod('GET');
            req.setHeader('Accept', 'application/json');
            HttpResponse res = http.send(req);
            String response = res.getBody();

            //return response;
            System.debug('response: ' + response);
            List <ziPersonSearchJSON.PersonRecord> addPeople = new List<ziPersonSearchJSON.PersonRecord>();
            List <Lead> ziAddlLeads = new List < Lead > ();

            ziPersonSearchJSON obj = ziPersonSearchJSON.parse(res.getbody());

            decimal totalResults = obj.PeopleSearchRequest.TotalResults;
            decimal callsReq = totalResults.divide(25,4).round(System.RoundingMode.UP);
            system.debug('need to make calls: ' + callsReq );

            for( ziPersonSearchJSON.PersonRecord person: obj.PeopleSearchRequest.PeopleSearchResults.PersonRecord) {
                addPeople.add(person);
                system.debug('Found: '+person.lastName);
            }

            //make additional calls and put them into the list 
            if(callsReq > 1){
                for( integer i = 2; i <= callsReq; i++ ){
                    //Make the call
                    system.debug('Making PersonSearch: '+i);
                    Http httpA = new Http();
                    HttpRequest reqA = new HttpRequest();
                    reqA.setendpoint(fullURL+'&page='+i);
                    system.debug(fullURL+'&page='+i);
                    reqA.setmethod('GET');
                    reqA.setHeader('Accept', 'application/json');
                    HttpResponse resA = http.send(reqA);
                    system.debug(resA.getbody());

                    try{
                    ziPersonSearchJSON objA = ziPersonSearchJSON.parse(resA.getbody());


 for( ziPersonSearchJSON.PersonRecord personA: objA.PeopleSearchRequest.PeopleSearchResults.PersonRecord) {
                        addPeople.add(personA);
                        system.debug('Found: '+personA.lastName);
                    }
                } catch (Exception e) {
                    //nothing
                    system.debug('nothing');
                }


            }

        }

        for (ziPersonSearchJSON.PersonRecord person: addPeople) {
            lead matchLead = new Lead();
            matchLead.lastName = person.lastName;
            matchLead.firstName = person.firstName;
            matchLead.company = person.CurrentEmployment.Company.CompanyName;
            matchLead.title = person.CurrentEmployment.JobTitle;
            //matchLead.Zoom_Info_Company_ID__c = person.CurrentEmployment.Company.companyId;
            matchLead.website = person.CurrentEmployment.Company.companywebsite;
            matchLead.Zoom_Info_ID__c = person.personId;
            matchLead.email = person.email; 
            matchLead.Zoom_Info_Direct_Line__c = person.phone;
            //matchLead.Zoom_Info_Company_Phone__c = person.CurrentEmployment.Company.CompanyPhone;
            //matchLead.ziLastUpdate__c = date.newInstance(1990, 1, 1);//will trigger ziUpdate
            ziAddlLeads.add(matchLead);
        }

        addpeople = null ;
        obj = null;

        system.debug('found these leads: '+ziAddlLeads);

        //dup checker 
        List<Lead> fl2 = new List<Lead>();

        if(ziAddlLeads.size() > 0) { 
            system.debug('running dup check');
            Set<String> companyOfNewLeads = new Set<String>(); 
            Set<String> emailOfNewLeads = new Set<String>();
            Set<String> fnameOfNewLeads = new Set<String>();
            Set<String> LnameOfNewLeads = new Set<String>();
            for(Lead l : ziAddlLeads)
            {
                if(l.email != null)
                {
                    emailOfNewLeads.add(l.email);
                }
                if(l.firstName != null)
                {
                    fnameOfNewLeads.add(l.firstname);
                }
                if(l.lastname != null)
                {
                    LnameOfNewLeads.add(l.lastname);
                }
                if(l.company != null)
                {
                    companyOfNewLeads.add(l.company.trim());
                }
            }

            Set<ID> lids = new Set<ID>();
            Set<ID> cids = new Set<ID>();
            List <Contact> ctc = [Select id, name, firstname, Zoom_Info_Individual_ID__c, lastname, email, contact.account.Name, focusId__c, focusStatus__c from contact where contact.account.Name in: companyOfNewLeads OR email in: emailOfNewLeads ];
            List <Lead> exisLd = [Select Id, firstName, Zoom_Info_ID__c, LastName, email, company, focusId__c from Lead WHERE (company in: companyOfNewLeads OR email in: emailOfNewLeads) and isConverted = false];

            // Dupe Checker Rules                       
            for(Lead li : ziAddlLeads)
            {
                //names +='The new leadis "' + li.firstname+ '" "' + li.lastname + '" "' + li.company +   '"!!';    
                Boolean found = false;
                for(Lead lp : exisLd)
                {
                    if (( li.firstname == lp.firstname && li.lastname == lp.lastname && li.company == lp.company) || (li.email == lp.email) && li.email != null || li.Zoom_Info_ID__c == lp.Zoom_Info_ID__c )
                    {
                        found = true;
                        system.debug('found '+li.firstName);
                        break;
                    }
                }

                for(Contact c : ctc)
                {
                    if( (li.firstname == c.firstname && li.lastname == c.lastname && li.company == c.account.Name ) || (li.email == c.email) && li.email != null || li.Zoom_Info_ID__c == c.Zoom_Info_Individual_ID__c)
                    {
                        found = true;
                        system.debug('found '+li.firstName);                     
                        break;
                    }                                   
                }
                if(found)
                    continue; 
                if(found == false)
                {
                    fl2.add(li);
                }
            }
            system.debug('new Leads:' + string.valueOf(fl2.size()));

            insert fl2;

        }
    }
}

Answer 1

The 100 callouts limit is per execution context, in context to what you are talking about that is each execute() iteration within your batch class.

It sounds to me like you could do to refactor your batch class to so that not all 100 are being consumed in one execute(). In reality, this could mean reducing your batch size down to as little as 1, e.g.

Database.executeBatch(new CalloutClass(), 1);

Answer 2

Ok, I was able to do the following...

Scheduled Batch 1 gets the leads that are marked as New and runs the first callout to determine how many results the API has for that company. For each page of results return, it creates a custom object callout__c that has a field for the url.

Scheduled Batch 2 then consumes those callout__c objects making the api call for each and creating the leads returned. At the end of processing them, it then deletes them.

Is this an advisable strategy?