Saving data from HTTP response in a 'for' loop

Question

Background:

• We're building an app, that will be periodically calling out to external web service to retrieve data in XML which we need to parse and save to Salesforce.

So far:

• We have an Apex Job that calls Schedulable Apex Class, which calls out Queueable Apex Class which then calls our Apex Class. In our Apex Class we're doing HTTP requests and DML operations.
• First we retrieve Account IDs from Salesforce via SOQL query and save them to a List.

List<Account> accounts =  [SELECT Id FROM Account WHERE something__c = true];
           List<String> accIds = new List<String>();
           for(Account acc : accounts){
               accIds.add(acc.Id);
           }

• Then via for loop we're making HTTP request to the external web service with our previously retrieved ID's.

getUpdateXml(accIds);
for(String accId : accIds){
String s = 'XML request';
           HttpRequest req = new HttpRequest();
           req.setEndpoint(endpoint);
           req.setTimeout(30000);
           req.setMethod('POST');
           req.setBody(s);
           req.setHeader('Content-Type', 'text/xml');
           req.setHeader('SOAPAction', '""');
           Http http = new Http();
           HTTPResponse res = http.send(req);

• We parse the HTTP response and save it to the List of strings.

String xml = res.getBody();
List<String> listS = new List<String>();
insertList(accId, listS);

• Then another for is creating new records under our Custom object, which is linked to Account object.

for (String sId : listS){
           something sm = new something__c();
           sm.Account__c = accId;
           sm.Name = sId;
           ...
           insert sm;
       }

• In the next HTTP request, we ask the web service about data which we need to save to the Account record. This is when we start getting error messages.

Problem:

• There are two error notices we're receiving. The first is "You have uncommitted work pending. Please commit or roll back before calling out.", we understand why we're receiving this error - because we're parsing XML response and simultaneously saving it into Salesforce (DML operation).
• We tried using @future calls and we received different error "Too many future calls: 51", which occurs because we have more than 50 responses within one 'for' loop.

Questions:

• How should we approach this problem - how to save (into a Salesforce object) more than 50 HTTP responses (in a for loop) without getting one of those two errors mentioned above?

Thank you very much!

Answer 1

It sounds like you are presently:

1. Making a first set of HTTP requests and holding the results in a list
2. Inserting the list items (not you should change that to insert them in one request i.e. build a list of something__c and then insert that list)
3. Making a second set of HTTP requests and this generates the "You have uncommitted work pending" error
4. Presumably doing some more DML

The ideal fix is to do all the HTTP calls first then do all the DML calls so the order would change to 1, 3, 2, 4.

But note that you can only make 100 callouts in one transaction (see Execution Governors and Limits) so using e.g. a batchable is a way to divide the work up into limited size pieces.