Creating records from csv is running twice

Question

I wrote a lightning component that reads a csv file and creating records with relationship, e.g opportunities and their related contacts in each row. some of the contacts already exists in org therefore every contact is checked by soql comparing some email and phone fields(for the complexity of data comparison in multiple fields I used soql rather than duplication error check), if exist relating the opportunity to it and if not creating new one and relating to the new contact.

Since a large set of data is processed, to avoid reaching limits I call the apex class from js helper for every small set of rows.

for (let i = 0; i < csvAsList.length; i = i + 25) {
            helper.CreateRecords(component, csvAsList.slice(i, i + 25), i >= csvAsList.length - 25);        
        }

Helper code:

CreateRecords : function (component,csv, isLast){
      var action = component.get('c.insertData');
      action.setParams({
        csvAsList : csv
      });
      if(isLast){
      action.setCallback(this, function(response) {
          var state = response.getState();
          if (state === "SUCCESS") {
              var result=response.getReturnValue();
              alert("Records uploaded succefully");
          }
          else if (state === "ERROR") {
              var errors = response.getError();
              if (errors) {
                  if (errors[0] && errors[0].message) {
                      console.log("Error message: " + errors[0].message);
                  }
              } else {
                  console.log("Unknown error");
                  alert('Unknown Error');
              }
          }
      }); 
    }
      $A.enqueueAction(action);    
  }

For a small set of records, even when apex class is called several times everything works fine, but when I try to upload the entire file (about 16,000 records) the process ends up creating each row twice (about 32,000 opportunities). For further clarclarification needs to emphasis that it's almost and not exactly twice.

Apex Class:

public class UploadRecordsFromCsvCtrl{
  @AuraEnabled
  public static string insertData(List<String> csvAsList){
    Callable npspApi = (System.Callable)Type.forName('npsp', 'Callable_API').newInstance();
    Boolean isNpspTriggerDisabled = (Boolean)npspApi.call('TDTM.DisableAllTriggers', new Map<String, Object>()); 
    String returnresponse = '';
    List<ContactAndItsOpportunity> conAndOppList = iterateOverRows(csvAsList);
    system.debug(conAndOppList.size());
    List<Opportunity> oppList = new List<Opportunity>();
      returnresponse = 'SUCCESS';
      for (ContactAndItsOpportunity conAndOpp : conAndOppList){
        List<Contact> conList = [SELECT Id, FirstName, LastName
                         FROM Contact
                         where (MobilePhone != null and MobilePhone = :conAndOpp.con.MobilePhone) Or (Phone != null and Phone = :conAndOpp.con.MobilePhone) Or (OtherPhone != Null and OtherPhone = :conAndOpp.con.MobilePhone) Or (HomePhone != null And HomePhone = :conAndOpp.con.MobilePhone) Or (npe01__WorkPhone__c != null and npe01__WorkPhone__c = :conAndOpp.con.MobilePhone) Or (HomePhoneUnformatted__c != null and HomePhoneUnformatted__c = :conAndOpp.con.MobilePhone) Or (MobilePhoneUnformatted__c != null and MobilePhoneUnformatted__c = :conAndOpp.con.MobilePhone) Or (formula_for_phone__c != null and formula_for_phone__c = :conAndOpp.con.MobilePhone) Or (WorkPhoneUnformatted__c != null and WorkPhoneUnformatted__c = :conAndOpp.con.MobilePhone) Or (OtherPhoneUnformatted__c != null and OtherPhoneUnformatted__c = :conAndOpp.con.MobilePhone) Or (Email != null and Email = :conAndOpp.con.Email) Or (npe01__WorkEmail__c != null and npe01__WorkEmail__c = :conAndOpp.con.Email) Or (npe01__HomeEmail__c != null and npe01__HomeEmail__c = :conAndOpp.con.Email) Or (npe01__AlternateEmail__c != null and npe01__AlternateEmail__c = :conAndOpp.con.Email)];
        try{
        if ((conList.size() == 0)){
          Database.SaveResult[] srList = Database.insert (new List<Contact>{ conAndOpp.con }, false);
          if (srList[0].isSuccess()){
            conAndOpp.opp.npsp__Primary_Contact__c = srList[0].getId();
            conAndOpp.opp.ContactId = srList[0].getId();
          } 
          // else{
          //   for (Database.Error err : srList[0].getErrors()){
          //     system.debug(err.getMessage());
          //     if (err instanceOf Database.DuplicateError){
          //       Database.DuplicateError duplicateError = (Database.DuplicateError)err;
          //       Datacloud.DuplicateResult duplicateResult = duplicateError.getDuplicateResult();
          //       List<DataCloud.MatchResult> matchRes = duplicateResult.getMatchResults();
          //       List<dataCloud.MatchRecord> matchRec = matchRes[0].getMatchRecords();
          //       Sobject obj = matchRec[0].getRecord();
          //       conAndOpp.opp.npsp__Primary_Contact__c = (Id) obj.get('Id');
          //       conAndOpp.opp.ContactId = (Id) obj.get('Id');
          //     }
          //   }
          // }
        } else{
          if (conAndOpp.con.LastName != 'UnKnown' && conAndOpp.con.LastName != '' && conList[0].LastName == 'UnKnown'){
            conList[0].LastName = conAndOpp.con.LastName;
            if (conAndOpp.con.LastName != '')
              conList[0].FirstName = conAndOpp.con.FirstName;
            update (conList[0]);
          }
          conAndOpp.opp.npsp__Primary_Contact__c = conList[0].Id;
          conAndOpp.opp.ContactId = conList[0].Id;
        }
        oppList.add(conAndOpp.opp);
        system.debug(conAndOpp);
      } catch (DmlException ex){
        system.debug(ex.getMessage());
        returnresponse = 'DML ERROR';
      } catch (Exception e){
        returnresponse = 'ERROR';
        system.debug(e.getMessage());
      }
      }
      try{
        for(Opportunity oppde : oppList){
          system.debug(oppde); 
        }
        Database.SaveResult[] srList = Database.insert (oppList, false);
      } catch (Exception e){
        system.debug(e.getMessage());
      }
   

    return returnresponse;
  }
  public static List<ContactAndItsOpportunity> iterateOverRows(List<String> csvAsList){
    List<ContactAndItsOpportunity> conAndOppList = new List<ContactAndItsOpportunity>();
    for (integer i = 0; i < csvAsList.size(); i++){
      List<String> csvLineAsList = handleCommasAndChangingCsvLineToList(csvAsList[i]);
      conAndOppList.add(new ContactAndItsOpportunity());
      conAndOppList[i].con = (new Contact(FirstName = csvLineAsList[2], Company__c = csvLineAsList[4], LastName = csvLineAsList[3], MobilePhone = csvLineAsList[8], Email = csvLineAsList[7], MailingStreet = csvLineAsList[9], OtherStreet = csvLineAsList[10], MailingCity = csvLineAsList[11], MailingState = csvLineAsList[12], MailingCountry = csvLineAsList[13], MailingPostalCode = csvLineAsList[14], RecordTypeId = '012b000000010UFAAY'));
      String currencyIsoCode = 'USD';
      if (!String.isBlank(csvLineAsList[6])){
        currencyIsoCode = csvLineAsList[6];
      }
      conAndOppList[i].opp = (new Opportunity(Name = 'c', CloseDate = Date.valueOf(csvLineAsList[0]), Amount = Decimal.valueOf(csvLineAsList[5].trim()), StageName = 'Paid', CurrencyIsoCode = currencyIsoCode));
    }
    return conAndOppList;
  }
  public static List<String> handleCommasAndChangingCsvLineToList(String csvLine){
    String prevLine = csvLine;
    Integer startIndex;
    Integer endIndex;

    while (csvLine.indexOf('"') > -1){

      if (startIndex == null){
        startIndex = csvLine.indexOf('"');
        csvLine = csvLine.substring(0, startIndex) + ':quotes:' + csvLine.substring(startIndex + 1, csvLine.length());
      } else{
        if (endIndex == null){
          endIndex = csvLine.indexOf('"');
          csvLine = csvLine.substring(0, endIndex) + ':quotes:' + csvLine.substring(endIndex + 1, csvLine.length());
        }
      }

      if (startIndex != null && endIndex != null){
        String sub = csvLine.substring(startIndex, endIndex);
        sub = sub.replaceAll(',', ':comma:');
        csvLine = csvLine.substring(0, startIndex) + sub + csvLine.substring(endIndex, csvLine.length());
        startIndex = null;
        endIndex = null;
      }
    }

    list<string> csvLineAsList = new List<String>();
    for (String column : csvLine.split(',', -1)){
      column = column.replaceAll(':quotes:', '"').unescapeCsv().replaceAll(':comma:', ',');
      // System.debug(column);
      csvLineAsList.add(column);
      // system.debug(csvLineAsList.size());
    }
    return csvLineAsList;
  }

  public Class ContactAndItsOpportunity{
    Contact con;
    Opportunity opp;

  }
}

Answer 1

Your CSV parser may be a broken implementation. If the data is being duplicated "almost", but not "exactly" twice, this suggests that your parsing the CSV incorrectly. Try experimenting with random, but valid, combinations of quotes, commas, newlines, and see if you get unexpected results. Compare your code with my own various CSV parser experiments, which function on a large variety of CSV files (e.g. different line endings, spacing, etc). I am fairly certain there is an implementation bug hiding here. Alternatively, consider using clint-side parsing, it's much faster than Apex parsing of CSV, which is what leads to us using "tricks" like indexOf, etc (we should ideally be using a character-by-character finite state machine for CSV parsing).

As far as the JS code, I don't recommend calling these methods all at once, as you're now introducing parallel processing in your Apex code, which can cause row locks, which might also explain some of the "not exactly" duplicating logic you're seeing. Instead, call each one at a time and wait for it to finish processing.

---

helper.CreateRecords(component, csvAsList);

---

({
  CreateRecords: function (component, csv) {
    let csvAsList = csv.splice(0, 25);
    var action = component.get("c.insertData");
    action.setParams({
      csvAsList,
    });
    action.setCallback(this, (response) => {
      var state = response.getState();
      if (state === "SUCCESS") {
        var result = response.getReturnValue();
        alert("Records uploaded succefully");
      } else if (state === "ERROR") {
        var errors = response.getError();
        if (errors) {
          if (errors[0] && errors[0].message) {
            console.log("Error message: " + errors[0].message);
          }
        } else {
          console.log("Unknown error");
          alert("Unknown Error");
        }
      }
      // Recursively call this method
      if (csv.length) {
        this.CreateRecords(component, csv);
      }
    });
    $A.enqueueAction(action);
  },
})

You should be handling every callback for errors (isLast isn't really appropriate).