1

Whenever records are inserted in Contacts(usually inserted in 1000's), we have a batch job that checks related customer account in SFDC and populates the contact.accountId field and couple other fields. This so far worked fine. Now, we have a scenario where 1 million contact records (approximately 5000 batches) should be inserted into SFDC using Bulk API, and even with flex queue enabled, I am running out of available batches.

Trigger:

trigger conTrigger on Contact (after insert) {
    ContactHandler.managecontacts(trigger.new);
}

ContactHandler.class

public static void managecontacts(List<Contact> acList) {
    Database.executeBatch(new UpdateContactsBatch(acList));
} 

UpdateContactsBatch.class

    public class UpdateContactsBatch implements Database.Batchable<sObject>, Database.Stateful {
        List<Contact> acList = new List<Contact>();

public UpdateContactsBatch (List<Contact> acList) {
this.acList = acList;
}

            public Iterable<sObject> start(Database.BatchableContext BC) {
                return [select id, accountId from contact where Id In: acList];
            }

            public void execute(Database.BatchableContext BC, List<AccountContacts__c> scope) {
                ContactTriggerHandler.managecontacts_execute(scope);        
            }

            public void finish(Database.BatchableContext BC) { }        

        }

One option I have is to turn off the trigger during this mass load and manually run the batch from anonymous window after inserting all contacts. (Tested this scenario and this is working fine in PreProd environment). In Production this requires 2 times deployment (one for activating trigger and one for deactivating. Donot prefer having custom settings to control trigger behaviour)

But would like to know if there is any other option to handle this scenario.

Problem: I changed the batch size to 5000 in my dataloader which is currently set to "Use Bulk API" but still seems like my trigger is getting 200 records at a time and for each load of records from trigger.new, a batch job will be queued

enter image description here

  • What is the method of invoking the bulk api? – NSjonas Feb 5 '16 at 2:45
  • I am currently using dataloader for it. In dataloader, settings > check the option "Use Bulk API" and you can define batch size there – sf_user Feb 5 '16 at 2:47
  • is there an option to set the concurrencyMode? – NSjonas Feb 5 '16 at 2:48
  • added screenshot of settings in dataloder. I dont see concurrency mode there – sf_user Feb 5 '16 at 2:50
  • Try checking "enable serial mode for bulk api"! Also REDUCE your batch size. I could be wrong but, if you run 1000 items, it will be split into trigger batches of 200 and you will end up with potentially 5 batches at once. Start small and work your way up. – NSjonas Feb 5 '16 at 2:51
1

So I think the problem is that the bulk data API is running multiple asynchronous batches and thus kicking off too many Batchable Contexts.

It looks like your batch process doesn't actually care what Id's you pass in and instead just processes everything?

If this is the case and you don't need real time (or close to realtime) updates, it might be a better idea to just schedule this to run.

Otherwise, if you need it to happen in near real time, you could use batch "chaining" in order the have the batch keep running as long as there are records to process.

The basic steps would look something like this:

  1. In the finish method, add a check to see if there are more records to process. If so kick off another batch.
  2. In your trigger, have it first check if the batch is already running. If so, you don't need to do anything. Otherwise, have it kick off the batch.

Update

If you have access to the JobInfo concurrencyMode param, you can set this to "Serial" and it will prevent multiple batches from running in parallel. I think this might solve you problem. If that alone doesn't work, try also reducing the batch size. Obviously, this will cause the import to take much (much) longer.

On the dataloader this Serial can be turned on by checking "Enable Serial Mode for Bulk Api".

  • Let me correct the code.. I got what you said that I am querying all the ids... Will paste updated code soon – sf_user Feb 5 '16 at 2:00
  • updated my code.. also, when loading contacts from external source using bulk API, I set the batch size to 5000 instead of 200 but still same error. Will look into chaining and see if it helps – sf_user Feb 5 '16 at 2:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.