24

I've recently been trying to come to terms with the oddities of the auto-"batching" of triggers. What I'm referring to is how when dealing with more than 200 records in a trigger execution, Salesforce will automatically "batch" the records into groups of 200. For example, if I run a trigger on 500 Account records, the logic in the trigger will be done three times in succession, on the first 200 records, then the next 200, then the last 100. Your logic will run 3 times in succession, and governor limits are never reset.

This doesn't pose much of a problem in a small-scale operation, but in working in a Salesforce instance with thousands of users, large quantities of data, and extensive automation, these issues are bound to come up.

My first question: what is the best practice to handle a trigger firstRun variable? Since workflows can often cause a trigger to run twice, it's important to pass through pieces of logic only the first time. As for order of execution, I've determined that the first 200 records will move through the trigger, then workflow, then the trigger again. Then the next "batch" will be processed, and so on. Because of this predictable order, my best approach is this (I simplified the overarching structure to demonstrate the logic):

public class MyObjectServices {
    public static Set<Id> recordsProcessed = new Set<Id>();

    public static void myTriggerMethod(Map<Id, MyObject__c> newMap){
        //The number of records processed in all "batches" to this point
        Integer sizeBefore = recordsProcessed.size();

        //Add the ids all records being processed this "batch". If they've already
        //been processed, the set will prevent duplicates from being added
        for (Id key : newMap.keySet()){
            recordsProcessed.add(key);
        }

        //Determines if the records included in the current "batch" of 200 
        //have been processed before
        if (recordsProcessed.size() != sizeBefore){
            //Trigger logic
        }
    }
}

However, this 1) doesn't work for before insert since the id doesn't yet exist, and 2) feels legitmately hacky. Are there any better ways to do this?

My second question: Is there any way to get the total number of records being processed in your trigger? Trigger.size will only give you the number being processed in that particular batch. We could use the recordsProcessed variable from above, but that will only give you the total number during the last "batch" of the trigger. And there's no way of knowing which batch that will be 100% of the time! Yes, if Trigger.size != 200, you know it's the last batch, but what if you're dealing with an exact multiple of 200?

My third question: finally, how do you handle limits? Suppose you update 10,000 of one type of record. This means 50 separate "batches" within the same execution! So, suppose you follow bulkification practices perfectly, you only have 2 queries (100 total) and 3 DML statements (150 total) to work with per batch before you hit governor limits. This is straight-up unworkable. All ideas I can come up with involve incredibly over-engineered methods of separating records into groups of 200 or less and using Queueables to process them. But this makes it very difficult to do any validation, and of course any implementation would be very complicated, so a dedicated technical architect would need to furiously review all incoming code.

So, do any of you have any insight how to face these problems in doing large-scale trigger automation?

EDIT: Adding an example to clear up some misconceptions about how trigger "batching" affects limits and static variables. Trigger:

trigger FooTrigger on Foo__c (before update) {

    System.debug('Trigger before handler call');
    FooTriggerHandler fth = new FooTriggerHandler(
        trigger.oldMap,
        trigger.newMap,
        trigger.old,
        trigger.new,
        trigger.isInsert,
        trigger.isUpdate,
        trigger.isDelete,
        trigger.isUndelete,
        trigger.isBefore,
        trigger.isAfter,
        trigger.size
    );
    System.debug('Trigger after handler call');
}

Apex class:

public class FooTriggerHandler {

    public static Set<Id> allRecordIds = new Set<Id>();
    public static Boolean firstRun = true;
    public static Integer count = 0;

    public FooTriggerHandler(Map<Id,Foo__c> oldMap, Map<Id,Foo__c> newMap, List<Foo__c> triggerOld, List<Foo__c> triggerNew, 
    Boolean isInsert, Boolean isUpdate, Boolean isDelete, Boolean isUndelete, Boolean isBefore, Boolean isAfter, Integer size){
        System.debug('Trigger.size: ' + size);
        System.debug('Entering dispatcher constructor');
        System.debug('DML before: ' + Limits.getDmlStatements());
        System.debug('DML rows before: ' + Limits.getDmlRows());
        System.debug('Queries before: ' + Limits.getQueries());
        System.debug('Query rows before: ' + Limits.getQueryRows());
        System.debug('Firstrun: ' + firstRun);
        firstRun = false;
        System.debug('Record Ids before: ' + allRecordIds.size());
        List<User> someUsers = [SELECT Id FROM User LIMIT 2];
        for (Foo__c f : triggerNew){
           allRecordIds.add(f.Id);
        }
        System.debug('Record Ids after: ' + allRecordIds.size());
        Contact contact1 = new Contact(LastName = 'TestContact' + count++);
        Contact contact2 = new Contact(LastName = 'TestContact' + count++);
        insert new List<Contact>{contact1, contact2};
        System.debug('DML after: ' + Limits.getDmlStatements());
        System.debug('DML rows after: ' + Limits.getDmlRows());
        System.debug('Queries after: ' + Limits.getQueries());
        System.debug('Query rows after: ' + Limits.getQueryRows());
    }

}

Meanwhile, I have a workflow rule running an update every time I update a Foo__c (in order to demonstrate the trigger re-running).

I run the following anonymous code: update [SELECT Id FROM Foo__c LIMIT 300];

And my debug logs:

//First trigger batch, first run
09:18:00:143 USER_DEBUG [3]|DEBUG|Trigger before handler call
09:18:00:144 USER_DEBUG [9]|DEBUG|Trigger.size: 200
09:18:00:144 USER_DEBUG [10]|DEBUG|Entering dispatcher constructor
09:18:00:144 USER_DEBUG [11]|DEBUG|DML before: 1
09:18:00:144 USER_DEBUG [12]|DEBUG|DML rows before: 300
09:18:00:144 USER_DEBUG [13]|DEBUG|Queries before: 1
09:18:00:144 USER_DEBUG [14]|DEBUG|Query rows before: 300
09:18:00:144 USER_DEBUG [15]|DEBUG|Firstrun: true
09:18:00:144 USER_DEBUG [17]|DEBUG|Record Ids before: 0
09:18:00:391 USER_DEBUG [22]|DEBUG|Record Ids after: 200
09:18:01:161 USER_DEBUG [26]|DEBUG|DML after: 2
09:18:01:161 USER_DEBUG [27]|DEBUG|DML rows after: 302
09:18:01:161 USER_DEBUG [28]|DEBUG|Queries after: 2
09:18:01:161 USER_DEBUG [29]|DEBUG|Query rows after: 302
09:18:01:161 USER_DEBUG [17]|DEBUG|Trigger after handler call
//First trigger batch, after workflow
09:18:01:798 USER_DEBUG [3]|DEBUG|Trigger before handler call
09:18:01:799 USER_DEBUG [9]|DEBUG|Trigger.size: 200
09:18:01:799 USER_DEBUG [10]|DEBUG|Entering dispatcher constructor
09:18:01:799 USER_DEBUG [11]|DEBUG|DML before: 2
09:18:01:799 USER_DEBUG [12]|DEBUG|DML rows before: 302
09:18:01:799 USER_DEBUG [13]|DEBUG|Queries before: 2
09:18:01:799 USER_DEBUG [14]|DEBUG|Query rows before: 302
09:18:01:799 USER_DEBUG [15]|DEBUG|Firstrun: false
09:18:01:799 USER_DEBUG [17]|DEBUG|Record Ids before: 200
09:18:01:890 USER_DEBUG [22]|DEBUG|Record Ids after: 200
09:18:01:952 USER_DEBUG [26]|DEBUG|DML after: 3
09:18:01:952 USER_DEBUG [27]|DEBUG|DML rows after: 304
09:18:01:952 USER_DEBUG [28]|DEBUG|Queries after: 3
09:18:01:952 USER_DEBUG [29]|DEBUG|Query rows after: 304
09:18:01:952 USER_DEBUG [17]|DEBUG|Trigger after handler call
//Second trigger batch, first run
09:18:02:623 USER_DEBUG [3]|DEBUG|Trigger before handler call
09:18:02:623 USER_DEBUG [9]|DEBUG|Trigger.size: 100
09:18:02:623 USER_DEBUG [10]|DEBUG|Entering dispatcher constructor
09:18:02:624 USER_DEBUG [11]|DEBUG|DML before: 3
09:18:02:624 USER_DEBUG [12]|DEBUG|DML rows before: 304
09:18:02:624 USER_DEBUG [13]|DEBUG|Queries before: 3
09:18:02:624 USER_DEBUG [14]|DEBUG|Query rows before: 304
09:18:02:624 USER_DEBUG [15]|DEBUG|Firstrun: false
09:18:02:624 USER_DEBUG [17]|DEBUG|Record Ids before: 200
09:18:02:624 USER_DEBUG [22]|DEBUG|Record Ids after: 300
09:18:02:624 USER_DEBUG [26]|DEBUG|DML after: 4
09:18:02:624 USER_DEBUG [27]|DEBUG|DML rows after: 306
09:18:02:624 USER_DEBUG [28]|DEBUG|Queries after: 4
09:18:02:624 USER_DEBUG [29]|DEBUG|Query rows after: 306
09:18:02:624 USER_DEBUG [17]|DEBUG|Trigger after handler call
//Second trigger batch, after workflow
09:18:03:180 USER_DEBUG [3]|DEBUG|Trigger before handler call
09:18:03:180 USER_DEBUG [9]|DEBUG|Trigger.size: 100
09:18:03:180 USER_DEBUG [10]|DEBUG|Entering dispatcher constructor
09:18:03:180 USER_DEBUG [11]|DEBUG|DML before: 4
09:18:03:180 USER_DEBUG [12]|DEBUG|DML rows before: 306
09:18:03:180 USER_DEBUG [13]|DEBUG|Queries before: 4
09:18:03:180 USER_DEBUG [14]|DEBUG|Query rows before: 306
09:18:03:180 USER_DEBUG [15]|DEBUG|Firstrun: false
09:18:03:180 USER_DEBUG [17]|DEBUG|Record Ids before: 300
09:18:03:511 USER_DEBUG [22]|DEBUG|Record Ids after: 300
09:18:03:603 USER_DEBUG [26]|DEBUG|DML after: 5
09:18:03:604 USER_DEBUG [27]|DEBUG|DML rows after: 308
09:18:03:604 USER_DEBUG [28]|DEBUG|Queries after: 5
09:18:03:604 USER_DEBUG [29]|DEBUG|Query rows after: 308
09:18:03:604 USER_DEBUG [17]|DEBUG|Trigger after handler call

Observations:

  1. Using a static first run variable only affects the first 200 records. The variable remains false when processing any additional records. Only use this when you legitimately want code to run exactly one time, not once with each record.
  2. Limits are not reset between batches, nor between re-runs due to workflow. You could technically query the same 500 records each time, and they would count repeatedly toward your query rows, potentially causing you to hit limits without actually working in bulk.
6
  • Gov limits are reset between chunks. At least for bulk API
    – Eric
    Jan 31, 2017 at 22:14
  • 1
    I'm not referring to bulk API - I'm talking about when Apex runs DML on >200 records.
    – SFDC Neuf
    Jan 31, 2017 at 22:29
  • Right, if you are running into limits you will need to move the process to Async. Sync transactions are not designed to do heavy lifting while the user waits. Thats one of the points of giving limits
    – Eric
    Jan 31, 2017 at 22:37
  • 1
    I am aware. The issue that I am concerned with is optimizing synchronous code to work within the limits as best as possible. If I can write efficient code and avoid needing to tell a client they can't do something synchronously, I'd like to do so.
    – SFDC Neuf
    Jan 31, 2017 at 22:57
  • Very nicely asked question. Hope my two cents helps, and I imagine you may get some others to chip in as well. In case no one has said it so far, welcome to SFSE! Your contributions have been great so far.
    – Adrian Larson
    Feb 1, 2017 at 0:23

3 Answers 3

10

However, this 1) doesn't work for before insert since the id doesn't yet exist, and 2) feels legitmately hacky. Are there any better ways to do this?

Your approach (Set<Id> recordsProcessed) is a good one. You do not have to prevent before insert trigger recursion, so it's really not an issue that records do not have an Id yet in that case. I don't find it to be "hacky", and this approach is more robust than a simple Boolean flag (which will only operate correctly on the first batch).

Is there any way to get the total number of records being processed in your trigger?

I don't believe so, unless you set it from the calling context. For instance, you could do something like:

public with sharing class LeadService
{
    public static Integer recordsToProcess = 0;

    // service methods
}

/*VVV calling context VVV*/
List<Lead> toUpdate; // = <some_list>
LeadService.recordsToProcess = toUpdate.size();
update recordsToProcess;

However, you might have futher updates to leads with different values. I would avoid this strategy and find other ways around this limitation. It shouldn't matter which batch comes last. If you want to make sure some logic happens after your trigger logic completes, consider asynchronous processing.

how do you handle limits?

Two common strategies for easing limits usage are to:

  • Use asynchronous processing for heavy lifting
  • Use lazy loading to re-use common data

With the former strategy, you can trade queries/dml/cpu for async calls. It can be more difficult to prevent trigger recursion, but you should be able to work around it by careful application of criteria (filters).

The latter can help when you have configuration data you will need in all your batches. It would look something like:

public static List<ConfigObject> configData
{
    get
    {
        if (configData == null)
            configData = [/*query*/];
        return configData;
    }
    private set;
}
8
  • The static variables reset between chunks correct? So the last part would not work across chunks/batches, right? "Note that static variable values are reset between API batches, but governor limits are not. Do not use static variables to track state information between API batches." -
    – Eric
    Feb 1, 2017 at 2:04
  • That's not how I remember it working...I'll double check shortly.
    – Adrian Larson
    Feb 1, 2017 at 2:05
  • If you had a recursion static variable and it was not reset then the next chunk of 200 records would not process because the recursion variable would still be set.....Just an illustration of why I believe it is being reset between chunks
    – Eric
    Feb 1, 2017 at 2:07
  • And if memory serves, there have been questions where that exact issue came up. Only the first chunk was processed.
    – Adrian Larson
    Feb 1, 2017 at 2:08
  • I just did a quick test and it appears the variable is still set between chunks. (it is not reset). I wonder then what the documentation is referring to when it says they are reset and the gov limits are not.....A bit confusing. I assume it is speaking of bulk API but gov limits would have to be reset between chunks there so it does not make sense in that context either. I feel like I knew this way back when but seem to have forgot it...lol
    – Eric
    Feb 1, 2017 at 2:21
5

My first question: what is the best practice to handle a trigger firstRun variable? Since workflows can often cause a trigger to run twice, it's important to pass through pieces of logic only the first time. As for order of execution, I've determined that the first 200 records will move through the trigger, then workflow, then the trigger again.

A trigger will process records in chunks of at most 200 records per/chunk or "batch". However, each chunk of records is part of the same execution context. If I write the following code.

Example Trigger

myAccount trigger on Account(afterInsert){

    if(TriggerHelper.runOnce == false){
        TriggerHelper.runOnce = true;
        map<id,contact> primaryContacts = new map([SELECT Id, AccountId, isPrimary 
            FROM Contact WHERE AccountId in trigger.newMap.keyset()]);
        map<Id,Id>accountIdToPrimaryContactId = new map<Id,Id>();
        for(Id contactId:primaryContacts.keyset()){
           accountIdToPrimaryContactId.put(primaryContacts.get(contactId),contactId);
        }
        // 1 query for entire Trigger execution context
        list<Task>tasksToInsert = new list<Task>();
        for(Account a:trigger.new){
            Task t = new Task();
            t.whatID = a.Id;
            t.Subject = 'Introduce New Account Owner';
            t.ActivityDate = today().addDays(7);
            t.Status = 'Not Started';
            t.WhoId = a.PrimaryContact.Id
            t.WhatId = accountIdToPrimaryContactId.get(a.Id);
            tasksToInsert.add(t);
        }
        insert tasksToInsert;
        // 1 DML Operation for entire Trigger Execution Context
    }
}

Helper Class to hold runOnce Public Static Boolean

Public Class TriggerHelper{

public static boolean runOnce = false;

}

In the above code, as you can see, runOnce is initially set to false when the trigger calls it the first time after which it immediately sets it to true. This ensures that the trigger cannot be called a second time.

If using a helper class for your trigger as you've posted in your question, your boolean needs to be implemented in an additional helper as I did above. It still works for a before trigger that includes an after trigger as below:

myAccount trigger on Account(beforeInsert,afterInsert){

    if(TriggerHelper.runOnce == false){
        TriggerHelper.runOnce = true;

        if(trigger.isbefore){
           MyObjectServices.myTriggerMethod(trigger.newMap)
        }else if(trigger.isAfter){
           MyObjectServices.myTriggerMethod(trigger.newMap)
        }
    }
}

If you wait and put the boolean in MyObjectServices, depending on the flow of the helper class, that can sometimes be problematic. In a case like yours where both the before and after use the same method, there's no issue. However, if they used different methods, you'd have a problem and would need to use a common entry point that also passed the trigger context to it in order to prevent/control the recursion. If you used an instance class as most trigger platforms do, you could locate the public static boolean in the instance of your service class since it would persist throughout the execution context. Again, depending on your code structure, it might be path dependent.

My second question: Is there any way to get the total number of records being processed in your trigger?

The best way that I know of would be to create a public static list of Ids then add each Id processed to that list in your trigger helper class. You could then get the final size of the list at the end of your transaction. However if you did that, it would only be available in your debug logs. The total number of records process should be available in your debug logs anyway if you've done something to each record that can be "tracked".

My third question: finally, how do you handle limits? Suppose you update 10,000 of one type of record. This means 50 separate "batches" within the same execution! So, suppose you follow bulkification practices perfectly, you only have 2 queries (100 total) and 3 DML statements (150 total) to work with per batch before you hit governor limits.

I think you may have some underlying misunderstandings of how the Force.com platform works which we may need to help clear up for you.

In the first trigger example I provided, regardless of how many records are processed, there will only be 1 DML statement that is executed for all of the records in the execution context. There will also only be 1 query that is executed for all of the records which will count against your limits. The platform will break up what's returned and processed into "chunks" internally. In both cases, only 1 execution of each type will be charged against your limits.

With that having been said, now is where other types of limits come into play that you need to be concerned with.

The Total number of records retrieved by SOQL queries is limited to 50,000 records for the entire execution context which applies not only to the trigger above, but also to any other triggers or Process Builders in the transaction's execution context.

You may have only run a single query in this trigger, but another query in a different trigger can also run a single query. Those two together could potentially return more than 50,000 records, causing you to exceed governor limits without exceeding the 100 SOQL query limit. It's not just the number of queries, but also the number of records throughout the transaction context that one has to be concerned with.

Be aware that if you include subqueries, each one of those subqueries counts as an additional query towards your governor limits. Its for these kinds of reasons that bulkification is so important. You don't get dinged for each time one of these is accessed while processing a "chunk" of records. Instead that's simply the way the platform chooses to break up the operations for you. You don't need to be concerned with the details of how it handles them beyond that.


The primary issue here is not related to just how many "buckets" of records are being processed during a trigger invocation, it's also related to the total number of records being processed at any given time. You can also get into situations where you might exceed the Maximum CPU time of 10,000 milliseconds or even the Maximum execution time for an Apex transaction of 10 minutes (increasingly happening with executions that also include Process Builder which is incredibly slow).

In other situations however, it could be related to the number of callouts being made or some other limit that might apply. Each one requires a different approach. That's when one begins to look at using @future calls to perform these transactions using asynchronous apex which frequently has higher limits as well as additional methods available that you can leverage to your advantage.

For reference, I recommend you look at the Salesforce Developer Limits Quick Reference, particularly the Apex Governor Limits Section which shows the limits for synchronous and asynchronous apex side by side.

5
  • Thanks for the insights! I do agree that execution time is often a larger problem with dealing in bulk, given that workflows and process builder can take an obscenely long time to run. However, you're incorrect about DML statements and queries only counting a single time during batching. See the debug logs in my edit above.
    – SFDC Neuf
    Feb 2, 2017 at 16:59
  • You're seeing DML for the number of rows that are happening in that "chunk" as data is processed, not the DML that's being charged against your trigger's helper class for the entire execution context - unless there's something unusual about the way your helper class is being called. You'd presumably have to go to the end of your debug log to see that. Your code is checking DML Rows, not getDMLStatements() and comparing with getLimitDMLStatements().
    – crmprogdev
    Feb 2, 2017 at 20:45
  • I'm checking both statements and rows - and they're both counting every time. With every chunk, it's adding to my totals. I'm not sure why the limit report at the end of the transaction would display otherwise.
    – SFDC Neuf
    Feb 2, 2017 at 21:37
  • It has to do with the context in which the statements are taken and to which the limits are applied. Your statements as recorded are not in the complete execution context and are misleading. When looking at the summaries, those should be reported and allow you to interpret them properly. All intermediate DML statements when executed for each "batch" should only count as a single DML statement for the same operation in the entire class within that execution context. What you're looking at doesn't reflect that.
    – crmprogdev
    Feb 3, 2017 at 0:06
  • 1
    I agree that they "should" count as a single DML statement or a single query. And there's no documentation to suggest otherwise. I was under the same misconception until I did a bit of digging. What's happening above is the trigger is running the same logic 4 times. And each time, the DML statements, DML rows, queries, and query rows are all accumulating. I just ran it again and looked at the final limits counts: 5 queries, 308 query rows, 5 DML statements, 308 DML rows. Do a test yourself, and if you get conflicting results, please send me your code!
    – SFDC Neuf
    Feb 3, 2017 at 16:50
-1

This is a very interesting problem, which I previously haven't paying attention to before, so I may have created a lot of issues for different company already.

Here is my solution to all your question hopefully that can inspire you.

Question1: the best way to handle this

my through is to provide another layer of abstraction on top of SObject which include counters to count against times been called in beforeInsert, afterInsert etc.

we are using Enterprise Design Patterns which means we can handle it on Domain layer, but if you are not using it, there no harm to create a extra layer

Question2: total count

for the total number you can create a static variable to count, which is similar to some of the previous suggestion before this post.

Question3: DML Limit/Other Limit

This should be answered for why do you want to handle it in here. If you are running into this issue maybe you should ask yourself about why not use Batch? because directly DML over a large number of record is not something salesforce been designed for.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .