Dealing with auto-batching of triggers

Question

I've recently been trying to come to terms with the oddities of the auto-"batching" of triggers. What I'm referring to is how when dealing with more than 200 records in a trigger execution, Salesforce will automatically "batch" the records into groups of 200. For example, if I run a trigger on 500 Account records, the logic in the trigger will be done three times in succession, on the first 200 records, then the next 200, then the last 100. Your logic will run 3 times in succession, and governor limits are never reset.

This doesn't pose much of a problem in a small-scale operation, but in working in a Salesforce instance with thousands of users, large quantities of data, and extensive automation, these issues are bound to come up.

My first question: what is the best practice to handle a trigger firstRun variable? Since workflows can often cause a trigger to run twice, it's important to pass through pieces of logic only the first time. As for order of execution, I've determined that the first 200 records will move through the trigger, then workflow, then the trigger again. Then the next "batch" will be processed, and so on. Because of this predictable order, my best approach is this (I simplified the overarching structure to demonstrate the logic):

public class MyObjectServices {
    public static Set<Id> recordsProcessed = new Set<Id>();

    public static void myTriggerMethod(Map<Id, MyObject__c> newMap){
        //The number of records processed in all "batches" to this point
        Integer sizeBefore = recordsProcessed.size();

        //Add the ids all records being processed this "batch". If they've already
        //been processed, the set will prevent duplicates from being added
        for (Id key : newMap.keySet()){
            recordsProcessed.add(key);
        }

        //Determines if the records included in the current "batch" of 200 
        //have been processed before
        if (recordsProcessed.size() != sizeBefore){
            //Trigger logic
        }
    }
}

However, this 1) doesn't work for before insert since the id doesn't yet exist, and 2) feels legitmately hacky. Are there any better ways to do this?

My second question: Is there any way to get the total number of records being processed in your trigger? Trigger.size will only give you the number being processed in that particular batch. We could use the recordsProcessed variable from above, but that will only give you the total number during the last "batch" of the trigger. And there's no way of knowing which batch that will be 100% of the time! Yes, if Trigger.size != 200, you know it's the last batch, but what if you're dealing with an exact multiple of 200?

My third question: finally, how do you handle limits? Suppose you update 10,000 of one type of record. This means 50 separate "batches" within the same execution! So, suppose you follow bulkification practices perfectly, you only have 2 queries (100 total) and 3 DML statements (150 total) to work with per batch before you hit governor limits. This is straight-up unworkable. All ideas I can come up with involve incredibly over-engineered methods of separating records into groups of 200 or less and using Queueables to process them. But this makes it very difficult to do any validation, and of course any implementation would be very complicated, so a dedicated technical architect would need to furiously review all incoming code.

So, do any of you have any insight how to face these problems in doing large-scale trigger automation?

EDIT: Adding an example to clear up some misconceptions about how trigger "batching" affects limits and static variables. Trigger:

trigger FooTrigger on Foo__c (before update) {

    System.debug('Trigger before handler call');
    FooTriggerHandler fth = new FooTriggerHandler(
        trigger.oldMap,
        trigger.newMap,
        trigger.old,
        trigger.new,
        trigger.isInsert,
        trigger.isUpdate,
        trigger.isDelete,
        trigger.isUndelete,
        trigger.isBefore,
        trigger.isAfter,
        trigger.size
    );
    System.debug('Trigger after handler call');
}

Apex class:

public class FooTriggerHandler {

    public static Set<Id> allRecordIds = new Set<Id>();
    public static Boolean firstRun = true;
    public static Integer count = 0;

    public FooTriggerHandler(Map<Id,Foo__c> oldMap, Map<Id,Foo__c> newMap, List<Foo__c> triggerOld, List<Foo__c> triggerNew, 
    Boolean isInsert, Boolean isUpdate, Boolean isDelete, Boolean isUndelete, Boolean isBefore, Boolean isAfter, Integer size){
        System.debug('Trigger.size: ' + size);
        System.debug('Entering dispatcher constructor');
        System.debug('DML before: ' + Limits.getDmlStatements());
        System.debug('DML rows before: ' + Limits.getDmlRows());
        System.debug('Queries before: ' + Limits.getQueries());
        System.debug('Query rows before: ' + Limits.getQueryRows());
        System.debug('Firstrun: ' + firstRun);
        firstRun = false;
        System.debug('Record Ids before: ' + allRecordIds.size());
        List<User> someUsers = [SELECT Id FROM User LIMIT 2];
        for (Foo__c f : triggerNew){
           allRecordIds.add(f.Id);
        }
        System.debug('Record Ids after: ' + allRecordIds.size());
        Contact contact1 = new Contact(LastName = 'TestContact' + count++);
        Contact contact2 = new Contact(LastName = 'TestContact' + count++);
        insert new List<Contact>{contact1, contact2};
        System.debug('DML after: ' + Limits.getDmlStatements());
        System.debug('DML rows after: ' + Limits.getDmlRows());
        System.debug('Queries after: ' + Limits.getQueries());
        System.debug('Query rows after: ' + Limits.getQueryRows());
    }

}

Meanwhile, I have a workflow rule running an update every time I update a Foo__c (in order to demonstrate the trigger re-running).

I run the following anonymous code: update [SELECT Id FROM Foo__c LIMIT 300];

And my debug logs:

//First trigger batch, first run
09:18:00:143 USER_DEBUG [3]|DEBUG|Trigger before handler call
09:18:00:144 USER_DEBUG [9]|DEBUG|Trigger.size: 200
09:18:00:144 USER_DEBUG [10]|DEBUG|Entering dispatcher constructor
09:18:00:144 USER_DEBUG [11]|DEBUG|DML before: 1
09:18:00:144 USER_DEBUG [12]|DEBUG|DML rows before: 300
09:18:00:144 USER_DEBUG [13]|DEBUG|Queries before: 1
09:18:00:144 USER_DEBUG [14]|DEBUG|Query rows before: 300
09:18:00:144 USER_DEBUG [15]|DEBUG|Firstrun: true
09:18:00:144 USER_DEBUG [17]|DEBUG|Record Ids before: 0
09:18:00:391 USER_DEBUG [22]|DEBUG|Record Ids after: 200
09:18:01:161 USER_DEBUG [26]|DEBUG|DML after: 2
09:18:01:161 USER_DEBUG [27]|DEBUG|DML rows after: 302
09:18:01:161 USER_DEBUG [28]|DEBUG|Queries after: 2
09:18:01:161 USER_DEBUG [29]|DEBUG|Query rows after: 302
09:18:01:161 USER_DEBUG [17]|DEBUG|Trigger after handler call
//First trigger batch, after workflow
09:18:01:798 USER_DEBUG [3]|DEBUG|Trigger before handler call
09:18:01:799 USER_DEBUG [9]|DEBUG|Trigger.size: 200
09:18:01:799 USER_DEBUG [10]|DEBUG|Entering dispatcher constructor
09:18:01:799 USER_DEBUG [11]|DEBUG|DML before: 2
09:18:01:799 USER_DEBUG [12]|DEBUG|DML rows before: 302
09:18:01:799 USER_DEBUG [13]|DEBUG|Queries before: 2
09:18:01:799 USER_DEBUG [14]|DEBUG|Query rows before: 302
09:18:01:799 USER_DEBUG [15]|DEBUG|Firstrun: false
09:18:01:799 USER_DEBUG [17]|DEBUG|Record Ids before: 200
09:18:01:890 USER_DEBUG [22]|DEBUG|Record Ids after: 200
09:18:01:952 USER_DEBUG [26]|DEBUG|DML after: 3
09:18:01:952 USER_DEBUG [27]|DEBUG|DML rows after: 304
09:18:01:952 USER_DEBUG [28]|DEBUG|Queries after: 3
09:18:01:952 USER_DEBUG [29]|DEBUG|Query rows after: 304
09:18:01:952 USER_DEBUG [17]|DEBUG|Trigger after handler call
//Second trigger batch, first run
09:18:02:623 USER_DEBUG [3]|DEBUG|Trigger before handler call
09:18:02:623 USER_DEBUG [9]|DEBUG|Trigger.size: 100
09:18:02:623 USER_DEBUG [10]|DEBUG|Entering dispatcher constructor
09:18:02:624 USER_DEBUG [11]|DEBUG|DML before: 3
09:18:02:624 USER_DEBUG [12]|DEBUG|DML rows before: 304
09:18:02:624 USER_DEBUG [13]|DEBUG|Queries before: 3
09:18:02:624 USER_DEBUG [14]|DEBUG|Query rows before: 304
09:18:02:624 USER_DEBUG [15]|DEBUG|Firstrun: false
09:18:02:624 USER_DEBUG [17]|DEBUG|Record Ids before: 200
09:18:02:624 USER_DEBUG [22]|DEBUG|Record Ids after: 300
09:18:02:624 USER_DEBUG [26]|DEBUG|DML after: 4
09:18:02:624 USER_DEBUG [27]|DEBUG|DML rows after: 306
09:18:02:624 USER_DEBUG [28]|DEBUG|Queries after: 4
09:18:02:624 USER_DEBUG [29]|DEBUG|Query rows after: 306
09:18:02:624 USER_DEBUG [17]|DEBUG|Trigger after handler call
//Second trigger batch, after workflow
09:18:03:180 USER_DEBUG [3]|DEBUG|Trigger before handler call
09:18:03:180 USER_DEBUG [9]|DEBUG|Trigger.size: 100
09:18:03:180 USER_DEBUG [10]|DEBUG|Entering dispatcher constructor
09:18:03:180 USER_DEBUG [11]|DEBUG|DML before: 4
09:18:03:180 USER_DEBUG [12]|DEBUG|DML rows before: 306
09:18:03:180 USER_DEBUG [13]|DEBUG|Queries before: 4
09:18:03:180 USER_DEBUG [14]|DEBUG|Query rows before: 306
09:18:03:180 USER_DEBUG [15]|DEBUG|Firstrun: false
09:18:03:180 USER_DEBUG [17]|DEBUG|Record Ids before: 300
09:18:03:511 USER_DEBUG [22]|DEBUG|Record Ids after: 300
09:18:03:603 USER_DEBUG [26]|DEBUG|DML after: 5
09:18:03:604 USER_DEBUG [27]|DEBUG|DML rows after: 308
09:18:03:604 USER_DEBUG [28]|DEBUG|Queries after: 5
09:18:03:604 USER_DEBUG [29]|DEBUG|Query rows after: 308
09:18:03:604 USER_DEBUG [17]|DEBUG|Trigger after handler call

Observations:

1. Using a static first run variable only affects the first 200 records. The variable remains false when processing any additional records. Only use this when you legitimately want code to run exactly one time, not once with each record.
2. Limits are not reset between batches, nor between re-runs due to workflow. You could technically query the same 500 records each time, and they would count repeatedly toward your query rows, potentially causing you to hit limits without actually working in bulk.

Answer 1

However, this 1) doesn't work for before insert since the id doesn't yet exist, and 2) feels legitmately hacky. Are there any better ways to do this?

Your approach (Set<Id> recordsProcessed) is a good one. You do not have to prevent before insert trigger recursion, so it's really not an issue that records do not have an Id yet in that case. I don't find it to be "hacky", and this approach is more robust than a simple Boolean flag (which will only operate correctly on the first batch).

Is there any way to get the total number of records being processed in your trigger?

I don't believe so, unless you set it from the calling context. For instance, you could do something like:

public with sharing class LeadService
{
    public static Integer recordsToProcess = 0;

    // service methods
}

/*VVV calling context VVV*/
List<Lead> toUpdate; // = <some_list>
LeadService.recordsToProcess = toUpdate.size();
update recordsToProcess;

However, you might have futher updates to leads with different values. I would avoid this strategy and find other ways around this limitation. It shouldn't matter which batch comes last. If you want to make sure some logic happens after your trigger logic completes, consider asynchronous processing.

how do you handle limits?

Two common strategies for easing limits usage are to:

• Use asynchronous processing for heavy lifting
• Use lazy loading to re-use common data

With the former strategy, you can trade queries/dml/cpu for async calls. It can be more difficult to prevent trigger recursion, but you should be able to work around it by careful application of criteria (filters).

The latter can help when you have configuration data you will need in all your batches. It would look something like:

public static List<ConfigObject> configData
{
    get
    {
        if (configData == null)
            configData = [/*query*/];
        return configData;
    }
    private set;
}

Answer 2

My first question: what is the best practice to handle a trigger firstRun variable? Since workflows can often cause a trigger to run twice, it's important to pass through pieces of logic only the first time. As for order of execution, I've determined that the first 200 records will move through the trigger, then workflow, then the trigger again.

A trigger will process records in chunks of at most 200 records per/chunk or "batch". However, each chunk of records is part of the same execution context. If I write the following code.

Example Trigger

myAccount trigger on Account(afterInsert){

    if(TriggerHelper.runOnce == false){
        TriggerHelper.runOnce = true;
        map<id,contact> primaryContacts = new map([SELECT Id, AccountId, isPrimary 
            FROM Contact WHERE AccountId in trigger.newMap.keyset()]);
        map<Id,Id>accountIdToPrimaryContactId = new map<Id,Id>();
        for(Id contactId:primaryContacts.keyset()){
           accountIdToPrimaryContactId.put(primaryContacts.get(contactId),contactId);
        }
        // 1 query for entire Trigger execution context
        list<Task>tasksToInsert = new list<Task>();
        for(Account a:trigger.new){
            Task t = new Task();
            t.whatID = a.Id;
            t.Subject = 'Introduce New Account Owner';
            t.ActivityDate = today().addDays(7);
            t.Status = 'Not Started';
            t.WhoId = a.PrimaryContact.Id
            t.WhatId = accountIdToPrimaryContactId.get(a.Id);
            tasksToInsert.add(t);
        }
        insert tasksToInsert;
        // 1 DML Operation for entire Trigger Execution Context
    }
}

Helper Class to hold runOnce Public Static Boolean

Public Class TriggerHelper{

public static boolean runOnce = false;

}

In the above code, as you can see, runOnce is initially set to false when the trigger calls it the first time after which it immediately sets it to true. This ensures that the trigger cannot be called a second time.

If using a helper class for your trigger as you've posted in your question, your boolean needs to be implemented in an additional helper as I did above. It still works for a before trigger that includes an after trigger as below:

myAccount trigger on Account(beforeInsert,afterInsert){

    if(TriggerHelper.runOnce == false){
        TriggerHelper.runOnce = true;

        if(trigger.isbefore){
           MyObjectServices.myTriggerMethod(trigger.newMap)
        }else if(trigger.isAfter){
           MyObjectServices.myTriggerMethod(trigger.newMap)
        }
    }
}

If you wait and put the boolean in MyObjectServices, depending on the flow of the helper class, that can sometimes be problematic. In a case like yours where both the before and after use the same method, there's no issue. However, if they used different methods, you'd have a problem and would need to use a common entry point that also passed the trigger context to it in order to prevent/control the recursion. If you used an instance class as most trigger platforms do, you could locate the public static boolean in the instance of your service class since it would persist throughout the execution context. Again, depending on your code structure, it might be path dependent.

My second question: Is there any way to get the total number of records being processed in your trigger?

The best way that I know of would be to create a public static list of Ids then add each Id processed to that list in your trigger helper class. You could then get the final size of the list at the end of your transaction. However if you did that, it would only be available in your debug logs. The total number of records process should be available in your debug logs anyway if you've done something to each record that can be "tracked".

My third question: finally, how do you handle limits? Suppose you update 10,000 of one type of record. This means 50 separate "batches" within the same execution! So, suppose you follow bulkification practices perfectly, you only have 2 queries (100 total) and 3 DML statements (150 total) to work with per batch before you hit governor limits.

I think you may have some underlying misunderstandings of how the Force.com platform works which we may need to help clear up for you.

In the first trigger example I provided, regardless of how many records are processed, there will only be 1 DML statement that is executed for all of the records in the execution context. There will also only be 1 query that is executed for all of the records which will count against your limits. The platform will break up what's returned and processed into "chunks" internally. In both cases, only 1 execution of each type will be charged against your limits.

With that having been said, now is where other types of limits come into play that you need to be concerned with.

The Total number of records retrieved by SOQL queries is limited to 50,000 records for the entire execution context which applies not only to the trigger above, but also to any other triggers or Process Builders in the transaction's execution context.

You may have only run a single query in this trigger, but another query in a different trigger can also run a single query. Those two together could potentially return more than 50,000 records, causing you to exceed governor limits without exceeding the 100 SOQL query limit. It's not just the number of queries, but also the number of records throughout the transaction context that one has to be concerned with.

Be aware that if you include subqueries, each one of those subqueries counts as an additional query towards your governor limits. Its for these kinds of reasons that bulkification is so important. You don't get dinged for each time one of these is accessed while processing a "chunk" of records. Instead that's simply the way the platform chooses to break up the operations for you. You don't need to be concerned with the details of how it handles them beyond that.

---

The primary issue here is not related to just how many "buckets" of records are being processed during a trigger invocation, it's also related to the total number of records being processed at any given time. You can also get into situations where you might exceed the Maximum CPU time of 10,000 milliseconds or even the Maximum execution time for an Apex transaction of 10 minutes (increasingly happening with executions that also include Process Builder which is incredibly slow).

In other situations however, it could be related to the number of callouts being made or some other limit that might apply. Each one requires a different approach. That's when one begins to look at using @future calls to perform these transactions using asynchronous apex which frequently has higher limits as well as additional methods available that you can leverage to your advantage.

For reference, I recommend you look at the Salesforce Developer Limits Quick Reference, particularly the Apex Governor Limits Section which shows the limits for synchronous and asynchronous apex side by side.

Answer 3

This is a very interesting problem, which I previously haven't paying attention to before, so I may have created a lot of issues for different company already.

Here is my solution to all your question hopefully that can inspire you.

Question1: the best way to handle this

my through is to provide another layer of abstraction on top of SObject which include counters to count against times been called in beforeInsert, afterInsert etc.

we are using Enterprise Design Patterns which means we can handle it on Domain layer, but if you are not using it, there no harm to create a extra layer

Question2: total count

for the total number you can create a static variable to count, which is similar to some of the previous suggestion before this post.

Question3: DML Limit/Other Limit

This should be answered for why do you want to handle it in here. If you are running into this issue maybe you should ask yourself about why not use Batch? because directly DML over a large number of record is not something salesforce been designed for.