Why isn't the DuplicateRecordItem trigger bulkified when triggered by the Automated Process user?

Question

The DuplicateRecordItem trigger doesn't seem to be bulkified when triggered by the Automated Process user, and I'm wondering if anybody here has experienced this or might know why. This could be a Salesforce bug (or a feature that's "working as designed"), but I'm not sure.

Given this scenario: duplicate management is enabled for leads (reporting only, not blocking), and you are inserting 5 leads in bulk. A duplicate is found for each lead.

What one might expect to see is a debug log for the Automated Process user where 5 DuplicateRecordItems are being inserted. This is happening, but one-by-one in the same execution context. (By the way - if you're following along at home, make sure you know how to debug the Automated Process user: Cannot capture debug logs for Duplicate Management trigger)

My debug log looks something like this:

17:25:54.0 (672964)|CODE_UNIT_STARTED|[EXTERNAL]|TRIGGERS
17:25:54.0 (1855848)|CODE_UNIT_STARTED|[EXTERNAL]|01q36000001GnM4|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event BeforeInsert for [new]
17:25:54.0 (201951200)|CODE_UNIT_FINISHED|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event BeforeInsert for [new]
17:25:54.0 (278330109)|CODE_UNIT_STARTED|[EXTERNAL]|01q36000001GnM4|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event AfterInsert for [0GL36000001Atx8]
17:25:54.0 (595800979)|CODE_UNIT_FINISHED|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event AfterInsert for [0GL36000001Atx8]
17:25:54.0 (656795028)|CODE_UNIT_STARTED|[EXTERNAL]|01q36000001GnM4|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event BeforeInsert for [new]
17:25:54.0 (666779292)|CODE_UNIT_FINISHED|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event BeforeInsert for [new]
17:25:54.0 (696109353)|CODE_UNIT_STARTED|[EXTERNAL]|01q36000001GnM4|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event AfterInsert for [0GL36000001Atx9]
17:25:54.0 (720758546)|CODE_UNIT_FINISHED|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event AfterInsert for [0GL36000001Atx9]
17:25:54.0 (748951344)|CODE_UNIT_STARTED|[EXTERNAL]|01q36000001GnM4|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event BeforeInsert for [new]
17:25:54.0 (753501365)|CODE_UNIT_FINISHED|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event BeforeInsert for [new]
17:25:54.0 (789856188)|CODE_UNIT_STARTED|[EXTERNAL]|01q36000001GnM4|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event AfterInsert for [0GL36000001AtxA]
17:25:54.0 (812664269)|CODE_UNIT_FINISHED|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event AfterInsert for [0GL36000001AtxA]
17:25:54.0 (837110798)|CODE_UNIT_STARTED|[EXTERNAL]|01q36000001GnM4|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event BeforeInsert for [new]
17:25:54.0 (840939576)|CODE_UNIT_FINISHED|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event BeforeInsert for [new]
17:25:54.0 (874777343)|CODE_UNIT_STARTED|[EXTERNAL]|01q36000001GnM4|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event AfterInsert for [0GL36000001AtxB]
17:25:54.0 (895501197)|CODE_UNIT_FINISHED|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event AfterInsert for [0GL36000001AtxB]
17:25:54.0 (921262665)|CODE_UNIT_STARTED|[EXTERNAL]|01q36000001GnM4|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event BeforeInsert for [new]
17:25:54.0 (925065733)|CODE_UNIT_FINISHED|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event BeforeInsert for [new]
17:25:54.0 (950455245)|CODE_UNIT_STARTED|[EXTERNAL]|01q36000001GnM4|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event AfterInsert for [0GL36000001AtxC]
17:25:54.0 (971136910)|CODE_UNIT_FINISHED|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event AfterInsert for [0GL36000001AtxC]
17:25:54.0 (991983991)|CODE_UNIT_FINISHED|TRIGGERS

In order to meet the requirements of the product we're building, I had to do some funkiness to get around SOQL and DML limit issues here. Because the trigger does a few SOQL queries and DML operations, I now have code in place to make sure that only 50 records of the object I'm working on are ever inserted at once. (It's not leads, thank goodness - it's an object that only my Apex code ever operates on.)

So, in summary: have you ever seen this? Do you know why Salesforce might be doing this?

Answer 1

You are not alone! We've seen Duplicate Record Set triggers behave in this way too.

Why isn't the DuplicateRecordItem trigger bulkified?

It isn't bulkified, because handling intra-import duplicates requires letting the records go in 1-by-1.

Duplicate Management is a very clever black box, and maybe we're making some assumptions about the low level implementation details, but this is as best as we can summarize: Transaction Context ain't Execution Context when autoproc does duplicate management

Let's say we load 500 unique records into Salesforce with the Data Import Wizard. Then we try to load the same 500 records in again. As expected, the first thing that happens is the dataset gets broken down into chunks of 200 records because we used the Bulk API.

But the interesting behaviour starts when the Automated Process user runs:

• Number of DML insert invocations? 500
• Number of Apex autoproc execution contexts? 15
• Number of DuplicateRecordSet update Trigger invocations? 6,47,6,6,47,47,47,6,47,47,6,47,47,47,47
• Number of records or trigger.size in each trigger invocation? 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1...

Key takeaway: the number of records we insert has little bearing on the number of records handled by duplicate management. Autoproc makes up its own execution contexts and its own transactions.

It's by design, some thought experiments help to appreciate why Salesforce do it this nuanced way:

• Q1) How to manage dupe A already in the database with dupe B already in the database?

• Q2) How to manage dupe A already in the database with dupe C approaching the database?

• Q3) how to manage dupe D approaching the database with dupe E approaching the database?

And the answers:

• A1) Pretty straightforward: they can persist a Duplicate Record Set and Duplicate Record Items.

• A2) Also straightforward: they can throw that DmlException if you want handle it synchronously.

• A3) Hang on. Neither record exists in the database. We call these intra-import duplicates.

Life gets very difficult with Q3 because there is no handle on the record. But by allowing the duplicate identifications to occur one-by-one, there is no Q3 scenario. Everything falls under Q1 or Q2.

How can we programmatically handle duplicates on a one-by-one basis?

We go with an Apex Trigger on the Duplicate Record Set:

• is the set being updated?
• is new RecordCount > old RecordCount?

If both of the above are a "yes" it's time to handle dupes! Maybe this works better with a picture. Green is insert trigger. Yellow is update trigger. Red is delete trigger.

(A) is the user creating (for example) a new Lead record

(B) autoproc creates a Duplicate Record Set having identified duplicates

(C) autoproc creates a Duplicate Record Item which points to the duplicate record

(D) an Item was created, which causes the RecordCount field to change on the parent Set

(E) you carefully implement logic (in the yellow diamond) to (eg) delete the duplicate records

Answer 2

Expanding on my comment as a possible solution:

Rather than use a trigger on the DuplicateRecordItem records, instead use one on the parent DuplicateRecordSet records that will be connected to the items via the DuplicateRecordSetId field.

The trick where will be that the DuplicateRecordSet will need to be inserted before any of the child records. As such, you couldn't immedatiely use the trigger to work with the child records.

Instead you could call an Asynchronous Apex method with the DuplicateRecordSet Id. This could be a @future method, or queueable Apex, batch apex, scheduled apex. The important part is that by the time the async method is called the child DuplicateRecordItem records would now be available to query in one go.

A batch job might be a good candicate here as you can also adjust the scope of each execution to stay within your SOQL and DML limits.

It would be a good idea to keep a static set of visited DuplicateRecordSet id's within the trigger implementation to ensure that you only process each ID once.

---

If the DuplicateRecordSet trigger isn't going to fire on every transaction then you could still use the triggers on DuplicateRecordItem to identify the DuplicateRecordSetId. The issue here would be if you need to distinguish between the old and new DuplicateRecordItem's. You could sort them by CreatedDate and ignore anything older then the most recent records.