6

The DuplicateRecordItem trigger doesn't seem to be bulkified when triggered by the Automated Process user, and I'm wondering if anybody here has experienced this or might know why. This could be a Salesforce bug (or a feature that's "working as designed"), but I'm not sure.

Given this scenario: duplicate management is enabled for leads (reporting only, not blocking), and you are inserting 5 leads in bulk. A duplicate is found for each lead.

What one might expect to see is a debug log for the Automated Process user where 5 DuplicateRecordItems are being inserted. This is happening, but one-by-one in the same execution context. (By the way - if you're following along at home, make sure you know how to debug the Automated Process user: Cannot capture debug logs for Duplicate Management trigger)

My debug log looks something like this:

17:25:54.0 (672964)|CODE_UNIT_STARTED|[EXTERNAL]|TRIGGERS
17:25:54.0 (1855848)|CODE_UNIT_STARTED|[EXTERNAL]|01q36000001GnM4|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event BeforeInsert for [new]
17:25:54.0 (201951200)|CODE_UNIT_FINISHED|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event BeforeInsert for [new]
17:25:54.0 (278330109)|CODE_UNIT_STARTED|[EXTERNAL]|01q36000001GnM4|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event AfterInsert for [0GL36000001Atx8]
17:25:54.0 (595800979)|CODE_UNIT_FINISHED|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event AfterInsert for [0GL36000001Atx8]
17:25:54.0 (656795028)|CODE_UNIT_STARTED|[EXTERNAL]|01q36000001GnM4|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event BeforeInsert for [new]
17:25:54.0 (666779292)|CODE_UNIT_FINISHED|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event BeforeInsert for [new]
17:25:54.0 (696109353)|CODE_UNIT_STARTED|[EXTERNAL]|01q36000001GnM4|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event AfterInsert for [0GL36000001Atx9]
17:25:54.0 (720758546)|CODE_UNIT_FINISHED|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event AfterInsert for [0GL36000001Atx9]
17:25:54.0 (748951344)|CODE_UNIT_STARTED|[EXTERNAL]|01q36000001GnM4|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event BeforeInsert for [new]
17:25:54.0 (753501365)|CODE_UNIT_FINISHED|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event BeforeInsert for [new]
17:25:54.0 (789856188)|CODE_UNIT_STARTED|[EXTERNAL]|01q36000001GnM4|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event AfterInsert for [0GL36000001AtxA]
17:25:54.0 (812664269)|CODE_UNIT_FINISHED|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event AfterInsert for [0GL36000001AtxA]
17:25:54.0 (837110798)|CODE_UNIT_STARTED|[EXTERNAL]|01q36000001GnM4|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event BeforeInsert for [new]
17:25:54.0 (840939576)|CODE_UNIT_FINISHED|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event BeforeInsert for [new]
17:25:54.0 (874777343)|CODE_UNIT_STARTED|[EXTERNAL]|01q36000001GnM4|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event AfterInsert for [0GL36000001AtxB]
17:25:54.0 (895501197)|CODE_UNIT_FINISHED|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event AfterInsert for [0GL36000001AtxB]
17:25:54.0 (921262665)|CODE_UNIT_STARTED|[EXTERNAL]|01q36000001GnM4|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event BeforeInsert for [new]
17:25:54.0 (925065733)|CODE_UNIT_FINISHED|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event BeforeInsert for [new]
17:25:54.0 (950455245)|CODE_UNIT_STARTED|[EXTERNAL]|01q36000001GnM4|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event AfterInsert for [0GL36000001AtxC]
17:25:54.0 (971136910)|CODE_UNIT_FINISHED|LAM_DuplicateRecordItemTrigger on DuplicateRecordItem trigger event AfterInsert for [0GL36000001AtxC]
17:25:54.0 (991983991)|CODE_UNIT_FINISHED|TRIGGERS

In order to meet the requirements of the product we're building, I had to do some funkiness to get around SOQL and DML limit issues here. Because the trigger does a few SOQL queries and DML operations, I now have code in place to make sure that only 50 records of the object I'm working on are ever inserted at once. (It's not leads, thank goodness - it's an object that only my Apex code ever operates on.)

So, in summary: have you ever seen this? Do you know why Salesforce might be doing this?

  • I recommend you post this to the Ideas Exchange, as it seems like a feature that needs to be improved. – Adrian Larson Oct 18 '16 at 15:51
  • 1
    What are you using to bulk insert the leads? It shouldn't matter, but might provide some context. – Daniel Ballinger Oct 18 '16 at 23:31
  • Maybe you could use the corresponding DuplicateRecordSet to find all the DuplicateRecordItem records in one go? Might need an async call on the DuplicateRecordSetId and a static set to track visited Ids. – Daniel Ballinger Oct 18 '16 at 23:38
  • 1
    @AdrianLarson Great feedback - idea is here: success.salesforce.com/ideaView?id=0873A000000cMbCQAU – Robert Watson Oct 19 '16 at 16:34
  • @DanielBallinger The records I'm inserting are being bulk inserted through Apex via an insert call. Your suggestion of using an async call is an interesting one and is definitely possible if we decide to scrap the current solution. – Robert Watson Oct 19 '16 at 16:40
4

You are not alone! We've seen Duplicate Record Set triggers behave in this way too.

Why isn't the DuplicateRecordItem trigger bulkified?

It isn't bulkified, because handling intra-import duplicates requires letting the records go in 1-by-1.

Duplicate Management is a very clever black box, and maybe we're making some assumptions about the low level implementation details, but this is as best as we can summarize: Transaction Context ain't Execution Context when autoproc does duplicate management

Let's say we load 500 unique records into Salesforce with the Data Import Wizard. Then we try to load the same 500 records in again. As expected, the first thing that happens is the dataset gets broken down into chunks of 200 records because we used the Bulk API.

But the interesting behaviour starts when the Automated Process user runs:

  • Number of DML insert invocations? 500
  • Number of Apex autoproc execution contexts? 15
  • Number of DuplicateRecordSet update Trigger invocations? 6,47,6,6,47,47,47,6,47,47,6,47,47,47,47
  • Number of records or trigger.size in each trigger invocation? 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1...

Key takeaway: the number of records we insert has little bearing on the number of records handled by duplicate management. Autoproc makes up its own execution contexts and its own transactions.

It's by design, some thought experiments help to appreciate why Salesforce do it this nuanced way:

  • Q1) How to manage dupe A already in the database with dupe B already in the database?

  • Q2) How to manage dupe A already in the database with dupe C approaching the database?

  • Q3) how to manage dupe D approaching the database with dupe E approaching the database?

And the answers:

  • A1) Pretty straightforward: they can persist a Duplicate Record Set and Duplicate Record Items.

  • A2) Also straightforward: they can throw that DmlException if you want handle it synchronously.

  • A3) Hang on. Neither record exists in the database. We call these intra-import duplicates.

Life gets very difficult with Q3 because there is no handle on the record. But by allowing the duplicate identifications to occur one-by-one, there is no Q3 scenario. Everything falls under Q1 or Q2.

How can we programmatically handle duplicates on a one-by-one basis?

We go with an Apex Trigger on the Duplicate Record Set:

  • is the set being updated?
  • is new RecordCount > old RecordCount?

If both of the above are a "yes" it's time to handle dupes! Maybe this works better with a picture. Green is insert trigger. Yellow is update trigger. Red is delete trigger.

(A) is the user creating (for example) a new Lead record

(B) autoproc creates a Duplicate Record Set having identified duplicates

(C) autoproc creates a Duplicate Record Item which points to the duplicate record

(D) an Item was created, which causes the RecordCount field to change on the parent Set

(E) you carefully implement logic (in the yellow diamond) to (eg) delete the duplicate records

duplicate-management-sequence-diagram

  • 1
    This was fantastic - thanks for this! Glad to know we aren't alone. I've also seen the oddness with the Trigger invocations - since we are limiting ourselves to 50 records, we're seeing two different debug logs: one with 47 records and one with 3 records. So that corresponds to the 47/6 pattern that you are seeing. The inter-import explanation makes total sense - that seems likely to be the reason why the autoproc is operating in this way. – Robert Watson Nov 8 '16 at 20:40
3
+25

Expanding on my comment as a possible solution:

Rather than use a trigger on the DuplicateRecordItem records, instead use one on the parent DuplicateRecordSet records that will be connected to the items via the DuplicateRecordSetId field.

The trick where will be that the DuplicateRecordSet will need to be inserted before any of the child records. As such, you couldn't immedatiely use the trigger to work with the child records.

Instead you could call an Asynchronous Apex method with the DuplicateRecordSet Id. This could be a @future method, or queueable Apex, batch apex, scheduled apex. The important part is that by the time the async method is called the child DuplicateRecordItem records would now be available to query in one go.

A batch job might be a good candicate here as you can also adjust the scope of each execution to stay within your SOQL and DML limits.

It would be a good idea to keep a static set of visited DuplicateRecordSet id's within the trigger implementation to ensure that you only process each ID once.


If the DuplicateRecordSet trigger isn't going to fire on every transaction then you could still use the triggers on DuplicateRecordItem to identify the DuplicateRecordSetId. The issue here would be if you need to distinguish between the old and new DuplicateRecordItem's. You could sort them by CreatedDate and ignore anything older then the most recent records.

  • 1
    One caveat with going with the trigger on the Duplicate Record Set approach is that Duplicate Record Sets are not created or updated every time a duplicate is found. For example, if you create 'Lead A' and it's found to be a duplicate with 'Lead B', and then some time later (an hour, a day, whatever), you create 'Lead C' which is also a duplicate of 'Lead B', a new Duplicate Record Item is inserted and related to the same Duplicate Record Set. So as far as I know (and I haven't yet confirmed), no Duplicate Record Set triggers fire. Side note: on vaca until 11/1 - will followup on this later – Robert Watson Oct 20 '16 at 14:55
  • @RobertWatson Interesting. I've added a few more thoughts on how to work around this. – Daniel Ballinger Oct 20 '16 at 18:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.