I'd like to build a system that can load the large volume of data for various users, with deduplication(when the records have existed in users' SFDC, the system should not insert it again) against the external ID which is the ID of the corresponding record on my side.

Two approaches come to my mind:

  1. Install the Apex class in the users' org, which is essentially an indexer that queries the ID for each record by calling out to a web service. When the indexing is completed, I can just do bulk upsert without worrying about the duplications.

  2. Pull all the data to my end using the scheduled jobs and do the dedupe by myself, then only push the net new records to the users' SFDC.

It seems that to maintain and deploy Apex class brings in extra overheads, but to have the indexer installed in user's org reduce the effort to maintain the replica of user's data.

Which one is the canonical way? Any suggestion is welcome!

  • 2
    Just use Upsert. If your records are marked with the external id field as an external id then SF will take care of all of this for you. – Jesse Milburn Apr 1 '16 at 19:07
  • @JesseMilburn thank you for the answer. The external ids are not there yet, I just wonder should I build an Apex class to populate the sobjects with that. – Yuanfei Zhu Apr 1 '16 at 22:33
  • So you are saying you have records already in SF without the external ID? – Jesse Milburn Apr 1 '16 at 23:41
  • Exactly, that's why I am thinking to build an indexer to add the external ID. – Yuanfei Zhu Apr 1 '16 at 23:47
  • How would it know what record to assign them to? How many records are you talking about? Why are the records there w/o the external ID? Sooo many questions. – Jesse Milburn Apr 1 '16 at 23:54

What seems important and relevant here is that you want your record to be the MASTER RECORD and your ID to External ID used by the orgs you'll be sending records to. Email addresses by their very nature are automatically external Ids for Contacts, so no sweat doing an upsert for Contacts regardless of whether dupes already exist in two or more orgs that have same Contact saved with different Ids and an "indexer generated external Id". You can always use the email address as the external Id in those situations.

But what do you do about the accounts they're associated with? I'm going to assume they're associated with business accounts as you didn't say if this is B2B or B2C. If B2B, the Account may or may not have the same unique email associated with the Account you can use as an external Id. The indexer generated Id in each org will be unique. That means you'll have more than one indexer generated Id you'll need to keep track of in YOUR database. Further, your record ID isn't being directly kept as the Master Record ID (indirectly, yes, but not directly).

Trying to update or resolve these external Ids if you try to merge them could be a real hassle for you and your end users/customers. As a customer, I wouldn't particularly want to deal with that hassle unless you had a way to automate it for me.

Perhaps there's a 3rd way that you've not considered which is to import their database into yours, assign a Master Record Id to it and do the same as you import data from other customers. If it exists, you merge it with existing records and push those updates using the existing Master Record Id.

For each new customer, you'll initially have the set-up hassle of bringing in their database with their Salesforce recordIds, finding matches in your database and where they exist, adding your existing Ids as the external Id. Where they don't, you create a new record and assign an external Id to it. That same database then gets exported to and imported into the your user's database which in the meantime has had the external Id field added to it. In that way, ever org gets the same external Id which corresponds to your Master Record Id. After the import is done, you can dump the temporary database of their original records as you'll have integrated it into your master database.

Is that more work for you? Yes, it would seem to me that it would be. Does it maintain a cleaner database for all? I would think it does.

From a Salesforce database standpoint, upserts are slower and create a higher load on the system than doing a push of new records for insert. That's something which is well documented. Just don't ask me to provide you with a link to it right now as I can't remember off-hand where that documentation can be found. I only know that I've seen it numerous times. I think it's in some of the dataloader related materials.

I hope this analysis is helpful to you in coming up with a solution that meets your and your customer's needs. As with most things, I don't believe there's always one "best practice" solution, instead only methods that either don't work well at all or are insecure vs methods that are of increasingly efficiency; the latter of course only applies when security is maintained. Sometimes the "ideal" simply isn't practical or affordable for most organizations, making it an unrealistic expectation or standard to hold an organization to.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.