I am writing a data migration project to take data from an external system and map it into SFDC as a hierarchical structure of custom objects using the Bulk Api 2.

The pattern is as follows:

  1. Insert parents in a job
  2. Return job results once completed
  3. Update "internal" models in data migration app with SFDC ids - this is where an External Id field would be useful
  4. Related child objects have an in memory reference to their parents
  5. Insert children in a job, where a field is parentId (sfdc lookup or master detail field) populated from the Ids of the records returned in the parent insert response
  6. Recursively call this code to support N tiers of data

Because the bulk api doesnt guarantee the order of the results of an insert will match the order the data was passed in. I need to marry up the records from the in memory models and the sfdc response. But, I dont have an external Id field to achieve this. Nor am I the creator of the SFDC package containing the custom objects, so I can't easily add one to each custom object to help me in this situation.

Is there a "nice" way of doing this?

A dirty solution would be to put a delimiter on the end of one of the text fields on each of the child custom object instances, put the external id on there, then consume it as an identifier in the response to marry the objects up to then populate the sfdc id on the in memory objects, then strip the delimiter and external id off the string and update the record in sfdc again "clean". But I would obviously prefer not to do this as it will double the calls/time.

Another dirty solution would be to join up all properties on an in memory object to a string, and the same in the bulk api response objects, and treat that as a unique identifier. But this isnt going to guarantee uniqueness necessarily

I was mistakenly under the impression that the data in the "bulk api job complete, get me the inserted data" response would be in the same order as I sent it in, but it seem to not be the case. I appreciate that normally relying on order of data is a bad thing, but I was aware that there wasnt an existing External Id custom field on the SFDC custom objects, so was trying to work with what I had already...

  • You've answered your own question. There's no default external ID in a custom object and there's no ordering. Thus, you need you own logical key that is going to be a composite of 1 or more fields. You should know what fields are unique in your dataset. If 2 records with N fields are duplicate (all N fields have the same values), surely a record ID or system metadata such as record create date would be enough to add uniqueness when composing a key. – identigral Jun 15 at 14:37
  • i did wonder - i was just confirming my suspisions i guess. date fields is a good idea, but some of them are just dates, not date times... ill see what i can do.. – Mike Sowerbutts Jun 15 at 14:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.