I am writing a data migration project to take data from an external system and map it into SFDC as a hierarchical structure of custom objects using the Bulk Api 2.
The pattern is as follows:
- Insert parents in a job
- Return job results once completed
- Update "internal" models in data migration app with SFDC ids - this is where an External Id field would be useful
- Related child objects have an in memory reference to their parents
- Insert children in a job, where a field is parentId (sfdc lookup or master detail field) populated from the Ids of the records returned in the parent insert response
- Recursively call this code to support N tiers of data
Because the bulk api doesnt guarantee the order of the results of an insert will match the order the data was passed in. I need to marry up the records from the in memory models and the sfdc response. But, I dont have an external Id field to achieve this. Nor am I the creator of the SFDC package containing the custom objects, so I can't easily add one to each custom object to help me in this situation.
Is there a "nice" way of doing this?
A dirty solution would be to put a delimiter on the end of one of the text fields on each of the child custom object instances, put the external id on there, then consume it as an identifier in the response to marry the objects up to then populate the sfdc id on the in memory objects, then strip the delimiter and external id off the string and update the record in sfdc again "clean". But I would obviously prefer not to do this as it will double the calls/time.
Another dirty solution would be to join up all properties on an in memory object to a string, and the same in the bulk api response objects, and treat that as a unique identifier. But this isnt going to guarantee uniqueness necessarily
I was mistakenly under the impression that the data in the "bulk api job complete, get me the inserted data" response would be in the same order as I sent it in, but it seem to not be the case. I appreciate that normally relying on order of data is a bad thing, but I was aware that there wasnt an existing External Id custom field on the SFDC custom objects, so was trying to work with what I had already...