1

We've implemented an ETL extraction out of SFDC using the "Change data capture" mechanism with SFDC as the Data Master (described here).

We do daily delta extractions using a filter on systemmodstamp and using the extract_all functionality of DataLoader CLI so we also include records with IsDeleted = true.

However we've noticed that sometimes deleted records are not returned as part of the ETL query result. In most cases they are, but sometimes they're not.

I suspect this has to do with many records being deleted at once, thus forcing a purge of the Recycle Bin within the day. This would mean that our end-of-day delta query will not pick up those deletes and the data will stay in our BICC system forever (as it will never be updated/deleted from SFDC again).

(Some background info on the recycle bin process here, by the way)

So the question is:

is there any way that we can make the delta queries in this daily ETL extraction process 100% reliable?

Can I force deleted records to stay in the Recycle Bin for at least 24 hrs? Or should we do a full database dump each night? (I hope not..) We can't be the first company struggling with this?

2
  • Hi Guy! I've linked one of my earlier answers that may be what you're looking for. Unlike SOQL, the Replication API guarantees results for up to 30 days, even in the event of so-called "hard deletes." It's intended to be used for your exact scenario-- to mirror data to a third-party system.
    – sfdcfox
    Commented Nov 15, 2016 at 22:08
  • 1
    spot on indeed. Too bad I didn't find it, would have saved me some typing :-) Commented Nov 15, 2016 at 22:29

0

Browse other questions tagged .