I've had a recurring problem where the copy of the Salesforce data in our data warehouse includes records which had been purged from Salesforce. This creates overcount in reports, undermining confidence. These are not deleted records - those I can identify using
queryAll() - rather they're marked "data not available" in the web GUI:
The data you were trying to access could not be found. It may be due to another user deleting the data or a system error. If you know the data is not deleted but cannot access it, please look at our support page.
My guess is that these records are being purged as duplicates, and that this process does not use
IsDeleted. I see a lot of messages similar to this one:
This is a duplicate campaign member contact. The primary campaign member contact is <link>
...in records which are no longer returned when I query the API.
However, there's no
IsPurged field in the records returned. How can I positively identify all records that have been removed from the live data set, if they're not marked
queryAll() doc mentions
MasterRecordID to identify records marked as duplicates. However, I only have such a field for
Lead. The disappearing records have been in the
OpportunityLineItem tables, and possibly others.
My workaround, for now, is to pull a whitelist of all live IDs created between dates X and Y, then compare that against my list of records. Any IDs in my list but not in the fresh list from Salesforce may have been purged. However, this produces false positives, so I need to validate each record in the web GUI before marking it as definitely dead.