Salesforce getUpdated() and getDeleted() consistency

Question

I am working on a backup application where there are multiple objects to be backed up. When the backup is complete, I stop the backup process and record the end time. For example, if there are about 70 objects to be backed up, the log will contain

backup_id, objName, startTime, endTime

where the startTime and endTime will be specific to the backup_id and not the objName. I wanted to know whether this approach would result in any issues for continuous backup.

In order to replicate the data from the previous backup date, I use

startTime = backup_old_endTime    
endTime=backup_curr_startTime 

Reading through the docs specified that the following fields have to be maintained for consistency: GetDeleted and GetUpdated.

1. Should I consider the fields earliestDateAvailable and latestDateCovered for this continuous backup process, i.e. make backupLog fields specific to each object?

2. The docs under getDeletedRecord mentions:

   If this value is less than endDate, the call will fail, and you should resynch your data before performing another replication.

   If I cover the endDate that is after the end date wouldn't that be a safer way? Can anyone give me an instance where it would fail?

EDIT: To make the question more clear, this is the snippet I use for updated and deleted records according to documentation:

            GetDeletedResult deletedResult = soapConnection.getDeleted(sObjName, startTime, endTime);
            Calendar earliestDate = deletedResult.getEarliestDateAvailable();
            long latestDate = deletedResult.getLatestDateCovered().getTimeInMillis();

            if (earliestDate.compareTo(endTime) < 0)
                LOGGER.error("getDeleted() may fail for " + sObjName + " as it can cover deleted records till "
                        + earliestDate.getTimeInMillis());
            LOGGER.info("Use " + latestDate + " as the start time for retrieving deleted records for " + sObjName);

The startTime and endTime is the same for all the objects when I run the application. Will this cause an issue?

EDIT2 I got the part for the complete backup covered where I did the method as explained in the answer and mentioned here.

This is what I have:

FULL:
record start_time
for every object
    run queryAll() where sysmodstamp <= start_time
    write response()
end for
record end_time
for every object
    run getUpdated(), getDeleted() with start_time, end_time
end for
for every object
    insert into log(job_id, object_name, start_time, end_time)
end for

INCREMENTAL:
get recent_backup_time
record start_time
for every object
    run getUpdated(), getDeleted() with recent_backup_time, start_time
    insert into log(job_id, object_name, recent_backup_time, start_time)
end for

My main concern is where does the getLatestDateCovered() and getEarliestDateAvailable() fit here?

Answer 1

I would expect this to be an an issue. There will likely be records at times that were updated or deleted between when your batch job began and ended that weren't returned in your initial query.

Some write their code to query for those records at the end of a job, retrieve those records, then finish backing them up before ending the process. You may want to consider doing the same.

Edit

What's returned will in part depend on how you write your query. If you use QueryAll(), your results will also include what's in the recycle bin too and has it's isDeleted flag set to true.

As I understood your question, you begin your backup at a certain time and run your query for the records on the object you want to create the backup for. At the end of the backup, you record the end time. As I understood your question, your query doesn't account for records that weren't backed up because they were changed while the previous backup was being run. That would have occurred prior to the last end time. The same will happen during each run, there will be records missed because they weren't captured by your query at the start of your batch.

What's returned by your query will be the state of the records at the time of your query. Any subsequent changes made to the records following your start time won't be reflected in your query results. As a result, the end time isn't relevant to the time range of your next backup. Only the start time of your previous back-up would seem relevant.

Unless you run a query at the end of your job for records changed between the start and end of the batch run, you won't have an up to date backup and will miss those records if you're going to use the end time as a reference.

Edit2

Okay, I understand your question now. Yes, you'll want to have the granularity for each record. What the documentation is saying in your question #2 is that if those two values don't overlap, the record won't exist for you to retrieve. When you call the getDeletedRecord method and try to query for the record it won't return a result when that's the case. If you don't track those values for each record you'd have no way of knowing if that would occur.