1

I am working on a backup application where there are multiple objects to be backed up. When the backup is complete, I stop the backup process and record the end time. For example, if there are about 70 objects to be backed up, the log will contain

backup_id, objName, startTime, endTime

where the startTime and endTime will be specific to the backup_id and not the objName. I wanted to know whether this approach would result in any issues for continuous backup.

In order to replicate the data from the previous backup date, I use

startTime = backup_old_endTime    
endTime=backup_curr_startTime 

Reading through the docs specified that the following fields have to be maintained for consistency: GetDeleted and GetUpdated.

  1. Should I consider the fields earliestDateAvailable and latestDateCovered for this continuous backup process, i.e. make backupLog fields specific to each object?

  2. The docs under getDeletedRecord mentions:

    If this value is less than endDate, the call will fail, and you should resynch your data before performing another replication.

    If I cover the endDate that is after the end date wouldn't that be a safer way? Can anyone give me an instance where it would fail?

EDIT: To make the question more clear, this is the snippet I use for updated and deleted records according to documentation:

            GetDeletedResult deletedResult = soapConnection.getDeleted(sObjName, startTime, endTime);
            Calendar earliestDate = deletedResult.getEarliestDateAvailable();
            long latestDate = deletedResult.getLatestDateCovered().getTimeInMillis();

            if (earliestDate.compareTo(endTime) < 0)
                LOGGER.error("getDeleted() may fail for " + sObjName + " as it can cover deleted records till "
                        + earliestDate.getTimeInMillis());
            LOGGER.info("Use " + latestDate + " as the start time for retrieving deleted records for " + sObjName);

The startTime and endTime is the same for all the objects when I run the application. Will this cause an issue?

EDIT2 I got the part for the complete backup covered where I did the method as explained in the answer and mentioned here.

This is what I have:

FULL:
record start_time
for every object
    run queryAll() where sysmodstamp <= start_time
    write response()
end for
record end_time
for every object
    run getUpdated(), getDeleted() with start_time, end_time
end for
for every object
    insert into log(job_id, object_name, start_time, end_time)
end for

INCREMENTAL:
get recent_backup_time
record start_time
for every object
    run getUpdated(), getDeleted() with recent_backup_time, start_time
    insert into log(job_id, object_name, recent_backup_time, start_time)
end for

My main concern is where does the getLatestDateCovered() and getEarliestDateAvailable() fit here?

1

I would expect this to be an an issue. There will likely be records at times that were updated or deleted between when your batch job began and ended that weren't returned in your initial query.

Some write their code to query for those records at the end of a job, retrieve those records, then finish backing them up before ending the process. You may want to consider doing the same.

Edit

What's returned will in part depend on how you write your query. If you use QueryAll(), your results will also include what's in the recycle bin too and has it's isDeleted flag set to true.

As I understood your question, you begin your backup at a certain time and run your query for the records on the object you want to create the backup for. At the end of the backup, you record the end time. As I understood your question, your query doesn't account for records that weren't backed up because they were changed while the previous backup was being run. That would have occurred prior to the last end time. The same will happen during each run, there will be records missed because they weren't captured by your query at the start of your batch.

What's returned by your query will be the state of the records at the time of your query. Any subsequent changes made to the records following your start time won't be reflected in your query results. As a result, the end time isn't relevant to the time range of your next backup. Only the start time of your previous back-up would seem relevant.

Unless you run a query at the end of your job for records changed between the start and end of the batch run, you won't have an up to date backup and will miss those records if you're going to use the end time as a reference.

Edit2

Okay, I understand your question now. Yes, you'll want to have the granularity for each record. What the documentation is saying in your question #2 is that if those two values don't overlap, the record won't exist for you to retrieve. When you call the getDeletedRecord method and try to query for the record it won't return a result when that's the case. If you don't track those values for each record you'd have no way of knowing if that would occur.

  • Thanks for the reply. I'm not still clear whether this would be an issue. Say an object O1 has a record o1 that is related to parent record o2 of object O2. I'm querying this between the fixed start/end time. If o1 is returned and before querying O2, o2 is deleted o2 will still be returned as the reference time is the same. However, using this during next iteration will record o1 as well as o2. Am I right? – dmachop Aug 25 '15 at 17:23
  • Can't edit the question. If I am using the static start/end time, can you explain to me whether there is any significance to using getEarliestDateAvailable() and getLatestDateCovered()? – dmachop Aug 25 '15 at 17:36
  • Edit link is at bottom left of your post. See my edited answer. – crmprogdev Aug 25 '15 at 18:13
  • I've got the full run covered. I wanted to know whether I should be storing granular information under log for every object or keep the current way for every job. See my EDIT2 for a more enhanced explanation. – dmachop Aug 25 '15 at 18:38
  • See my Edit #2. Yes, you'll want that granularity. – crmprogdev Aug 25 '15 at 19:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.