Command line data loader incremental/Delta backup

Question

I have one requirement based on command line data loader but with incremental or delta backup.I need to write data on files which is located to other external system.Could you please help me in achieving this requirement.I am aware of Command line data loader but writing data on some other system with incremental backup i have never tried.

Answer 1

Data loader itself does not support incremental/delta backup/extract from SFDC, but there are various ways to achieve it with or without data loader.

Option 1: add extra condition to your query in process-conf.xml <entry key="sfdc.extractionSOQL" value="Select Id, Name FROM Account where LastModifiedDate >= YESTERDAY"/>
This works, but not very reliable because you have to make sure that your data loader process runs once a day (no more, no less), which may not always be possible.

Option 2: process-conf.xml template with pre-processor shell script

create process-conf.xml-template and instead of SQL constant use a placeholder, e.g. <entry key="sfdc.extractionSOQL" value="Select Id, Name FROM Account where LastModifiedDate >= #LAST_EXTRACT_DATE#”/> Create a shell script which copies process-conf.xml-template to process-conf.xml and replaces #LAST_EXTRACT_DATE# with actual date/time when last process was run (you will have to track that date yourselves), and then runs actual data loader process.

Option 3: not data loader
Your requirement is Data Loader, but if you are open to other options then you may want to consider one of existing tools designed specifically for this task. For example command line tool https://github.com/neowit/backup-force.com. Here is what its minimum config looks like:

sf.username = [email protected]
sf.password = password-and-token
sf.serverurl = https://login.salesforce.com
backup.objects=Account, Contact
# extract all records with LastModifiedDate >= 'date of last retrieval', i.e. Incremental extract
backup.global.where=LastModifiedDate >= $Object.LastModifiedDate