Dataloader CLI export records modified since process last run date

Question

What

I am trying to use Dataloader CLI to export Contacts created or modified since the process ran the last time.

The process runs properly, but it always retrieves the same records, and I want it to extract only the ones that have been modified (or created) since this process ran the last time.

Examples

This is how I run the process:

C:\Program Files (x86)\salesforce.com\Data Loader\bin>process.bat "C:\dataloadercli" contact-extract

This is part of the output message shown when I run the process:

Rate: 3.600.000 records per hour. Estimated time to complete: 0 minutes and 0 seconds. There are 9 successes and 0 errors. 2015-05-29 08:13:14,434 INFO [contact-extract] progress.NihilistProgressAdapter doneSuccess (NihilistProgressAdapter.java:63) - The operation has fully complet ed. There were 9 successful extractions and 0 errors.

My process-conf.xml is like this:

<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE beans PUBLIC "-//SPRING//DTD BEAN//EN" "http://www.springframework.org/dtd/spring-beans.dtd">
<beans>

<bean id="contact-extract"
          class="com.salesforce.dataloader.process.ProcessRunner"
          singleton="false">
      <description>Descarga los contactos</description>
        <property name="name" value="contact-extract"/>
        <property name="configOverrideMap">
            <map>
                <entry key="sfdc.debugMessages" value="false"/>
                <entry key="sfdc.debugMessagesFile" value="c:\\salesforce\\logs\\sfdcSoapTrace.log"/>
                <entry key="sfdc.endpoint" value="https://test.salesforce.com"/>
                <entry key="sfdc.username" value="xxxxx@xxxx"/>
                <entry key="sfdc.password" value="xxxxxxx"/>
                <entry key="process.encryptionKeyFile" value="c:\\dataloadercli\\key.txt"/>
                <entry key="sfdc.timeoutSecs" value="600"/>
                <entry key="sfdc.loadBatchSize" value="200"/>
                <entry key="sfdc.extractionRequestSize" value="500"/>
                <entry key="sfdc.extractionSOQL" value="select ID__c, MobilePhone, Birthdate FROM Contact WHERE ID__c != NULL"/>
                <entry key="process.operation" value="extract"/>
                <entry key="process.mappingFile" value="c:\\dataloadercli\\mappings\\contact-extract-map.sdl"/>
                <entry key="sfdc.entity" value="Contact"/>
                <entry key="dataAccess.type" value="csvWrite"/>
                <entry key="dataAccess.name" value="c:\\dataloadercli\\desdeSalesforce\contact-extract.csv"/>
                <entry key="process.useEuropeanDates" value="true"/>
                <entry key="dataAccess.readUTF8" value="true"/>
                <entry key="dataAccess.writeUTF8" value="true"/>
                <entry key="process.initialLastRunDate" value="2005-12-01T00:00:00.000-0800"/>
            </map>
        </property>
    </bean>

</beans>

Questions

Any idea about how can I do this??

Thanks!

Answer 1

There is no automatic solution for Apex CLI out of the box where you can use the process.lastRunDate from the properties file. However you can set a date filter like below which will limit the rows returned. The first example is where you hardcode a set datetime and the second example you can use a datetime function that is part of the SOQL.

1. Example (Pull everythng modified after midnight on August 1, 2015) select ID__c, MobilePhone, Birthdate FROM Contact WHERE ID__c != NULL AND LastModifiedDate > 2015-08-01T00:00:00.000Z

2. Example: (Pull Everything modified in the past 3 days) select ID__c, MobilePhone, Birthdate FROM Contact WHERE ID__c != NULL AND LastModifiedDate = LAST_N_DAYS:3

Answer 2

You can use @process.lastRunDate@ in the query. An example can be found on Data Loader Guide, page 34.

Answer 3

I have found that Spring Placeholder and SpEL work. While Dataloader allows for use of @process.lastRunDate@ in SQL queries, queries against some outside data source for IMPORT into Salesforce, it does not allow for @process.lastRunDate@ in SOQL, to limit data to be EXTRACTED form salesforce. However, it is noted in the Dataloader documentation that it is based on SPRING. From Dataloader documentation: The Data Loader configuration files are based on the Spring Framework, which is an open-source,full-stack Java/J2EE application framework. The Spring Framework allows you to use XML files to configure beans. Each bean represents an instance of an object; the parameters correspond to each object's setter methods. See the bean LASTRUN using PropertyPlaceholderConfigurer and the use of #{} wrapping around ${} below. ${} is placeholder replacement syntax to replace with named property from some property source (in this case a property file) and #{} is SpEL syntax, allowing for doing SUBSTR on the returned property value.

<!DOCTYPE beans PUBLIC "-//SPRING//DTD BEAN//EN" "http://www.springframework.org/dtd/spring-beans.dtd">
<beans>

    
  <!--Have not yet found a way to use LAST RUN file on a process by process basis when job is running multiple extracts-->
  <bean id="lastRun" class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer " lazy-init="default">
      <property name="location" value="../../LastRun/Export Opportunities_lastRun.properties"/>        
      <property name="ignoreResourceNotFound" value="true" />          
      <property name="ignoreUnresolvablePlaceholders" value="true" />  
      <property name="valueSeparator" value="?" />     
                                                          
  </bean> 
  
  
  <bean id="A-ExportOpportunitiesCsv" class="com.salesforce.dataloader.process.ProcessRunner" singleton="false">
    <description> *** Export(Extract) Opportunities ***"</description>
    <property name="name" value="Export Opportunities"/>
    <property name="configOverrideMap">
      <!-- Overrides and additions to values provided in config.properties -->
      <map>
        <entry key="process.operation"          value="extract"/>
        <entry key="process.outputError"        value="OpportunitiesERROR.csv"/>
        <entry key="process.outputSuccess"      value="OpportunitiesSUCCESS.csv"/>
        <entry key="dataAccess.type"            value="csvWrite"/>
        <entry key="dataAccess.name"            value="..\..\DATA\Opportunities.csv"/>
        <entry key="dataAccess.writeUTF8"       value="false"/>
        <entry key="sfdc.entity"                value="Opportunity"/>
        <entry key="sfdc.extractionRequestSize" value="2000"/>
        <entry key="sfdc.extractionSOQL" 
               value="SELECT id, CreatedDate, LastModifiedDate, SystemModstamp
                      FROM Opportunity 
                      WHERE SystemModstamp >= 
    #{ '${process.lastRunDate}'.substring(0,10)+'T00:00:00.000-0400'}
                      and  IsDeleted = FALSE
                     "/>
      </map>
    </property>
   
  </bean>