2

I need to integrate a custom force.com app with data feeds that should be scheduled to run nightly and upsert data into different custom objects. I need to create a nightly schedule with dependency management. For example, "Do not load file b until file a has completed, etc."

I also need some alerting capability that notifies individuals when the scheduled job has completed and with job details (#success/#error). There would also need to be error logging by record/row similar to the logs that the Apex Data Loader and Jitterbit provide.

The most important requirement is that the credentials for the "running user" should not be stored in a third-party solution like Jitterbit's server.

Is there any Salesforce.com solution that will meet these requirements? Or, is the only solution to build an integraiton that calls the Salesforce.com SOAP/REST APIs?

  • Where are your files located? And when you say 'custom force.com app' do you mean native on the platform using Apex? – Andrew Fawcett Feb 21 '14 at 18:22
  • The files would be stored on the customer's internal file share/network storage. Our product is a 100% native (no external integrations or external credentials) force.com custom application. Our customer seeks a Salesforce.com-based integration that doesn't require third-party software agreements or security issues. We are an ISV partner on the appexchange (and a Financial Force customer for our internal CRM/financials). – Steve Richardson Feb 22 '14 at 20:11
  • Thanks Steve, formulating an approach here, one more question, do you have a feeling for the maximum file size? – Andrew Fawcett Feb 22 '14 at 23:04
  • The largest data file would be the HR data feed of approximately 80,000 rows of data which in CSV format is ~67,000 KB. Each of the other "feeds" would be much smaller (less than 10K records). This assumes each "load/upsert" would use the full data file, but a delta data file with changes only is acceptable if that type of logic is possible. – Steve Richardson Feb 23 '14 at 1:01
  • Please let me know if you have any additional insights, thank you for your help. – Steve Richardson Feb 25 '14 at 17:58
2

You can't load files from external systems from Salesforce, as it has no access to those systems. You probably could upload some files as documents area and process those using scheduled Apex, but you might hit limits depending on the size of the files.

The way that we've handled this in the past is using the Apex Data Loader in command line mode. The credentials are encrypted and stored on a local machine along with the data loader install and the file loads are configured through the dataloader process-conf.xml file. We the wrote a Java application that handled the orchestration of the loads - raising errors if files hadn't appeared by a particular time, starting load jobs according to a schedule using the quartz scheduler.

You could do this in a more script based way by using cron (or similar) to execute a script that carries out the load if the files are present, but it gets a little more tricky to introduce dependencies that way.

| improve this answer | |
  • Thank you for your insights. My understanding of using the Apex Data Loader from the command line is that you must still run the "job" from a PC connected to the internet using a windows task manager job. I could be wrong. Ideally, for a real enterprise integration solution, there would not be any client/local PC. Also, as you mentioned, there is no solid method for data file dependency management. – Steve Richardson Feb 22 '14 at 20:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.