2

I need to integrate a custom force.com app with data feeds that should be scheduled to run nightly and upsert data into different custom objects. I need to create a nightly schedule with dependency management. For example, "Do not load file b until file a has completed, etc."

I also need some alerting capability that notifies individuals when the scheduled job has completed and with job details (#success/#error). There would also need to be error logging by record/row similar to the logs that the Apex Data Loader and Jitterbit provide.

The most important requirement is that the credentials for the "running user" should not be stored in a third-party solution like Jitterbit's server.

Is there any Salesforce.com solution that will meet these requirements? Or, is the only solution to build an integraiton that calls the Salesforce.com SOAP/REST APIs?

Improve this question
6
  • Where are your files located? And when you say 'custom force.com app' do you mean native on the platform using Apex?
    – Andrew Fawcett
    Commented Feb 21, 2014 at 18:22
  • The files would be stored on the customer's internal file share/network storage. Our product is a 100% native (no external integrations or external credentials) force.com custom application. Our customer seeks a Salesforce.com-based integration that doesn't require third-party software agreements or security issues. We are an ISV partner on the appexchange (and a Financial Force customer for our internal CRM/financials).
    – Steve Richardson
    Commented Feb 22, 2014 at 20:11
  • Thanks Steve, formulating an approach here, one more question, do you have a feeling for the maximum file size?
    – Andrew Fawcett
    Commented Feb 22, 2014 at 23:04
  • The largest data file would be the HR data feed of approximately 80,000 rows of data which in CSV format is ~67,000 KB. Each of the other "feeds" would be much smaller (less than 10K records). This assumes each "load/upsert" would use the full data file, but a delta data file with changes only is acceptable if that type of logic is possible.
    – Steve Richardson
    Commented Feb 23, 2014 at 1:01
  • Please let me know if you have any additional insights, thank you for your help.
    – Steve Richardson
    Commented Feb 25, 2014 at 17:58

1 Answer 1

Reset to default
2

You can't load files from external systems from Salesforce, as it has no access to those systems. You probably could upload some files as documents area and process those using scheduled Apex, but you might hit limits depending on the size of the files.

The way that we've handled this in the past is using the Apex Data Loader in command line mode. The credentials are encrypted and stored on a local machine along with the data loader install and the file loads are configured through the dataloader process-conf.xml file. We the wrote a Java application that handled the orchestration of the loads - raising errors if files hadn't appeared by a particular time, starting load jobs according to a schedule using the quartz scheduler.

You could do this in a more script based way by using cron (or similar) to execute a script that carries out the load if the files are present, but it gets a little more tricky to introduce dependencies that way.

Improve this answer
1
  • Thank you for your insights. My understanding of using the Apex Data Loader from the command line is that you must still run the "job" from a PC connected to the internet using a windows task manager job. I could be wrong. Ideally, for a real enterprise integration solution, there would not be any client/local PC. Also, as you mentioned, there is no solid method for data file dependency management.
    – Steve Richardson
    Commented Feb 22, 2014 at 20:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .