13

Ok after hours of searching I cannot believe that something that would seem like an every day need does not have any videos, tutorials, etc directly related to it.

Heroku is tough for me as it seems so hard to get even the simplest app up and running (My issue, kinda like when one first starts learning SF, it is a steep learning curve as it is not the concept that is difficult but all the thing you have to take into consideration).

Any way, we now have

  • External Objects - GREAT - Easy - once data is in there
  • Lightning Connect -> Not good for this as data will be removed from SF and from what I can tell it is a two way sync only so if data is deleted from SF it is deleted from Heroku

All I am trying to accomplish is to:

  1. Use batch apex / loader to push old Custom Object (2 years or older) to, for now, a Heroku Postgres Database
  2. Delete the data from Salesforce
  3. Use remote objects to access the data.

2 and 3 are easy BUT my question is:

What is the best way to get the data to the Heroku postgres database?

Options That I have explored:

  1. Informatica or similar Cloud Data loaders on a scheduled basis
  2. Building a complete REST API in Heroku (learning curve) to be able to use batch apex to send the data in a JSON format to Heroku

I am sure someone has tackled this already and I am looking for confirmation that the two options I have above are the two best options. Also, can anyone confirm that for this purpose that #1 is the quickest and easiest way to go? #2 is a steep curve for me as the workings of deploying an app are fraught with all kinds of issues with the CLI.

Reason behind doing the archive: Cost of data storage on SFDC

I appreciate any insight that can be offered and any amazing resources that have help you progress from that first step of trial to the big step of understanding.

To summarize, best practice for archiving data from Salesforce to a Heroku postgres database with a focus on getting the data from Salesforce TO Heroku.

  • 3
    be aware of the cost associated with lightning connect / external objects if you aren't already. – Phil B Jun 29 '15 at 17:33
  • 1
    @PhilB - Yea, thats why I posted the bounty. I "heard" is was like 4K a month. At that point is makes more sense just to pay for more storage in SF as you would need almost 40 gigs of data before the cost of lightning would be lower that the data storage costs.....Either way looking for solutions and good example of ways that the "pros" would/have tackle(d) this problem... – Eric Jun 29 '15 at 17:58
  • I would vote #2. It's not as super straightforward as number 1, but a simple API to just shove data into a data lake isn't too complicated, and Salesforce serializes just fine. You could even push the data into a CouchDB instance and not worry about schema, just push the serialized JSON and know that your Salesforce data is inherently structured. You could probably even do this from an after delete trigger for maximum archival love. – Christian Carter Jun 30 '15 at 5:12
6
+50

Definitely a common use case that Salesforce is aware of and working on solving. See this pilot: https://developer.salesforce.com/releases/release/Summer15/big+objects

If you aren't under a deadline, Waiting might be the best answer.

#safeharbor as always.

| improve this answer | |
1

It is just my opinion and may not qualify as Answer. I have used informatica to do this kind of stuff but you will still spend time mapping data and objects. Just decide on the size of data , if its too high then go for informatica. I reckon #1 is quickest way to go. Or try talend another amazing app.

I have used REST to connect to Heroku for simple applications but not for data archival. Also check out heroku connect

| improve this answer | |
1

The issue that literally killed Heroku for my organization was the lack of CRUD support with the native integration between Heroku and Salesforce. This was actually something that was "promised" to us at one point, and I feel your dilemma falls pretty close to ours.

Developing the API side on Heroku is an option, but you would be much better of using something like Informatica, Jitterbit, SnapLogic, or Boomi to move the data if you want to go for a scale-able solution. Now these things obviously cost money, so your options might be also limited to what the business is actually willing to spend on this. Note that taking this route means you don't necessarily have any dependency on Heroku anymore and could easily switch your archived storage to something like Azure or Amazon if those are more familiar to you. The end goal here should really just be to get the data out of Salesforce, and put somewhere else that Salesforce can talk to natively through OData. The only limitations there might be if you are storing sensitive data where you might be forced into Postres, but you should know your limitations on that side already.

| improve this answer | |
0

Same problem we have solved by running a scheduler app which is deployed on Heroku and runs on every day basis and hit Salesforce through REST APIs. We have authenticated our app from Salesforce using Oauth2. Once we get the response from Salesforce we processed it and save it to Postgres database.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.