The product I'm developing is going to act like acorporate backup solution at first but there are some data processing features that will be developed later that are the real focus.

Let's say I wanted to get a full copy of every single record in, say, an Account table (followed by every other table in the instance later on). I've dealt with clients that easily have over 1.5 million records in their Account history. What is the preferred way to do this without causing governor or other limit problems with as few api calls as necessary? Speed won't be a blg deal at first, I just want to do right for the customer's instance.

This could be done either way: externally from API requests from a python app on an external server or from a script written in Apex, but the end result is the same-- every single record in my SQL database ready to be queried in bulk with a happy customer whose instance hasn't been inconvenienced.

I believe I might have a few options. Batch apex might be one of them, but then there's also the ability to create batch export jobs from the API that I believe will export a full csv that you just have to build a parcer for. There might be other directions I'm not aware of.

Since this is going to be an appexchange listing for many clients at once, NO SALESFORCE CONNECT. It needs to be done through code, not through expensive packages.

Thank you very much for your thoughts!

  • Hello @user1508232, it depends) If you have less then 2000 records you can just use RestApis, but if you have a lot more like few millions you should consider using BulkApi. I'll try find an article for APIs comparison – ytiq Aug 17 '20 at 22:21
  • I guess that article should provide more information that I could ever put in the comment) developer.salesforce.com/blogs/2015/03/… – ytiq Aug 17 '20 at 22:22
  • the SFDC backup ISVs use BulkAPI _ OwnBackup, Druva, Gearset, ... – cropredy Aug 19 '20 at 18:32

For initial, multimillion record syncs, you need to use the Bulk API.

Once you've done an initial replication, the best API for ongoing replication depends on your specific application architecture. You might want to be using the Replication API (available in REST or SOAP variants), or Change Data Capture. But the Bulk API is really the only candidate for your large-scale initial sync over API.

I believe it is unwise to attempt large-scale data syncing via functionality implemented in Apex. It is hard to do in a resilient and performant way and is highly sensitive to governor limits.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.