0

  1. We have to load the entire Salesforce Lead data(this data might be millions records)to external system using Bulk API.

  2. Once all the data loaded and we thought to initiated to catch the added/modified/deleted records on daily basis using the Rest API /vXX.X/sobjects/SObjectName/updated/?start=startDateAndTime&end=endDateAndTime.

Could you guys suggest us a good approach & limitations to load the millions of records to external system.

Improve this question

1 Answer 1

Reset to default
0

  1. Bulk API has low limits - 10k batches over a 24-hour period with 10k records per batch. While it's certainly possible to use Bulk API to accomplish an export of a large dataset, you'll be doing a lot of work on managing the chunking of the dataset, queuing the jobs, making sure the jobs complete and so on. The Data Loader is a better out of the box tool as it does a lot of these tasks for you.

  2. Calling a REST API may run into a number of limitations. Whether these limitations will be a problem for you depends on the volume of changes to your data. Change Data Capture is a Salesforce-recommended approach for capturing these types of changes. Salesforce is moving toward event-driven architecture in a number of areas, this is one of them.

Another architectural alternative to loading the records from external system into Salesforce and then capturing the changes is to...not load them at all. You can keep the records in the external system and use the External Objects feature to make the relevant data be available in Salesforce "on the fly". External Objects is based on OData specification and implementing that will likely require middleware but it might be cheaper to maintain in the long run. Salesforce ships some connectors to popular 3rd party apps.

Improve this answer
4
  • 1
    External Objects require the Salesforce Connect feature, though, which is not cheap. Just FYI.
    – Charles T
    Commented Aug 26, 2019 at 19:49
  • @CharlesT but might be cheaper in the long run
    – identigral
    Commented Aug 26, 2019 at 20:47
  • Perhaps? But I can tell you, 3 years working for an SI partner, we have never had a customer of any size including Enterprise willing to pay for Salesforce Connect. Once they see the price tag they always opt to pay for some cheaper middleware or development instead.
    – Charles T
    Commented Aug 27, 2019 at 3:38
  • @CharlesT Yup, have witnessed a similar dynamic and our customers are all Enterprise. Conspiracy theory on why Connect is so expensive: without Connect, data resides in SF and the usage mushrooms from there.
    – identigral
    Commented Aug 27, 2019 at 3:51

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .