I am implementing for an approach to integrate from Salesforce to our 3rd party system. Usually our customer has a large set of Contact data and they want to integrate to our system. Meanwhile, our system need to generate our Id and store back as an external Id into Salesforce.

When data amount is small this is quite achievable via Request & Response mode. However, when the quantity is large, Salesforce has various limits - SOQL, DML, etc. So what is the suggested approach in achieving this?


There are multiple options here that you can consider depending on the size of data involved.

  1. Heroku Connect - Uses the SOAP and Bulk APIs provided by Salesforce. While the SOAP API is limited to 50,000 records, the Bulk API is optimized for loading large sets of data making use of asynchronous processing to retrieve batches of records in parallel. It does have a limit of 5,000 calls per day so this may/ may not work based on your data volume.

  2. Data Chunking - Chunking your data into smaller sets is an effective way of ensuring that you do not hit API limits even with large data sets. You may also want to look at this white paper which describes how you should configure your environment to handle large data sets.

  3. Use a ETL tool such as Informatica, Dell Boomi or a similar tool. This blog post offers some good insights into using this approach coupled with Bulk API which works well for large data sets and complex topologies such as the one shown here.

enter image description here

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.