We have external system which retrieves the data from Salesforce by using Rest Api. It was working fine till the return resultSet is limited. However, when we are working on large data set it works as follows :

  1. Perform rest callout for retrieving the data from Salesforce.
  2. Suppose the result set is 2,00,000 then we realized that the requestForQuery REST operation only returned the first 2000 records and by using the nextRequestsUrl field returned in the REST call response with SFRestRequest::requestWithMethod function will retrieve the next 2000 records.
  3. This process takes good amount of time which hampers our business flow.

To overcome this we thought of using BULK API reader with PK chunking enabled which creates batch jobs in salesforce by dividing the resultset as per the chunk size we specified. However, we observed that the IDs of soft-deleted records are counted when the query is split into chunks, but the records are omitted from the results.. This might hit the daily batch limit in future which is 10,000 per day.

So guys, is there any other recommended solution approach for handling or retrieving such large data sets from salesforce or is there any way we can ignore the creation of unwanted batch job by ignoring soft deleted records.

  • 1
    Question: do you need to query all those records, every day? Why can't you use the synchronization API calls (getUpdated, getDeleted)? By synchronizing the data across to your other system, you only need to check for the latest updates, and you can use the retrieve API call to get 2,000 records per call using only indexes, so it's a lot faster.
    – sfdcfox
    Jun 27, 2018 at 16:56
  • Hi @sfdcfox, we have to retrieve all the newly created records timely. I am planning to retrieve those records after every 1 hour. As the volume of data (newly created record) is very high and it will take sufficient amount of time for processing. Jun 27, 2018 at 17:08
  • You can call getUpdated and getDeleted with a five minute window if you want to.
    – sfdcfox
    Jun 27, 2018 at 17:21
  • @sfdcfox : This solutions works for us but it will retrieve all records that have been updated (added or changed) within the given timespan for the specified object. Additionally, I need to retrieve filtered records only which won't be possible in this case. Jun 27, 2018 at 17:45
  • Well, you need to do something; you can't really make the API go any faster than it already does (e.g. larger records per batch, etc). If you need to download 2,000,000 records or whatever, it's going to take time. It's a limitation of computers in general, not Salesforce. You need faster bandwidth in order to get faster results, which means replicating the database using getUpdated/getDeleted and querying a local database.
    – sfdcfox
    Jun 27, 2018 at 17:48


You must log in to answer this question.

Browse other questions tagged .