We have external system which retrieves the data from Salesforce by using Rest Api. It was working fine till the return resultSet is limited. However, when we are working on large data set it works as follows :
- Perform rest callout for retrieving the data from Salesforce.
- Suppose the result set is 2,00,000 then we realized that the requestForQuery REST operation only returned the first 2000 records and by using the nextRequestsUrl field returned in the REST call response with SFRestRequest::requestWithMethod function will retrieve the next 2000 records.
- This process takes good amount of time which hampers our business flow.
To overcome this we thought of using BULK API reader with PK chunking enabled which creates batch jobs in salesforce by dividing the resultset as per the chunk size we specified. However, we observed that the IDs of soft-deleted records are counted when the query is split into chunks, but the records are omitted from the results.. This might hit the daily batch limit in future which is 10,000 per day.
So guys, is there any other recommended solution approach for handling or retrieving such large data sets from salesforce or is there any way we can ignore the creation of unwanted batch job by ignoring soft deleted records.