8

I'm currently using the bulk-api to return all records for a given sObject.

I can make it work via one very large batch, but I think it would be more efficient if I could split it into multiple batches and process them while more are being retrieved.

Is this possible? If so, how?

I know there would be various ways to pre-process the records or use something such as DateCreated fields to split on but that won't be a consistent enough solution to be worth implementing.

Thanks.

2
  • What's your reason for using Bulk API? What kind of volume of records are you dealing with? You could use the regular API and set the batch size up to 2000 (the maximum). Then you make multiple calls in with querymore and you can be processing the records as you retrieve more.
    – Doug B
    Commented Jul 11, 2013 at 7:28
  • 1
    "I'm currently using the bulk-api to return all records for a given sObject." I'm usually exporting 300k+ records, the standard API is far too slow for this. Commented Jul 11, 2013 at 8:37

3 Answers 3

1

Hi Tom for your question we need to divide this requirement into two parts:-

1) Retrieving the data from BulkApi queries, I hope no problem for you in that.

The data will get recieved in the form of files with the max. size per file is 1GB.

So as soon as u will get a CSV file out there the second task is to parse this first CSV file(considerring 2-3 more files are in quue) into the salesforce in batches.

So your second requirement will get fulfilled with the help of the function available here:- http://boards.developerforce.com/t5/Apex-Code-Development/Batch-Insert-of-Records-in-CSV-file/td-p/131413

This function will parallely start your procedure. and combination of two fulfill your task completly

0

I've implemented splitting the query into batches in the salesforce_bulk_query gem. It's done through CreatedDate or any other date field on the sObject. Check it out at https://github.com/cvengros/salesforce_bulk_query

0

We can use PK Chunking for splitting the huge sets of records into batches using Bulk Query

Refer this link

https://developer.salesforce.com/docs/atlas.en-us.api_asynch.meta/api_asynch/async_api_headers_enable_pk_chunking.htm

You will have to add the header such as if the requirement is to have 100,000 as the batch size then you could use

bulkConnection.addHeader("Sforce-Enable-PKChunking", 100000);

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .