Setting the batch size via Bulk API

Question

I'm using the Bulk API to make updates to ~50k records. I'm splitting those up into jobs of about 5k apiece, but I was hoping there'd be a way to control the batch size similar to that of using Dataloader (ie: you can tell it to process in chunks of 200 records at a time, or 1 record at a time).

Is this configurable? Or if not is there a different way to achieve this? The main issue is that some of the jobs fail due to a record triggering too many workflows that bog down the execution time, so I'd rather try and isolate those and use smaller batch sizes.

Answer 1

The Bulk API is intended to load large numbers of records in parallel asynchronously. It also has really small daily limits. You could technically set up 10,000 batches of 1 record each, but then you'd hit your daily limit. Use the normal synchronous API instead. If you need batches of smaller than about 1,000 records, the Bulk API is not for you. Please note that, unlike the synchronous API, each batch has a limit of 10 minutes (with retries for additional chunks), and each chunk of 200 records has 5 minutes, instead of the usual smaller time limits associated with synchronous updates.

With the synchronous API, the batch size is how many records are submitted at once per API call. In the Bulk API, the batch size is how many records are submitted in a file, which is termed a batch. The batch is then broken down in to chunks of 100/200 records each (depending on API version). As long as each chunk runs in less than 5 minutes, you should be okay. If you're not able to get decent performance with values smaller than 1,000 or so, it's simply going to be too "expensive" in terms of daily limits to use this API.