Auto Chunk Batch with BulkAPI

Question

I'm trying to use Bulk API v1 to upsert massive Data using JSON.

When I try to run the job via the standard bulk API, the next error shown up :

InvalidBatch : Exceeded number of records: 10002. Number of records should be less than or equal to 10001

I just want to know, Salesforce does not chunk my records into separate batches automatically?

Do I need to ask my clients to chunk the batch themself, and call the bulk API few times?

What I'm doing is :

URL : /services/async/50.0/job ---> /services/async/50.0/job/jobId/batch

Headers :

Content-Type: application/json; charset=UTF-8

Accept: application/json X-SFDC-Session:00D1l0000008dIX!AQQ...

I also tried to add this line to the header:

Sforce-Enable-PKChunking: chunkSize=9000; //tried this, tried "true" - same result

Many Thanks,

Answer 1

Bulk API 2.0 does chunk your data for you:

Salesforce creates a separate batch for every 10,000 records in your job data, up to a daily maximum of 150,000,000 records. If the limit is exceeded while processing your job data, the remaining data isn’t processed. The ingest job is marked as having failed.

Bulk API 1.0 does not, you manually chunk your data as you upload batches to a job:

Maximum number of records in a batch 10,000

(The walkthroughs of bulk upload processes in the linked Developer Guide go through the details of the process, e.g. Add a Batch to a Job).

PK Chunking is a tool you use with Bulk Query; it doesn't affect load operations.

If you need this behavior, switch to Bulk API 2.0. Note that the API is quite different so you'll probably need to rebuild portions of your code.