5

When I'm trying to insertBulk a lot of records using jsforce, only 10k records get inserted (single bulk job is being registered). The same happens with bulkDelete, when piping query result to destroyBulk job.

As a workaround, I'm using the following code to chain bulk calls, but to be honest, that doesn't seem like the correct way to do that. Is there any better way?

function bulkJob(sfdcRecords, resolve, reject, begin, end) {
  begin = begin || 0;
  end = end || 10000;
  let records = sfdcRecords.slice(begin, end);
  conn.sobject('MyObject__c').insertBulk(records, (err, rets) => {
    if (err) {
      reject(err);
    } else {
      if (end > sfdcRecords.length) {
        resolve();
      }
      bulkJob(sfdcRecords, resolve, reject, end, end + 10000);
    }
  });
}
Improve this question

1 Answer 1

Reset to default
6

Looks like that number is an intentional API Limit.

From the Bulk API Documentation:

Batch size

  • Batches for data loads can consist of a single CSV, XML, or JSON file that is no larger than 10 MB.
  • A batch can contain a maximum of 10,000 records.
  • A batch can contain a maximum of 10,000,000 characters for all the data in a batch.
  • A field can contain a maximum of 32,000 characters.
  • A record can contain a maximum of 5,000 fields.
  • A record can contain a maximum of 400,000 characters for all its fields.
  • A batch must contain some content or an error occurs.

I think the "correct" way to insert a larger number is to create all the batches at the same time, then submit them together- but I've never worked with the Bulk API, so Im a little fuzzy on the details.. Seems like using createBulk & splitting your data into a bunch of jobs would be a good solution, but thats very close to what you already have..

Improve this answer
4
  • 1
    That seems more like a batch chunk size limit. I really don't care about chunks size, I'd like this bulk method or API itself to get its job done.
    – Bart Juriewicz
    Commented Dec 20, 2017 at 14:50
  • "The chunk size depends on the API version. In API version 20.0 and earlier, the chunk size is 100 records. In API version 21.0 and later, the chunk size is 200 records."
    – battery.cord
    Commented Dec 20, 2017 at 14:53
  • To submit all the records in the same job might require a different api, but from what I can tell, creating a large number of separate batch jobs with sequenced data, and submitting them at the same time, and managing the exceptions asynchronously seems like a better solution than waiting for each job to be processed before inserting the next batch, but I don't know a lot about your use case.
    – battery.cord
    Commented Dec 20, 2017 at 14:55
  • 1
    When using e.g. dataloader, there's no need to do anything else than just select csv and use bulk DML. 10k seems to be the standard DML limit, so I can't see much benefit here to be honset. I still feel like there is (or should be...) a way to do is without creating chunks manually
    – Bart Juriewicz
    Commented Dec 21, 2017 at 9:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .