4

I have never worked on handling large datasets and reading through the documentation raised the below questions in my mind:

1) If I need to load 10,000,000 (10 million) records into salesforce, Can I use Dataloader and select the 'Use Bulk API' option on settings screen?

2) What exactly is the use of 'Enable serial mode for Bulk API' on the settings screen? Documentation is not really helpful on this..

3) If I have 2 million records to load and using dataloader, do I need to enable the Bulk API option? If I enable what is the advantage?

  • 1
    Hey, those are basically 3 separate questions :-). This question may be easier to answer if you focus on a single aspect. 1= yes. 3=> async & monitoring. – Samuel De Rycke Sep 6 '15 at 20:52
  • @SamuelDeRycke For the first question, If i dont select Bulk API option, I can't load more than 5 milliom records right? Just to confirm. – javanoob Sep 6 '15 at 20:54
9

1) If I need to load 10,000,000 (10 million) records into salesforce, Can I use Dataloader and select the 'Use Bulk API' option on settings screen?

The data loader supports any size file that your system can handle and Java can read in memory. In theory, it could support hundreds of millions of records. The bulk API has smaller limits, but the data loader handles this for you transparently.

2) What exactly is the use of 'Enable serial mode for Bulk API' on the settings screen? Documentation is not really helpful on this..

In the default mode, the system splits the file load into pieces and runs them concurrently. This is faster in wall clock time (the physical amount of time needed to complete processing), but can cause row locking errors. Changing to serial mode will greatly increase the physical time needed to process the file, because all records in the file will be processed sequentially, but reduces the odds of row locking errors. Use this mode only if you get many locking errors because of the default parallel processing mode.

3) If I have 2 million records to load and using Data Loader, do I need to enable the Bulk API option? If I enable what is the advantage?

In parallel mode, the processing time can be significantly reduced, because the program is not idle waiting on single batches, and doesn't need to be multi-threaded. In serial mode, the bandwidth for the files can be used up front, and CSV uses less bandwidth than SOAP, so there's some benefit in loading speed, and a network interruption after the initial upload won't affect the processing (compared to SOAP mode, where the upload is spread over time).

  • But the documention says that use dataloader only up to 5 million records and anything above that use product from appexchange..Why salesforce doesn't recommend using dataloader for more than 5 million records? – javanoob Sep 7 '15 at 2:41
  • 3
    @javanoob Probably because that's a "safe" limit for the data loader. Actual mileage may vary depending on system memory and Java configuration. I've used it to modify well over ten million records with just a few fields, but if you're using hundreds of fields, you'll find you may run out of memory or experience freezing, etc. There's no actual limit in the code that cuts you off at exactly 5 million rows (e.g. you won't see an error "Too many rows: 5000001" or some such). – sfdcfox Sep 7 '15 at 5:19
1
  1. Dataloader is a good to insert upto 5 million records. You can give a try by doing it into 2 csv files each of 5 million.

  2. Bulk API allows you to run the inserts on dataloader asynchronously. Using the serial mode a batch of 200 records can be processed at a time. The serial mode is better than the parallel to the database contention.

  3. Bulk API is very fast when you are working on large sets of data like 2 million. The SOAP API the default of dataloader will be comparatively slow tot he Bulk API.

  • @sfdcfox: 1 small question in this context. I have a requirement to send 10 million records to Salesforce from a Mediation System. I can write the records in CSV in the system. Should I upload these files using DataLoader with Bulk API on , or should I make call from Mediation System itself to Call Bulk API Request? – Deep Singhal Sep 19 '17 at 5:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.