3

We have a requirement to export around 15 millions records.

Which option will be better Salesforce DataLoader or Jitterbit Cloud DataLoader or any other better option available.

Can we export around 15 millions or more records in one go ! is it possible ? will CSV file able to handle that many record if exported in one go.

Also can we do this using Report API wherein we can create a report and fetch reports data using API, is it advisable considering volume of data?

Please advise the best option to go with, for exporting large volume of records from Salesforce.

  • Is it 15 million records in a single object, or across multiple objects? – xn. Jun 13 '19 at 15:17
3

Any tool that supports the Bulk API, such as Data Loader, should work fine. If your exporting data from an object or objects that support PK Chunking, you will probably want to use it.

To provide one data point, testing an export of about 15 million Tasks using queryAll (to included deleted/archived records) and a chunk size of 250k, writing to a zipped CSV file took about 17 minutes:

$ ro -i -c 250000 Task
INFO[0001] Logged in as 'cwarden@example.com' (API 45.0)
INFO[0002] Starting Backup                               organization="Example Org" username=cwarden@example.com
INFO[0002] Backing up object                             label=Tasks object=Task
INFO[0004] Waiting for pk chunking to complete           object=Task
...
INFO[0016] Waiting for bulk export to complete           object=Task
...
INFO[1046] Added Task.csv to 2019-06-13-084702.zip       object=Task
INFO[1046] Backup complete.
1

When you need to extract 10s or 100s of millions of records from Salesforce, you’ll get better performance and reliability if you split the job into a number of separate queries that each retrieve a smaller portion of the data. When the number of records in a single query is lower than the selectivity threshold of the Salesforce Query Optimizer, the platform can process the queries more efficiently. The new PK Chunking feature of the Bulk API automates this process by using the Primary Key (Record ID) of an object to break up the data into manageable chunks and query them separately. This feature is supported for all custom objects, many standard objects, and their sharing tables.

https://developer.salesforce.com/blogs/engineering/2015/03/use-pk-chunking-extract-large-data-sets-salesforce.html

0

The Reports wont work, API wont give more than 2.000 values(https://help.salesforce.com/articleView?id=rd_reports_limits.htm&type=5). Data loader has a limit of 5 million (http://resources.docs.salesforce.com/202/20/en-us/sfdc/pdf/salesforce_data_loader.pdf). Gernerally there is no limitation on csv-file-size, so i would recommend jitterbit.

  • I don't believe there is a limit on Bulk Extracts in Data Loader, other than what is imposed by the Bulk API itself. I might be wrong, but the linked document is quite old and I think it's referring to loads. – David Reed Jun 13 '19 at 13:23
  • The REST and SOAP APIs will give you up to 2,000 records at a time for a query, but the response will include the url to fetch the next batch of records. So it could be used to fetch lots of records; it just might take a while. If you are querying many fields or large fields, you may get fewer than 2,000 records in each result set. – xn. Jun 13 '19 at 17:16
0

If the frequency is less like Weekly, Monthly or One Time,

You can use Standard SF Weekly Data Backup export, they provide Zipped CSV for you to download easily.

You can use bulk API export, if you worry about API limits you can ask salesforce for one time increase in limit for free.

-2

My low/no code solution would be to use the free Jitterbit Data Loader and their SOQL query to loop through 1 million at a time (maybe 500k... depends on the number of columns).

I prefer Jitterbit's dataloader over Salesforce's because JB lets you can recycle the entire process. Build it once, and modify after each pass instead of rebuilding each time.

A code solution (that I don't fully understand) is mentioned here, using powershell. https://stackoverflow.com/questions/39207752/how-to-export-csv-file-with-more-than-1-048-576-rows-of-data-via-powershell

Not the answer you're looking for? Browse other questions tagged or ask your own question.