4

We have a requirement to export around 15 millions records.

Which option will be better Salesforce DataLoader or Jitterbit Cloud DataLoader or any other better option available.

Can we export around 15 millions or more records in one go ! is it possible ? will CSV file able to handle that many record if exported in one go.

Also can we do this using Report API wherein we can create a report and fetch reports data using API, is it advisable considering volume of data?

Please advise the best option to go with, for exporting large volume of records from Salesforce.

Improve this question
1
  • Is it 15 million records in a single object, or across multiple objects?
    – xn.
    Commented Jun 13, 2019 at 15:17

5 Answers 5

Reset to default
6

Any tool that supports the Bulk API, such as Data Loader, should work fine. If you're exporting data from an object or objects that support PK Chunking, you will probably want to use it.

To provide one data point, testing an export of about 15 million Tasks with ro using queryAll (to included deleted/archived records) and a chunk size of 250k, writing to a zipped CSV file took about 17 minutes:

$ ro -i -c 250000 Task
INFO[0001] Logged in as '[email protected]' (API 45.0)
INFO[0002] Starting Backup                               organization="Example Org" [email protected]
INFO[0002] Backing up object                             label=Tasks object=Task
INFO[0004] Waiting for pk chunking to complete           object=Task
...
INFO[0016] Waiting for bulk export to complete           object=Task
...
INFO[1046] Added Task.csv to 2019-06-13-084702.zip       object=Task
INFO[1046] Backup complete.
Improve this answer
4
  • Do you happen to have a gist or github repo that references the code?
    – techbusinessman
    Commented Jun 18, 2020 at 22:13
  • Here's the code where the force cli enables PK chunking: github.com/ForceCLI/force/blob/…
    – xn.
    Commented Jun 19, 2020 at 15:12
  • Thanks. I was more meaning for the app that you built to do the backup. I'm trying to do the same thing and was hoping to not have to build from scratch :)
    – techbusinessman
    Commented Jun 19, 2020 at 18:15
  • Ro isn't open source, but it does PK chunking similar to the code in force cli.
    – xn.
    Commented Jun 19, 2020 at 18:41
2

When you need to extract 10s or 100s of millions of records from Salesforce, you’ll get better performance and reliability if you split the job into a number of separate queries that each retrieve a smaller portion of the data. When the number of records in a single query is lower than the selectivity threshold of the Salesforce Query Optimizer, the platform can process the queries more efficiently. The new PK Chunking feature of the Bulk API automates this process by using the Primary Key (Record ID) of an object to break up the data into manageable chunks and query them separately. This feature is supported for all custom objects, many standard objects, and their sharing tables.

https://developer.salesforce.com/blogs/engineering/2015/03/use-pk-chunking-extract-large-data-sets-salesforce.html

Improve this answer
0

The Reports wont work, API wont give more than 2.000 values(https://help.salesforce.com/articleView?id=rd_reports_limits.htm&type=5). Data loader has a limit of 5 million (http://resources.docs.salesforce.com/202/20/en-us/sfdc/pdf/salesforce_data_loader.pdf). Gernerally there is no limitation on csv-file-size, so i would recommend jitterbit.

Improve this answer
2
  • I don't believe there is a limit on Bulk Extracts in Data Loader, other than what is imposed by the Bulk API itself. I might be wrong, but the linked document is quite old and I think it's referring to loads.
    – David Reed
    Commented Jun 13, 2019 at 13:23
  • The REST and SOAP APIs will give you up to 2,000 records at a time for a query, but the response will include the url to fetch the next batch of records. So it could be used to fetch lots of records; it just might take a while. If you are querying many fields or large fields, you may get fewer than 2,000 records in each result set.
    – xn.
    Commented Jun 13, 2019 at 17:16
0

If the frequency is less like Weekly, Monthly or One Time,

You can use Standard SF Weekly Data Backup export, they provide Zipped CSV for you to download easily.

You can use bulk API export, if you worry about API limits you can ask salesforce for one time increase in limit for free.

Improve this answer
-2

My low/no code solution would be to use the free Jitterbit Data Loader and their SOQL query to loop through 1 million at a time (maybe 500k... depends on the number of columns).

I prefer Jitterbit's dataloader over Salesforce's because JB lets you can recycle the entire process. Build it once, and modify after each pass instead of rebuilding each time.

A code solution (that I don't fully understand) is mentioned here, using powershell. https://stackoverflow.com/questions/39207752/how-to-export-csv-file-with-more-than-1-048-576-rows-of-data-via-powershell

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .