9

I need to migrate 5GB of attachments to Salesforce.

I've created the mapping file and have been able to successfully upload some files using Data Loader (Lexi Loader).

I'm currently running into the following intermittent issues:

  • Data loader fails with the error "failed to create batch"
  • Data loader hangs at 999 entries
  • Data loader will only work if batch size is set to 1

I've looked at using DataLoader.io, but my some of my attachments are up to 100MB and DataLoader.io seems to have a 5MB limit.

It seems like I must be missing something obvious because I know people migrate large CRM datasets to Salesforce.com every day.

What's a reliable way to import thousands of attachments to Salesforce?

6

Salesforce has a limit to the size of the attachments you can upload into Salesforce (assuming they are 'Attachments' and not Chatter Files). The limit for attachments was recently increased from 5MB to 25MB. For Salesforce Files, it is 2GB. See the below article.

https://help.salesforce.com/apex/HTViewHelpDoc?id=collab_files_size_limits.htm&language=en_US

When I am doing a large number of attachments I sort the CSV file to put the smallest files first, then split the file into groupings, that way I get through the majority of the files quickly.

I change the batch size to 10, load all the files that are 2 MB or less, then drop the batch size down to 5, then load the files that are 5 MB, and then finally change the batch size to 1, to load the last group.

The batch size limit for each upload is 50 MB. See the below article.

https://help.salesforce.com/apex/HTViewSolution?urlname=Error-Maximum-size-of-request-reached-Maximum-size-of-request-is-52428800-bytes-1327107576367&language=en_US

For files that are over 25 MB, you may need to speak to Salesforce to see if they can increase the limit.

  • 1
    There's apparently a limit on SOAP maximum payload size, which effectively limits the maximum size of file you could possibly upload with the Data Loader. The Chatter API could be used to upload files up to 2GB, but this has other restrictions. – sfdcfox Oct 21 '14 at 6:46
  • Thanks for the information. I was eventually able to upload all the attachments by splitting them into very small batches and using DataLoader. I was hoping DataLoader would intelligently batch uploads. – Andrew Hopper Oct 21 '14 at 13:51
2

I never was able to get DataLoader to work reliably. I ended up using the excellent restforce (https://github.com/ejholmes/restforce) Ruby gem to craft a small script to upload attachments. This worked perfectly. I think my days with DataLoader are over.

require 'restforce'
require 'csv'

client = Restforce.new :username => '',
    :password       => '',
    :security_token => '',
    :client_id      => '',
    :client_secret  => ''

CSV.open("import_results.csv", "w") do |csv|

line = 0
CSV.foreach('attachment_mapping_file.csv') do |row|
  begin
    r = client.create 'Attachment', Body: Restforce::UploadIO.new(File.expand_path(row[6], __FILE__), row[5]),
      ParentId: row[1],
      OwnerId: row[2],
      Name: row[4]
    csv << [i,row[0],row[3],r].join(",")
  rescue
    csv << [i,row[0],row[3],"ERROR"].join(",")
  end
  line += 1
end

end
  • Hi Andrew. I'm trying to use this as well but having difficulties getting the file path right. Could you share an example line in the csv file that you used? – Guy Clairbois Dec 17 '14 at 21:21
  • Never mind, I figured it out... thanks for posting the answer! – Guy Clairbois Dec 17 '14 at 21:28
0

For the sake of completeness: Informatica Dataloader also has a (paid) FileIO plugin to support uploading/downloading of binary files. It might be of help to you.

BTW, I tried it but couldn't get it to work properly.

0

Maximum size of attachment that salesforce supports as of now is 25MB. So you cannot upload your 100MB files to salesforce as attachment. You can find step by step instructions regarding how to load attachment using data loader in below link,

http://www.shellblack.com/administration/import-attachments-using-the-data-loader/

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.