24

I need to download the binary body of all the attachments in my organization. I'd like to do so with the least amount of API calls. It is my understanding that the SOAP API returns a single attachment body (in base64) for each "page" of result (aka you need to use "queryMore" once for every attachment)

The problem with this approach is that it costs one API call for each attachment. For other objects where there is no base64 field, you can easily retrieve 1000 items within a single page.

The REST API is similar. You can get multiple attachments within a single response but the response does not contain the body itself. It contains an URL to get the body.. so you end up having to do one extra call to download the body from this URL.

What is the most "API call" efficient way to get all the attachment bodies?

EDIT: The solution needs to be automated. I suspect the bulk API might do what I need.

  • GuiSim - are you ok with me retagging this question to remove the 'rest' and 'soap' tags, leaving the 'rest-api' and 'soap-api' tags? I think standardizing on abc-api would be good - see meta.salesforce.stackexchange.com/questions/4/… – metadaddy Aug 1 '12 at 22:38
  • If you do figure it out using the Bulk API, you can answer your own question, then mark your answer as correct, helping future travelers. – metadaddy Aug 2 '12 at 1:23
  • @metadaddy sure, no problem – GuiSim Aug 6 '12 at 11:57
8

If this is a one time export or monthly export that needs to be done, you can use the Monthly Export Service built in to Salesforce. Go to Setup->Data management->Data Export and click "Export Now" or "Schedule Export" and make sure "Include images, documents, and attachments" is checked along with any other data you want to export. Salesforce will export the data, zip it up and e-mail you when the export is available for download.

  • Is this something that can be automated using a Salesforce API ? – GuiSim Aug 1 '12 at 12:06
  • You can schedule it to run every month automatically but I don't believe there's any way to download it or check that the backup is done via the API. – E.J. Wilburn Aug 1 '12 at 14:11
  • I've raised a separate question about how the Data Export download can be automated. Automate the downloading of Data Export zip files – Daniel Ballinger Aug 10 '12 at 10:30
3

If you query a base64 field, it will only return one row. I assume this is because that field can be pretty big. Not really sure if there is an easy way around it. I usually create a download queue on my client that queries the SOAP API one file at a time.

You might want to try creating a web service that will gather all the base64 fields on the server side and send them all to your client in one call.

  • 3
    Note that if you take the custom webservice approach you need to be mindful of the 6MB heap space limit, so this is really only viable with small attachments. – ca_peterson Jul 31 '12 at 20:50
  • Agree with Scott the one record batch size for downloading base64 fields such as attachment is a limit enforced by Salesforce. You can do a backup to get an initial set of the files and then do nightly downloads of those that remain. I wouldn't recommend attempting to download all attachments on a regular basis thought through the API. Storing the files locally would be more efficient – John De Santiago Jul 31 '12 at 20:52
3

You cannot query binary fields with the bulk api (only upload is permitted). The only way around the one result Soap limit is to use the Data Export feature (most efficient but limited to every 7 days) or aggregate the records via apex and then try to serialize them to JSON or XML and expose the serialized list via an apex webservice of your own, but it will be difficult to extract many more records than the one Soap limit, as the Apex stack space limits will bite as you query out multiple records. It will depend on how large the files (Body of the records) are.

I think the only way to automate the Data Export is to hack the HTML with Javascript in a browser, but you can only do an export every seven days, which you can schedule via the Administration UI, so automating it is really about pulling the download(s) once it is prepared rather than 'kicking it off'. The only way to know if the download is prepared is to visually inspect or trap and parse the email that is sent out from Salesforce. Also, you only have 48 hours to download the export(s) once it has been prepared, then the file(s) is deleted.

2

You can always do a full data export from under Setup > Data Management > Data Export. The attachments should come out in the export zip files. It's technically the most API-efficient option as it doesn't even use the API! However it can't be automated if that's what you're looking for.

2

Depending on the volume, a possible solution might be to start by exporting all the current attachments using the Weekly Export feature. A scheduled (non-apex) job that runs hourly or daily could then download any new attachments since the job was last run (querying by attachment.sysmodstamp). You'd have to figure out a way to handle deleted attachments in your local data store.

2

Create a query Bulk API job for the Attachment object using the XML content type.

Use the query SELECT Id, Body FROM Attachment.

The result will contain the Body element with the contents base64-encoded.

Unfortunately, PK chunking is not supported for the Attachment object, and with a lot of Attachment records the bulk job may fail with "Retried more than 15 times". Therefore, you may need to fall back to fetching /sobjects/Attachment/<id>/Body for each Attachment.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.