26

I need to download the binary body of all the attachments in my organization. I'd like to do so with the least amount of API calls. It is my understanding that the SOAP API returns a single attachment body (in base64) for each "page" of result (aka you need to use "queryMore" once for every attachment)

The problem with this approach is that it costs one API call for each attachment. For other objects where there is no base64 field, you can easily retrieve 1000 items within a single page.

The REST API is similar. You can get multiple attachments within a single response but the response does not contain the body itself. It contains an URL to get the body.. so you end up having to do one extra call to download the body from this URL.

What is the most "API call" efficient way to get all the attachment bodies?

EDIT: The solution needs to be automated. I suspect the bulk API might do what I need.

Improve this question
3
  • GuiSim - are you ok with me retagging this question to remove the 'rest' and 'soap' tags, leaving the 'rest-api' and 'soap-api' tags? I think standardizing on abc-api would be good - see meta.salesforce.stackexchange.com/questions/4/…
    – metadaddy
    Commented Aug 1, 2012 at 22:38
  • If you do figure it out using the Bulk API, you can answer your own question, then mark your answer as correct, helping future travelers.
    – metadaddy
    Commented Aug 2, 2012 at 1:23
  • @metadaddy sure, no problem
    – GuiSim
    Commented Aug 6, 2012 at 11:57

6 Answers 6

Reset to default
8

If this is a one time export or monthly export that needs to be done, you can use the Monthly Export Service built in to Salesforce. Go to Setup->Data management->Data Export and click "Export Now" or "Schedule Export" and make sure "Include images, documents, and attachments" is checked along with any other data you want to export. Salesforce will export the data, zip it up and e-mail you when the export is available for download.

Improve this answer
3
  • Is this something that can be automated using a Salesforce API ?
    – GuiSim
    Commented Aug 1, 2012 at 12:06
  • You can schedule it to run every month automatically but I don't believe there's any way to download it or check that the backup is done via the API.
    – E.J. Wilburn
    Commented Aug 1, 2012 at 14:11
  • I've raised a separate question about how the Data Export download can be automated. Automate the downloading of Data Export zip files
    – Daniel Ballinger
    Commented Aug 10, 2012 at 10:30
4

You cannot query binary fields with the bulk api (only upload is permitted). The only way around the one result Soap limit is to use the Data Export feature (most efficient but limited to every 7 days) or aggregate the records via apex and then try to serialize them to JSON or XML and expose the serialized list via an apex webservice of your own, but it will be difficult to extract many more records than the one Soap limit, as the Apex stack space limits will bite as you query out multiple records. It will depend on how large the files (Body of the records) are.

I think the only way to automate the Data Export is to hack the HTML with Javascript in a browser, but you can only do an export every seven days, which you can schedule via the Administration UI, so automating it is really about pulling the download(s) once it is prepared rather than 'kicking it off'. The only way to know if the download is prepared is to visually inspect or trap and parse the email that is sent out from Salesforce. Also, you only have 48 hours to download the export(s) once it has been prepared, then the file(s) is deleted.

Improve this answer
3

If you query a base64 field, it will only return one row. I assume this is because that field can be pretty big. Not really sure if there is an easy way around it. I usually create a download queue on my client that queries the SOAP API one file at a time.

You might want to try creating a web service that will gather all the base64 fields on the server side and send them all to your client in one call.

Improve this answer
2
  • 4
    Note that if you take the custom webservice approach you need to be mindful of the 6MB heap space limit, so this is really only viable with small attachments.
    – ca_peterson
    Commented Jul 31, 2012 at 20:50
  • 1
    Agree with Scott the one record batch size for downloading base64 fields such as attachment is a limit enforced by Salesforce. You can do a backup to get an initial set of the files and then do nightly downloads of those that remain. I wouldn't recommend attempting to download all attachments on a regular basis thought through the API. Storing the files locally would be more efficient
    – John De Santiago
    Commented Jul 31, 2012 at 20:52
3

Create a query Bulk API job for the Attachment object using the XML content type.

Use the query SELECT Id, Body FROM Attachment.

The result will contain the Body element with the contents base64-encoded.

Unfortunately, PK chunking is not supported for the Attachment object, and with a lot of Attachment records the bulk job may fail with "Retried more than 15 times". Therefore, you may need to fall back to fetching /sobjects/Attachment/<id>/Body for each Attachment.

Improve this answer
2

You can always do a full data export from under Setup > Data Management > Data Export. The attachments should come out in the export zip files. It's technically the most API-efficient option as it doesn't even use the API! However it can't be automated if that's what you're looking for.

Improve this answer
2

Depending on the volume, a possible solution might be to start by exporting all the current attachments using the Weekly Export feature. A scheduled (non-apex) job that runs hourly or daily could then download any new attachments since the job was last run (querying by attachment.sysmodstamp). You'd have to figure out a way to handle deleted attachments in your local data store.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .