Ability of Apex Rest services to accept 50MB file, how is the heap size calculated in this case?

Question

I have the below code where in we are accepting the blob data from the REST Apex class. We are inserting the document (blob) that we are receiving as a feed item. I was able to send file sizes upto 54 MB after which it started erroring out.

The governor limit states that the heap size for synchronous transactions is 6MB then how is it that this REST class is able to accept file sizes 9 times the heap size limit? and when we print the limits in the debug even that is not showing a high number at all. Where is this file stored when the class receives it ? How is the heap size getting calculated?

@RestResource(urlMapping='/uploadFile/*')
global class REST_StoreAttachment {

    @httpPost
    global static String postMethod(){

        Blob requestBody=RestContext.request.requestBody;

        system.debug('Heap size before'+ limits.getHeapSize() + ' ==== '+ Limits.getLimitHeapSize());

        ContentVersion content = new ContentVersion();
        content.Title = 'test';
        content.PathOnClient = 'fileName.txt';
        content.VersionData = requestBody;
        content.Origin = 'H';

        insert content;

        FeedItem post = new FeedItem();
        post.ParentId = '00000000000000';         //hard coded to a Id
        post.Body = 'New attachment added';
        post.RelatedRecordId = content.Id; 
        post.Type = 'ContentPost';
        insert post;

        system.debug('Heap size after'+ limits.getHeapSize() + ' ==== '+ Limits.getLimitHeapSize());
        return 'success';
    }  
}

System Debug

Heap size before2189 ==== 6000000

Heap size after2514 ==== 6000000

Answer 1

Interesting. I suspect that what might be going on is, certain objects retrieved from certain methods point directly to underlying Java representations without being processed as part of Apex, and remain so until manipulated. Such objects sometimes possess weird behaviours compared to objects created directly in Apex.

One example I've seen is after deserialization of JSON. If a JSON attribute maps to an Integer attribute in the Apex class I deserialize it to, but a number like 1.00 was in the JSON, Apex somehow jams a Decimal into an Integer container instead of converting to the integer 1. Only when you try and actually do Integer math on it does it complain that it's not an Integer.

Just a thought: suppose you redundantly convert the blob to Base64 and back, and assign that to VersionData. Does it start enforcing heap limits now?

Answer 2

There seem to be following limits on standard REST SObject Basic Information or SObject Rows APIs. If you are uploading multi part message you should be able to upload following sizes as per documentation:

https://developer.salesforce.com/docs/atlas.en-us.api_rest.meta/api_rest/dome_sobject_insert_update_blob.htm

Using the SObject Basic Information or SObject Rows APIs, the maximum file size for uploads is 2 GB for ContentVersion objects and 500 MB for all other eligible standard objects. Using the SObject Collections API, the maximum total size of all files in a single request is 500 MB.

For non-multipart messages which seems to be a case of the custom endpoint the limits are as below:

You can insert or update blob data using a non-multipart message, but you are limited to 50 MB of text data or 37.5 MB of base64–encoded data.

Which corresponds to the error limit you are getting now. Might be related to your issue potentially.

Answer 3

Heap Limits ----- The only context that can even store 20 MB of data in RAM is Email Services. That means you'd have to kick off an email to Salesforce just to get your code to have enough memory to load your file in memory, much less have a chance of doing anything. Batch Apex, future methods, and Queueable all only have a 12 MB limit.

You have more in the heap than just file data. Heap can be tricky to measure anyway because of garbage collection but you can see how much data you added by taking a diff: you can calculate the increase in heap by below

Integer startingHeap = Limits.getHeapSize(); // add data to heap Integer addedHeap = Limits.getHeapSize() - startingHeap;

Is this sync call or async call? for async call heap limit is high (10Mb) Below is the salesfoce limit for file. Heap limit is different than the file limit Content: maximum file size • 2 GB • 2 GB (including headers) when uploaded via Chatter REST API • 2 GB (including headers) when uploaded via REST API • 38 MB when uploaded via SOAP API • 10 MB when uploaded via Bulk API • 10 MB for Google Docs • 10 MB when uploaded via Visualforce