How is heap size calculated?

Question

Scenario

I'm trying to send a lot of data from Apex to VisualForce Page via JS remoting because I want to create an Excel file , so just for testing purpose I uploaded a text file (4.87 MB, 5110400 Characters) as a static resource I got the file in Apex added it to a list of string.

Problem

I added the body of Static Resource to the list twice, below is my Apex and VisualForce code:

Apex Class:

public with sharing class HeapSize_Controller {

    @remoteAction
    public static List<String> generateRandomObject() {
        List<String> fileData = new List<String>();
        StaticResource sr = [
            SELECT
                Id,
                Body
            FROM
                StaticResource
            WHERE
                Name = 'RandomFile'
            LIMIT 1
        ];
        String body = sr.Body.toString();
        fileData.add(body);
        fileData.add(body);
        System.debug('========== Heap Size :: ' + Limits.getHeapSize());
        return fileData;
    }

}

VisualForce Page:

<apex:page showHeader="true" sidebar="true" controller="HeapSize_Controller">
    <apex:includeScript value="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.2.1/jquery.min.js"/>
    <script type="text/javascript">
        var j$ = jQuery.noConflict();
        j$(document).ready(function() {
            getData();
        });

        function getData() {
            Visualforce.remoting.Manager.invokeAction(
                '{!$RemoteAction.HeapSize_Controller.generateRandomObject}',
                    function(result, event){
                        if (event.status) {
                            console.log('======= List :: '+ result);
                        }
                        else {
                            console.log('======= Error Message :: '+ event.message);
                        }
                    },
                {escape: true,buffer: false}
            );
        }
  </script>
</apex:page>

Now the heap size in Apex is 10.22 MB (as per Debug logs)

But, when the page is loaded the remoting throw this error:

Why? Is it failing because Synchronous Limit for Heap Size is 6MB? Then why does the remoting say its exceeding 15 MB?

And when I do this in my Apex method,

//String body = sr.Body.toString();
fileData.add(sr.Body.toString());
fileData.add(sr.Body.toString());
System.debug('========== Heap Size :: ' + Limits.getHeapSize());

The heap size now is 15.33 MB, even though I only added it twice to the list.

I then went ahead and did this:

//String body = sr.Body.toString();
fileData.add(sr.Body.toString());
fileData.add(sr.Body.toString());
sr = new StaticResource();
System.debug('========== Heap Size :: ' + Limits.getHeapSize());

Which made sense when I saw the debug logs, the heap size now was 10.22 MB again. which leads me to believe that object sr holds about 5 MB?

Can someone please help me understand? Doesn't Remoting response handle upto 15 MB data? The Documentation says so.

Answer 1

Salesforce uses SI megabytes in all of their documentation, not SI mebibytes. This is confusing for Windows users and hardware programmers, who are used to seeing mebibytes called megabytes, and may not even know what a mebibyte is. The difference is that the SI system is a decimal-based system, while the other classic nomenclature is a binary-based system. This means that a string that is 5,110,400 is 5.11 MB, not 4.87 MB. You'll notice that 10.22 MB happens to be exactly double 5.11 MB, which is where you're getting that number from.

However, there's a twist: strings count all Unicode characters under 0x10000 as one "byte", despite them clearly being words (two bytes), and everything 0x10000 or higher as two bytes, despite taking up to 4 bytes of actual memory. So, depending on the contents of your file, your heap may show 10.22 MB while your actual string size may be as large as 20.44 MB, which would exceed the 15 MB (15,000,000 byte) limit.

Also, I believe the response is actually base64 encoded, which would increase the total response size by another 4/3, so 10.22 MB would be closer to 13.63 MB. Realistically, the heap size reported is usually a lot smaller than the actual amount of data that will be transferred over the wire.

Answer 2

You have more in the heap than just file data. Heap can be tricky to measure anyway because of garbage collection but you can see how much data you added by taking a diff:

Integer startingHeap = Limits.getHeapSize();
// add data to heap
Integer addedHeap = Limits.getHeapSize() - startingHeap;