Limits.getHeapSize() wildly inaccurate?

Question

So I have scenario where I need to process a bunch of blobs. I've tried to optimize my code and use Limits.getLimitHeapSize() to detect prevent limit exceptions, by first breaking the attachments into groups so the SOQL query doesn't exceed the heap. Below is a simplified example of what that looks like.

However, I am finding Limits.getLimitHeapSize() to be wildly inaccurate and thus not predicable.

Set<Id> batch = new Set<Id>();
Integer batchContentSize = 0;
for(Attachment attch : [SELECT BodyLength FROM Attachment WHERE ...]){

    System.debug('Heap @ Start: ' + (Limits.getLimitHeapSize()-Limits.getHeapSize()));

    //If we added the next image, we'd be at risk of hitting heap limit!  Sync and reset
    if(Limits.getHeapSize() + batchContentSize + attch.BodyLength*2 > maxHeapSize){

        List<Attachment> attachments = [SELECT Body
                                        FROM Attachment
                                        WHERE Id IN :batchIds
                                        ORDER BY BodyLength DESC];
        System.debug('Heap Remaining After Query: ' + (Limits.getLimitHeapSize()-Limits.getHeapSize()));
        //loop in revese & remove items from list to try and free up heap space ASAP
        for(Integer i = attachments.size() - 1; i>=0; i--){
            Attachment attchBody = attachments.remove(i);
            String payload = EncodingUtil.base64Encode(attchBody.Body);
            //do stuff with payload
            foo(payload);
        }

        batchContentSize = 0;
        batchIds = new Set<Id>();
    }

    batchContentSize += attch.BodyLength;
    batch.add(attch.Id);

}

I have triple checked that I don't have any memory leaks due to bad scope (everything is localized to the smallest possible scope). However, either memory isn't being freed up or Limits.getLimitHeapSize() isn't calculating properly.

Example output of Debug: Heap @ Start: 5399262 Heap Remaining After Query: 461459 Heap @ Start: 461364 Heap Remaining After Query: -4476045 //somehow exception doesn't even get thrown here...

Answer 1

Heap Remaining After Query: -4476045 //somehow exception doesn't even get thrown here...

That's expected, actually. The heap governor limit is only checked every few milliseconds. I've managed to use nearly 60 MB of heap in a synchronous method without causing a governor limit to happen. You can briefly exceed the limit, but you shouldn't rely on this behavior.

String payload = EncodingUtil.base64Encode(attchBody.Body);

When you encode a Blob as a base64-encoded string, you're using up the original size of the blob, plus the size again, plus an extra 33%. This means that if your original blob is 3,000,000 bytes, at the moment where you've base64-encoded the blob you'd be at 7,000,000 bytes of used heap. In other words, use the Blob directly whenever possible.

As long as you check shortly after allocating memory, and free your memory as soon as possible, you can usually avoid the heap governor limit.

Answer 2

I suspect it is your inner SOQL query that gets the Attachment body that is causing the problem by retrieving the Body for all the batchIds.

This one:

List<Attachment> attachments = [SELECT Body
                                    FROM Attachment
                                    WHERE Id IN :batchIds
                                    ORDER BY BodyLength DESC];

I'd have thought it would only load the Body for the last attachment you checked the BodyLength on.

If you really did want to load all the Attachments for batchId's then you would need to calculate the total BodyLength of all those attachments to check against the current heap limit.

Answer 3

Looks like the funky-ness was being cause by the encoding of the blob
EncodingUtil.base64Encode(attchBody.Body); After coming across Passing Strings by value causes memory problem, I realized the way I was encoding the string and passing it to other functions was BAD! It still doesn't explain the wonky Limits.getLimitHeapSize() results, but now that I've refactor to pass the blob, it seems to be returning as expected.