1

So I have scenario where I need to process a bunch of blobs. I've tried to optimize my code and use Limits.getLimitHeapSize() to detect prevent limit exceptions, by first breaking the attachments into groups so the SOQL query doesn't exceed the heap. Below is a simplified example of what that looks like.

However, I am finding Limits.getLimitHeapSize() to be wildly inaccurate and thus not predicable.

Set<Id> batch = new Set<Id>();
Integer batchContentSize = 0;
for(Attachment attch : [SELECT BodyLength FROM Attachment WHERE ...]){

    System.debug('Heap @ Start: ' + (Limits.getLimitHeapSize()-Limits.getHeapSize()));

    //If we added the next image, we'd be at risk of hitting heap limit!  Sync and reset
    if(Limits.getHeapSize() + batchContentSize + attch.BodyLength*2 > maxHeapSize){

        List<Attachment> attachments = [SELECT Body
                                        FROM Attachment
                                        WHERE Id IN :batchIds
                                        ORDER BY BodyLength DESC];
        System.debug('Heap Remaining After Query: ' + (Limits.getLimitHeapSize()-Limits.getHeapSize()));
        //loop in revese & remove items from list to try and free up heap space ASAP
        for(Integer i = attachments.size() - 1; i>=0; i--){
            Attachment attchBody = attachments.remove(i);
            String payload = EncodingUtil.base64Encode(attchBody.Body);
            //do stuff with payload
            foo(payload);
        }

        batchContentSize = 0;
        batchIds = new Set<Id>();
    }

    batchContentSize += attch.BodyLength;
    batch.add(attch.Id);

}

I have triple checked that I don't have any memory leaks due to bad scope (everything is localized to the smallest possible scope). However, either memory isn't being freed up or Limits.getLimitHeapSize() isn't calculating properly.

Example output of Debug: Heap @ Start: 5399262 Heap Remaining After Query: 461459 Heap @ Start: 461364 Heap Remaining After Query: -4476045 //somehow exception doesn't even get thrown here...

2
  • Should your inner SOQL query that gets the Attachment body be performed for all the batchIds? I'd have thought it would only load the body for the last attachment you checked the BodyLength on. Sep 4, 2017 at 1:18
  • @DanielBallinger that's an artifact from the way I tried to simplify it. The goal is to perform as few queries as possible without at the same time exceeding the heap size. So I'm breaking it into batches where each batch should remain under the limit. (There's definitely a bug in this code where the last batch will not be processed)
    – NSjonas
    Sep 4, 2017 at 1:57

3 Answers 3

2
Heap Remaining After Query: -4476045 //somehow exception doesn't even get thrown here...

That's expected, actually. The heap governor limit is only checked every few milliseconds. I've managed to use nearly 60 MB of heap in a synchronous method without causing a governor limit to happen. You can briefly exceed the limit, but you shouldn't rely on this behavior.

String payload = EncodingUtil.base64Encode(attchBody.Body);

When you encode a Blob as a base64-encoded string, you're using up the original size of the blob, plus the size again, plus an extra 33%. This means that if your original blob is 3,000,000 bytes, at the moment where you've base64-encoded the blob you'd be at 7,000,000 bytes of used heap. In other words, use the Blob directly whenever possible.

As long as you check shortly after allocating memory, and free your memory as soon as possible, you can usually avoid the heap governor limit.

1
  • ^ya, I figured that out pretty quickly. Fixed my issue by passing the blob around until the very end, sending the request and then nulling the blob & req out immediately. Also, I don't think heap is check strictly on time, but also has something to do with allocation. I did some testing and it seems like I could exceed the heap by 4-6x and run a "wait" loop with no issues: for(Integer i = 0; i < 1000000; i++){} However, if a added a String concat on each iteration, it would fail once the string hit about 10000 characters.
    – NSjonas
    Sep 5, 2017 at 5:49
2

I suspect it is your inner SOQL query that gets the Attachment body that is causing the problem by retrieving the Body for all the batchIds.

This one:

List<Attachment> attachments = [SELECT Body
                                    FROM Attachment
                                    WHERE Id IN :batchIds
                                    ORDER BY BodyLength DESC];

I'd have thought it would only load the Body for the last attachment you checked the BodyLength on.

If you really did want to load all the Attachments for batchId's then you would need to calculate the total BodyLength of all those attachments to check against the current heap limit.

1
  • I didn't add code for it, but the first step was just to retrieve attachments and their BodyLength. I then use this info to try to break up the actually querying of the Body to avoid exceeding heap size. Sorry, I should have posted that up
    – NSjonas
    Sep 4, 2017 at 2:00
1

Looks like the funky-ness was being cause by the encoding of the blob
EncodingUtil.base64Encode(attchBody.Body); After coming across Passing Strings by value causes memory problem, I realized the way I was encoding the string and passing it to other functions was BAD! It still doesn't explain the wonky Limits.getLimitHeapSize() results, but now that I've refactor to pass the blob, it seems to be returning as expected.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.