How to manage heap size in apex in sandbox

Question

In a sandbox environment, I am writing an inbound email handler. I process the email's attachments asynchronously. I am getting the following error:

System.LimitException: Apex heap size too large: 28966369

on the following line:

String encodedString = EncodingUtil.convertToHex(filedata);

Can someone explain to me why I'm getting this error given the following debug statements?

The code that produced these debugs:

System.debug(Limits.getHeapSize() + '/' + Limits.getLimitHeapSize());
String encodedString = EncodingUtil.convertToHex(filedata);
System.debug(Limits.getHeapSize() + '/' + Limits.getLimitHeapSize());

I seem to always surpass the heap size limit as my code is running and handling the large attachment files. Upon processing a 6MB+ attachment file, I receive the error.

1. How is the heap size increasing if my entire code is on a loop with variables that keep overwriting themselves? The debugs suggest that this is in fact happening, that the heap size is getting reset upon each iteration, but then...
2. how does converting to hex from 15m bytes result in a limit exception while the same conversion is okay in the previous iterations?
3. How can I check beforehand whether I can process (convert to hex) a byte collection if I can't know a reliable threshold value, given the debugs?
4. I read here that on production, I will receive errors upon surpassing 12MB in heap size for asynchronous transactions while the same limits apply but aren't enforced on sandbox. Is this true? In that case how should I proceed?

Answer 1

The governor limit on heap size is using Salesforce's "flexible" approach (has been for several releases). That means that:

• If your pod is healthy (not too busy serving other customers' requests)
• You haven't been going over the limit too often
• You aren't asking for too much additional leeway in the governor limit

Then Salesforce can choose to allow you to go over the documented (hard) limit.

As for why your heap is increasing on each iteration, my best guess would be that overwriting a variable (or having a variable go out of scope) doesn't immediately cause the garbage collector to run (Apex is compiled down to java bytecode, so Java's GC is in control, I'd imagine).

Unless you're able to get the size of the attachment from an external source, it'll be somewhere between difficult and impossible to determine if you can process a given attachment without running into a governor limit.

One thing you can do in your code is to track how much heap you have left (while we can go above the documented limit, that's not guaranteed to be allowed. The safe approach would be to use the documented limit as a cutoff).

If heap left - some safety threshold you define < the average heap used by a single attachment, then you can decide to stop processing further attachments (or figure out a way to chain another async call to pick up where you left off). Using the average gives you a general idea of if you can process another (and if it's a bit bigger, the flexible heap limit can possibly help you).

In the end though, Salesforce just isn't friendly to data that's larger than 3 MB (if you need to make a copy of the data, like you do when deserializing JSON, for example, then you're immediately at the synchronous transaction limit of 6 MB) and you risk running into issues.