7

We have a custom apex webservice in which we do certain operations and later call a managed package apex method that returns a JSON string which we deserialize and send back to the client. For example purposes let's assume the following JSON structure is returned from the managed package method:

{
    "x": "xxx",
    "y": "yyy",
    "z": "zzz",
    "lines": [{
        "a": "aaa",
        "b": "bbb",
        "c": "ccc",
        "record": {
            "attributes": {
                "type": "Account",
                "url": "/services/data/v44.0/sobjects/Account/001000000000000000"
            },
            "Id": "001000000000000000",
            "AccountNumber": "12345",
            "Contacts": {
                "totalSize": 1,
                "done": true,
                "records": [{
                    "attributes": {
                        "type": "Contact",
                        "url": "/services/data/v44.0/sobjects/Contact/003S00000000000000"
                    },
                    "Id": "003S00000000000000",
                    "Phone": "22323423423",
                    "MobilePhone": "345345345345"
                }]
            }
        }
    }]
}

Taking the above example we could assume that we could deserialize the above JSON string into the following classes:

public class MainModel {

    String x;
    String y;
    String z;
    LinesModel[] lines;
}


public class LineModel {

    String a;
    String b;
    String c;
    Account record;
}

As we can see the LineModel states that the record attribute has an Account sobject type, therefore it can span children relationships of Contacts, Opportunities and many other children relationships.

Therefore we could deserialize the JSON the following way:

MainModel model = (MainModel)JSON.deserialize(jsonString, MainModel.class);

Before returning the model to our webservice caller we want to get rid of some information that the manage package includes but we don't want, for example the Contacts children tree for each Account.

In order to remove it, we are doing the following:

Map<String, Object> genericModel = (Map<String, Object>)JSON.deserializeUntyped(jsonString);
Object[] lines = (Object[])genericModel.get('lines');

Map<String, Object> itemObj;
Map<String, Object> record;

for (Object item : lines) {

    itemObj = (Map<String, Object>)item;
    record = (Map<String, Object>)itemObj.get('record');

    if (record.containsKey('Contacts')) {
        record.remove('Contacts')
    }
}

MainModel model = (MainModel)JSON.deserialize(JSON.serialize(genericModel), MainModel.class);

The issue we're facing is that when the response from the manage package has many nested records the desarialization process (the deserializeUntyped call) fails with an Apex heap memory limit. When I say nested records, in the context of the above example there could maybe be only 20 Accounts but each account maybe having 60 contacts.

Since we don't have control on what the manage package returns in the JSON we cannot avoid having those unnecessary children.

We've tried removing the children using regex directly on the JSON string the following way to avoid unnecessary deserialization for cleaning:

String textPattern = '\\"Contacts\\":.*records.*\\[.*]},';

String resultText = jsonString.replaceAll(textPattern, '');

MainModel model = (MainModel)JSON.deserialize(JSON.serialize(resultText), MainModel.class);

But this results on a System.LimitException: Regex too complicated error because the JSON string is too large.

Any ideas on how we can clean the unwanted JSON data before returning it to the caller that could be viable?

Please take into account that the above is an example very simplified for the sake of understanding the real issue, the real JSON has many other relationships and objects that need cleaning and is more complex but the scenario is pretty much the same.

  • 6
    This doesn't sound like a memory leak to me, just a very large json string. The synchronous limit on heap size is ~6MB, and the async limit is ~12MB. If you're trying to manipulate that JSON, I'd expect that you'd only be able to handle a JSON string less than half that size (the string remains in memory, and you need at least as much memory to deserialize). Do you happen to have any idea of the size of your JSON string(s)? – Derek F Jan 9 '19 at 17:19
  • 1
    I've edited the question title to reflect Derek's point that this isn't a leak but rather hitting the limit. – Keith C Jan 9 '19 at 20:08
  • Great point @DerekF, indeed we'll need to take that into further consideration to avoid hitting the limits even if the deserialization doesn't include the non desired info – jonathanwiesel Jan 10 '19 at 9:14
  • Regex too complicated can usually be solved by breaking up a large string with substringBefore/substringAfter in a while loop. Collect the parts in a list and run the regex individually on them. – Alexander Johannes Jan 10 '19 at 10:54
10

In your wrapper class LineModel, don't use Standard Account. If you do that you will also deserialize children. Instead, create Apex Wrapper for Account with fields you need. This will be more memory efficient.

public class LineModel {

        String a;
        String b;
        String c;
        MyAccount record;
    }

    public Class MyAccount{
        String Id;
        String AccountNumber;
    }

    public class MainModel {

        String x;
        String y;
        String z;
        List<LineModel> lines;
    }
| improve this answer | |
  • 1
    +1. This is indeed a very neat approach to deal with any such limitations. – Jayant Das Jan 9 '19 at 20:57
  • Somehow, I didn't expect this to work (given that serialized SObjects contain extra data, i.e. the attributes attribute). However, copy/pasting the JSON from the question, and your deserialization classes, it all worked without a hitch. After mulling it over a bit, I suppose I shouldn't have been surprised. The only gotcha here is that this won't work with JSON.deserializeStrict(). +1 – Derek F Jan 9 '19 at 21:33
  • This seems like a great solution, in fact I marked as the correct answer until I figured out that if custom fields are included in the JSON they cannot be added to the serialization class because an attribute name with double underscores will cause an Invalid character in identifier: customfield__c error. – jonathanwiesel Jan 10 '19 at 9:27
  • I'm aware that using different names in the class (jsonString.replaceAll('customfield__c', 'customfield_X)) could work but could that could hit again the Regex too complicated error? Maybe I'm wrong since it's not a regex but a plain string, yet I'm concerned on the CPU limit while replacing 50 custom fields in a 2MB string – jonathanwiesel Jan 10 '19 at 9:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.