1

I am writing a class that splits text file apart. Chunking them into 5000 line groupings so I can upload the files contents to another limited system.

The text file contains a Json object on each line. Separated by newline characters.

{"Name":"Dog"}
{"Name":"Dog"}
{"Name":"Dog"}
{"Name":"Dog"}

My first version of the code used Split to rip the files down. later using the output map to fire off callouts to another system in groupings of 5000.

//get the files contents for all Files
List<ContentDocument> cdList = [
  SELECT Id, LatestPublishedVersion.VersionData
  FROM ContentDocument
  WHERE Id IN :contentDocumentIds
];

System.debug('cdList:' + cdList);

//Rip files apart and store in a map
Map<Id, List<String>> cdLinesMap = new Map<Id, List<String>>();
for (contentDocument cd : cdList) {
  String doc = cd.LatestPublishedVersion.VersionData.toString();

  cdLinesMap.put(cd.Id, new List<String>(doc.split('\n')));
}

This code worked on small files - but at around 10000 lines it hit an Error 'Regex too complicated' issue in Apex. Specifically due to .split().

What is another method to break these files down without hitting heap,regex or CPU limits in Apex?

----------Update March 19th 2:22PM EST

After some iteration based on the comments - Heres my current version. I am unforuntaley hitting a heap limit running it. With the error cropping up at the two substring lines.

for (contentDocument cd : [
  SELECT Id, LatestPublishedVersion.VersionData
  FROM ContentDocument
  WHERE Id IN :contentDocumentIds
]) {

  String doc = cd.LatestPublishedVersion.VersionData.toString();
  cd.LatestPublishedVersion.VersionData = null; // save memory
  Integer indexOf = doc.indexOf('\n');
  String[] lines = new String[0];
  while (indexOf != -1) {
    lines.add(doc.substring(0, indexOf));
    doc = doc.substring(indexOf + 1);
    indexOf = doc.indexOf('\n');
  }

  lines.add(doc);

----------Update March 20th 12:17PM EST

unfortunately the 12mb heap size blocked me on all vanila apex attempts. I tried the dataweave library extended to apex. That got me further than apex could. But I got limited by the size file I can provide to the dataweave script via soql.

If it helps anyone for small files heres my dataweave script:

%dw 2.0
input inputData text/plain
output application/json

// Split the input data into individual JSON objects
var jsonObjects = inputData splitBy "\n" map ((item) -> 
read(item, "application/json"))

// Group JSON objects into chunks of 5000 lines each
var batchSize = 5000

fun batchSplit(arr, size) =
    if (sizeOf(arr) <= size)
        [arr]
    else
        [arr[0 to size - 1]] ++ batchSplit(arr[size to -1], size)

---
batchSplit(jsonObjects, batchSize)
4
  • 1
    Have you considered using DataWeave and calling it in Apex? developer.salesforce.com/docs/atlas.en-us.apexcode.meta/… Commented Mar 19 at 17:31
  • @JefersonChaves never heard of it - I appreciate the idea - ill go read up on it! Commented Mar 19 at 18:29
  • Salesforce Apex has heap size and callout size and CPU limits that are hard (impossible?) to work around. And no streaming APIs that could let you access part of a file. Do the large text files have to be stored in Salesforce? If they do then you may be looking at running something on e.g. Heroku that reads the file in chunks and breaks it up and writes to your other system. Days of work instead of hours of work.
    – Keith C
    Commented Mar 19 at 19:16
  • 1
    @JefersonChaves I gave the dataweave idea a shot! it did better than my most recent updated solution did. But I got blocked by the limitation of file size I can provide to the script via soql. Great suggestion tho. Commented Mar 20 at 16:16

1 Answer 1

2

You can use indexOf to find your strings:

String doc = cd.LatestPublishedVersion.VersionData.toString();
cd.LatestPublishedVersion.VersionData = null; // save memory
Integer indexOf = doc.indexOf('\n');
String[] output = new String[0];
while(indexOf != -1) {
  output.add(doc.substring(0, indexOf));
  doc = doc.substring(indexOf+1);
  indexOf = doc.indexOf('\n');
}
output.add(doc);
cdLinesMap.put(cd.Id, output);
12
  • 2
    Just loading the file data with Apex could cause a heap space issue, especially if both that file data and the split data are both held in memory concurrently...
    – Phil W
    Commented Mar 19 at 17:20
  • 2
    @PhilW I agree. A SOQL for-each loop would be the best idea. This algorithm is more meant to showcase how to get the data out. The algorithm "eats" the data as it moves, limiting heap to just about 100% of the original string size. That said, I added a null assignment to make this run 1:1 for heap size.
    – sfdcfox
    Commented Mar 19 at 17:28
  • 1
    Depending on the implementation, use of substring could increase heap usage (until garbage collection is performed). I've no idea how it behaves in Apex, it must be said.
    – Phil W
    Commented Mar 19 at 17:40
  • 1
    @PhilW the heap is weird in Apex. You can exceed the limit as much as you want (I once got past 138 MB of heap), but if you're over the limit at the instant it decides to see if you're over the limit, you get the LimitException. Not even calling Limit.getHeapSize() will necessarily throw the exception, it's just whenever the runtime decides to check. This algorithm should be safe enough, assuming the file itself doesn't exceed the limit.
    – sfdcfox
    Commented Mar 19 at 17:58
  • 1
    @RyanSherry Yeah, the indexOf should have been updated. As far as hitting heap, we're already as efficient as can be. You may need to go asynchronous (Queueable/Batchable) to double your heap limits and/or process files in smaller sizes.
    – sfdcfox
    Commented Mar 19 at 18:51

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .