5

​Can somebody please help me to assess the feasibility of my solution with the new DataWeave feature?  

For many years I maintain an Open Source solution to receive and monitor App Error emails from Managed Packages called ISV Cockpit. As I learned that App Analytics Logs are the future also of Error reporting I wanted to enhance my solution also by pulling those logs. As App Analytics CSV logs also contain a myriad of other information those files are HUGE (depending on the time frame Gigabytes) I cannot download, parse and filter in Apex due to Governor Limits.

And here is where I thought about DataWeave as demonstrated on Dreamforce '22. A small function that outside of Apex limits, downloads parses, and filters the CSV down to a sizable chunk of error logs. Which then can be inserted as records in the PBO org.  

Feasible? Sensible? Doable?

  1. Are Functions usable in Scratch orgs already? How?
  2. Can I package such Scripts in an Open Source Unlocked Package or Managed Package?
  3. Will DataWeave help me overcome the Memory and CPU limits of Apex?
  4. Did you stumble across existing code that does something similar?

Something like:

%dw 2.0
input downloadUrl
var logs = readUrl("downloadUrl", "application/csv")
output application/apex
---
logs map(log) -> {
    dtm_RecordedDate__c : log.timestamp_derived,
    txt_UserId__c : log.user_id_token,
    txt_OrgId__c : log.organization_id,
    txt_Namespace__c : log.managed_package_namespace,
    txt_VisualforcePage__c : log.url,
    lkp_Package__c : log.package_id,
    lkp_PackageVersion__c : log.package_version_id,
    txt_ExceptionType__c : log.exceptionType,
    txt_ApexClass__c : log.class_name,
    txt_Method__c : log.method_name,
} as Object {class: "AppError__c"}
2
  • I've only come across and used App Analytics for package usage and engagement via CRUD operation counts for custom objects and anonymised component usages in the ISV package. I'm not aware of any coverage of errors. Can you point to where in the docs you read about error reporting?
    – Phil W
    Commented Sep 30, 2022 at 12:30
  • 1
    Search for stack_trace and ApexUnexpectedException in developer.salesforce.com/docs/atlas.en-us.packagingGuide.meta/… Commented Sep 30, 2022 at 13:10

2 Answers 2

6

DataWeave in Apex is currently at the Dev Preview stage in Winter '23. As part of being Dev Preview it is currently restricted to Scratch Orgs that explicitly have the feature enabled.

A small function that outside of Apex limits, downloads parses, and filters the CSV down to a sizable chunk of error logs.

This won't be possible. DataWeave in Apex operates within the Apex sandbox. The means that we:

  1. enforce the transaction CPU limit
  2. enforce the transaction Heap limit
  3. isolate the DW script from environment interaction
    1. I.e. you can't use readUrl()

It won't let you process files that would have exceeded the Apex heap limit, so CSV files in the gigabytes won't be possible.

Can I package such Scripts in an Open Source Unlocked Package or Managed Package?

Not as currently part of the developer preview, but as part of bringing this feature to GA packaging will be supported.

Will DataWeave help me overcome the Memory and CPU limits of Apex?

DataWeave in Apex doesn't remove the Heap or CPU limits. Rather, it allows you to make better use of them with a tool that is specifically designed for data transformation. So, with an Async operation you could process much larger files within the 12MB heap limit. It won't however scale to allow you to read files larger than the total heap. So gigabyte files are out of scope.

5

At Dreamforce '22 I was told that DataWeave runs in the same transaction as Apex and is shares the CPU time limit with the Apex transaction (similar to Flow). I don't know if this is true for Heap space as well.

I was also told that DataWeave had it's ability to talk to the outside world limited, Apex has to deliver all of the inputs at this time. Some example DW scripts, none of which communicate with the outside world. One of them shows exporting as XLSX but notes that DW in Apex doesn't support that behavior at this time. https://github.com/developerforce/DataWeaveInApex/tree/main/force-app/main/default/dw

I don't think there is any way as of now (the product is still not GA and can't be used in production at this time) to process files larger than what the heap space can hold in Apex because it has to hit Apex before it is sent to DataWeave.

I don't have any citations for any of this though, couldn't find any of this in the available documentation; https://developer.salesforce.com/docs/atlas.en-us.apexref.meta/apexref/apex_namespace_dataweave.htm

2
  • The heap usage is also enforced over the DataWeave Script. It can be more efficient in heap usage for large transformations than Apex, but it still has an upper limit. Correct that the DW scripts are prevented from interacting with the environment outside of Apex. I.e. no readURL. Commented Sep 30, 2022 at 19:25
  • 1
    While your answer is correct, I've posted a separate answer in my role as Apex Product Manager. Commented Sep 30, 2022 at 19:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .