0

Can anybody guide me through any framework which I can refer in order to achieve this use case?I am not able to get my head around it on how to do it efficiently.

I want to process records in near real time when the limit for no of dml rows have exceeded, now since the trigger is not a right place to do it, initiating async processing from a trigger could be highly problematic as rightly mentioned here link,

I am thinking to do it at on the application level may be using platform events instead of batch when the limit has exceeded, but again we'll have to consider following pointers:

1.With platform events we'll have to ensure that are not fired again and again with subsequent updates from workflow or process builder on the record. 2.Take care of limits for Platform events.

Any help is appreciated. Thanks in advance!

7
  • Do you need to reflect any of your changes back in the UI or over a REST API immediately, or can the changes be performed asynchronously in general terms?
    – Phil W
    Apr 14 at 7:24
  • @PhilW So, if the limit set by me or in general has been exceeded, the changes can be performed asynchronously.Does this answer your question?
    – Isha
    Apr 14 at 7:28
  • What I am getting at is, if this processing is initiated from a user (or API) action against one or more records, does this processing make changes that should be immediately visible within those same records? E.g. the user edits an Account to set some custom field value and the processing needs to update the Contacts related to the account in some way that then causes some other custom field to be updated on the Account itself, does this latter change have to be immediately reflected back to the user in the UI (and therefore need to happen synchronously)? Your answer suggests not.
    – Phil W
    Apr 14 at 7:38
  • In which case, is there a real requirement to ever perform updates synchronously "if possible"? Recall that having this sync vs async processing makes your code more complex and therefore a) more error prone and b) more costly to maintain.
    – Phil W
    Apr 14 at 7:39
  • @PhilW Got it,Let's take an example of what I am trying to achieve so let's say a user uploads thousand of records of a child object, so now I am performing an operation based on some criteria on those child records and updating it back on parent object, now there are chances that a user might upload more records than my class can handle let's say he uploads more than 10000 child records corresponding to 10,000 parent as in each parent has one child record, my class won't be able to update more than 10,000 parent in that scenario and hence I want to do it in async manner in this scenario
    – Isha
    Apr 14 at 8:02
1

As per the discussion:

  1. KISS ("Keep It Simple, Stupid") and avoid complexity in your code; since async updates are acceptable when there's too much data, always do this processing asynchronously. This avoids later problems when the number of records for which DML is performed changes perhaps due to introduction of flows or other processes in later updates to the solution. Now, that said, the over-all solution isn't really that simple, but at least you know everthing is processed through the same code flow which simplifies testing and maintenance.
  2. Mark the records that need to be processed in some way in the database so you can find and process them without having to track specific IDs - doing this database level tracking is robust against issues with "lost platform events", for example. We do this with two Datetime fields and a single Checkbox formula field:
    1. A ProcessingLastRequired__c datetime field, set in the trigger to System.now() when the record needs to be processed.
    2. A ProcessingLastPerformed__c datetime field, set in the actual "processor" using a timestamp initialized to System.now() when the processor was initiated.
    3. A PendingProcessing__c checkbox, a formula like: NOT(ISBLANK(ProcessingLastRequired__c)) && (ISBLANK(ProcessingLastPerformed__c) || ProcessingLastPerformed__c < ProcessingLastRequired__c)
  3. Whenever setting the ProcessingLastRequired__c field in the child record, only do so when absolutely necessary. For example, in before insert, always set this value but in before update only set the value when the changes applied to the record warrant the record being re-processed.
  4. We use timestamps because this makes the solution robust against multiple updates in different (parallel or near parallel) sessions or across multiple DMLs in the same session.
  5. We have the formula to make it easy to later query records where the last required time is after the last performed time, something you cannot achieve directly in SOQL WHERE clauses.
  6. When, in the trigger, you find you actually set ProcessingLastRequired__c for at least one record, you should generate and publish a platform event, e.g. ProcessChildren__e (with publish behaviour "after commit"), if you haven't already generated one in the current session (you can track this with a simple static boolean flag).
  7. Implement a trigger based platform event subscriber for the ProcessChildren__e platform event. It doesn't matter how many events it receives, it should simply consume them all and then:
    1. Check to see if there's already a "processor" batch running or queued (check the AsyncApexJob table for entries for the "processor" batch class in "Holding", "Queued", "Preparing" or "Processing" Status). If not, create and execute the "processor" batch.
  8. In the "processor" batch, a Batchable implementation, in start, again check to see if there's a "processor" running or queued:
    1. If not, simply return a query locator that selects the child records and required fields WHERE PendingProcessing__c is TRUE.
    2. If there is, return a query locator like "SELECT Id FROM Child WHERE Id = NULL" to effectively "abort" this inappropriate execution. Why do this here as well as before starting the batch? It avoids race conditions. One of the really important points here is that the platform guarantees that at most one batch's start method is called at any one time.
  9. In the "processor" start, before returning the query locator, grab System.now() and store it as an attribute in the "processor" class. This makes sure that the processor always applies a fixed time during its over-all execution.
  10. In the "processor" execute, process the child records as you need to, updating the parent records, but also updating the processed child records, setting ProcessingLastPerformed__c using the timestamp captured in start.
  11. In the "processor" finish you can optionally perform a COUNT query to see if there are name child records left with PendingProcessing__c set TRUE, and if there are, you can have the batch re-execute itself (via Database.executeBatch) since you know there's stuff to be processed and you don't need to wait for further platform events to be processed.
3
  • Thank you so much for such a detailed answer.
    – Isha
    Apr 14 at 9:39
  • Though I didn't get the 9 pointer, how does storing system.now() help in making sure that processor takes fixed time overall?
    – Isha
    Apr 14 at 10:07
  • 1
    The value from 9 is used in 10. Basically we know that Salesforce evaluates the query locator at the time start is invoked and this is the time we want to track against.
    – Phil W
    Apr 14 at 10:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.