Platform Events a nightmare - How to make them work in real life

Question

I am trying to build a Platform Event-based application logger where arbitrary application code calls Tracker.log(...); and this is sent as an Event into an external database via an API. As it's important that

1. Events are sent even if the transaction fails
2. The tracking itself should merely never interfere with the app code
3. Tracking should not share Governor limits with triggering transaction

I moved the actual EventBus.publish(...) to a separate Queueable and configured the event to Publish immediately.

But when there is a DML or Callout after the Tracker is called like this:

Tracker.log(...);
...
app.makeCallout();

it fails with:

Failed calling callout:XYZ :You have uncommitted work pending. Please commit or rollback before calling out

Answer 1

On the assumption that it is not the end of the world if the odd logger message gets lost (remember that PEs do not have guaranteed delivery), the approach I would take is to separate the platform event from the delivery to an external system along the lines of what I did in this example github repo, but:

1. Without using an actual "Command" object in the database (you don't want to commit data into the originating transaction as this may fail and roll back).
2. Using a Publish Immediately event that contains the logger message detail.

The main difference is that the event must contain all relevant detail directly and each must be specifically processed. Throwing away events isn't an option here (unlike in my demo that only uses the event as a means to initiate processing).

You can use infrastructure features, such as event and replay IDs to ensure that you don't try to process too many events into callouts in one call to the platform event subscriber trigger - or ensure that the queueable can hold the total data to process and can self-chain to address the logger details that could not be sent immediately.

Answer 2

Queueing an Queueable is effectively DML. You're inserting a record to have the platform do something AFTER your transaction COMMITS.

The COMMIT part is important -- if your transaction fails (you hit a LimitException or SystemException), the entire transaction is rolled back, including the queued Queueable.

https://developer.salesforce.com/docs/atlas.en-us.apexcode.meta/apexcode/apex_queueing_jobs.htm

If an Apex transaction rolls back, any queueable jobs queued for execution by the transaction aren’t processed.

Instead, you definitely want to call EventBus.publish() in your outer transaction (and have your PE set to Publish Immediately) so that it doesn't get rolled back regardless of the outer transaction outcome.

Regarding your concern about affecting the runtime characteristics of the application code. You need to be cognizant of the PE daily limit within the org -- for this reason, most log frameworks have configuration to flick the logging levels up or down to tune it to the specific use case (i.e. continuous logging of coarse application usage vs. trace level logging for debugging a specific scenario).

Implementing the configuration of the log level will also add some amount of overhead to the transaction.

In short, you won't get logging without some performance impact. The penalty should be minor but your definition of minor might vary from someone else's definition.