Does Platform Event / CDC uses same Async queue as Queueable/ Future?

Question

We have a huge concurrent influx of incoming custom api requests. (1000+ under 5 seconds approx) Because of this we were getting ConcurrentPerOrgLongTxn Limit exceeded.

We moved from Sync processing in Custom web service to Async processing. i.e we used System.enqueueJob to process the request asynchronously, assuming the processing would take max 1 second.

From past few days a huge influx of events started coming to our custom webservice, and our System.enqueueJob gets fired, but the requests goes in Queued status, We can see after running this SOQL

SELECT COUNT() FROM AsyncApexJob WHERE Status = 'Queued' Some 70K records are in Queued status and only 3-4 in Processing status.

Not sure of the reason though. But we are planning to move to Platform Events or CDC instead of System.enqueueJob, and then have an Async Trigger for CDC or Listen to Platform Events from trigger and then process the incoming requests.

My question is "Does CDC/Platform Event uses the same Async Queue which System.enqueueJob (Queueable) uses?"

Answer 1

You'll see that AsyncApexJob lists out what mechanisms use it

Represents an individual Apex sharing recalculation job, a batch Apex job, a method with the future annotation, or a job that implements Queueable

Platform events do not use the same "queue" as Phil answered and have their own, separate limits to consider and track with the services/data/v54.0/limits endpoint

• HourlyPublishedPlatformEvents
  • High-volume platform event notifications published per hour
• MonthlyPlatformEvents
  • If your org doesn’t have the high-volume platform event or Change Data Capture add-on. Includes usage for both high-volume platform events and Change Data Capture events.
• MonthlyPlatformEventsUsageEntitlement
  • If your org has the high-volume platform event or Change Data Capture add-on. This value includes usage for both high-volume platform events and Change Data Capture events.

Important to note that Change Data Capture (CDC) "change events" are specialized platform events. As such, the allocations for CDC (and platform events above) are shared when it comes to event delivery and why some of the above contains delivery information for both combined.

Change events are based on platform events. The default allocations and usage-based entitlement of delivered events are shared between change events and high-volume platform events.

Monitor Platform Event Publishing and Delivery Usage goes over how the REST limits endpoint provides info on both CDC & Platform Events delivery - but, there's a PlatformEventusageMetric which allows getting usage with a distinction between the two.

The usage metrics stored in PlatformEventUsageMetric are separate from the REST API limits values. Use the REST API limits to track your monthly delivery and publishing usage against your allocations. The monthly CometD-client delivery usage that the limits API returns is common for platform events and change data capture events. PlatformEventUsageMetric breaks down usage of platform events and change data capture events so you can track their usage separately.

---

In terms of what does it use if it's not using AsyncApexJob - the Considerations for Publishing Platform Events goes into some details on what it's doing

Platform events are temporarily persisted to and served from an industry-standard distributed system during the retention period. A distributed system doesn’t have the same semantics or guarantees as a transactional database. As a result, we can’t provide a synchronous response for an event publish request. Events are queued and buffered, and Salesforce attempts to publish the events asynchronously. In rare cases, the event message might not be persisted in the distributed system during the initial or subsequent attempts. This means that the events aren’t delivered to subscribers, and they aren’t recoverable.

Since you seem most interested in subscribing through Apex Triggers - Subscribe to Platform Event Notifications with Apex Triggers covers that area

An Apex trigger processes platform event notifications sequentially in the order they’re received. The order of events is based on the event replay ID. An Apex trigger can receive a batch of events at once. The maximum batch size in a platform event trigger is 2,000 event messages. The order of events is preserved within each batch. The events in a batch can originate from one or more publishers.

Unlike triggers on standard or custom objects, triggers on platform events don’t execute in the same Apex transaction as the one that published the event. The trigger runs asynchronously in its own process. As a result, there can be a delay between when an event is published and when the trigger processes the event.

Answer 2

Does CDC/Platform Event uses the same Async Queue which System.enqueueJob (Queueable) uses?

No. Platform Events are not actually async executions; they are essentially synchronous executions in a separate transaction, where the decoupling is via use of an "event queue" of some kind.

One of the problems I have with the way the documentation frames the execution as "async" is that the platform considers then as synchronous, from the limits perspective. The documentation authors completely failed to consider this aspect.

That is why I frame it as "synchronous" in a decoupled transaction where the (distributed, in-memory) queue provides that decoupling, and the "queue consumer" processes the invocations for trigger and CometD subscribers.

My understanding is that this "event queue" does not persist in the database, which is how come delivery is not guaranteed - the queue is in-memory and thus could get lost if the server/cluster dies. (I imagine the "queue" tech is something like IBM MQ, Oracle AQ or similar but have no actual knowledge of this.)

Importantly, if a behaviour is not explicitly documented by Salesforce they can, in principle, choose to change it at any time. I would therefore say you probably don't want to rely on whether multiple trigger-based subscribers run sequentially or in parallel since as far as I know this is not documented. (Kris Goncalves's answer nicely links to the official documentation available.)