Synchronize Data between External System and ISV app

Question

My AppExchange Managed Package and its related External System share a central data object. Records of those objects can be created, modified, and deleted both in Salesforce and the External System. I am now thinking of how to build this integration best and see those two options:

1. Packaged Platform Event ObjectChange__e, which will be emitted and consumed by both backend (using a ConnectedApp) and app
2. Two separate custom APIs. An Apex REST endpoint and one on the backend to receive notifications from the other side

I never worked with Packaged Platform Events, but from what I have read I lean strongly toward the Platform Event solution because:

3. Less custom code for Endpoint creation
4. Platform events seem to have better Limits that Apex Rest
5. The asynchronous nature of Platform Events with Replay and real async look like a much more stable and flexible "decoupling"

Did you already build one or the other solution? Why and how did it work out? Are they good docs that argument for on or the other?

Answer 1

A bit late to the party, but here's an answer composed from my comments around integration with off-platform systems.

General Points

Previous Experience

We have historically used callouts to send messages from Salesforce to an external system, and had the external system use the REST or SOAP APIs to provide updates (or call bespoke processing).

On some occasions we have had to do it all from the Salesforce side, using polling to fetch updates from the external system.

It rather depends on the integration patterns that the external system supports.

Keep the integration outside the core package

We have always decoupled this from the core product. IMHO you should always put integrations in a separate "extension package" - now even easier to do given you have the ability to have multiple 2GPs in the same namespace and to have APIs between these that are hidden from package users.

We do this because customers frequently have different use cases for a (possibly common) external system or require integration between our product and different external systems.

Roll your own or use middleware

If you are doing point-to-point integration, rolling your own is OK. However, if you need to integrate with more than one external system it is almost certain you should use middleware to do the coordination, routing, data transformations etc.

See this other Q&A for more on this topic.

Platform Events and why to be cautious

Platform Events have no guarantees for delivery. They can get lost if the infrastructure suffers an issue. For me that means you should not use them to transport valuable data that you cannot reconstruct from the data in the database.

This is covered in the documentation thus:

In rare cases, the event message might not be persisted in the distributed system during the initial or subsequent attempts. This means that the events aren’t delivered to subscribers, and they aren’t recoverable.

Platform Events also have significant limits on CometD (and other) transport. This can prevent you from scaling your integration. Obviously there are also REST/SOAP API limits to worry about, but these are higher.

Answer 2

I actually went the route of pub/sub AND a custom API. I pushed updates in salesforce to Amazon SNS/SQS and the other system that had 500k customers hitting it had the custom API for more real-time push back to SF as well as some functions to manually pull certain data.

Found exactly what one of the commenters said - network hiccups didn’t get me to 100% reliability on delivered messages. Platform Events, Streaming API, everything SF-based seemed to at the end of the day be missing ~5% of messages after audits when an external system was involved.

Also bolted Loggly into the process with alerts if errors were sent or gaps in data did happen. Nothing is 100% but after getting burned with PE and SAPI, I never looked back.

Ultimately there was an SF help article that talked about a protocol incompatibility or something (this was years ago, tried to find the link) when communicating either direction with PE or SAPI to an environment within a high availability setup (load and region) at AWS. Talked back and forth with a Sr. Product Manager and after about three weeks of struggling, we had to flip to the SNS/SQS and custom API model. The recommendation from Salesforce was to do periodic playbacks of messages to do a durability check, but we opted not to build a whole app layer to manage that.

Qualifying...this was a "higher"-volume situation, probably 100-200k events in 24 hours. PE within the platform is from what I have experienced firsthand, bulletproof. I've never had an undelivered or missed message when I'm just cycling the event back to another "thing". I did also deploy cross-platform PE for some lower-volume applications (under 50k/day) and it seemed to work into a groove where I never heard of any glitches.

Answer 3

Platform events has some painful limits (https://developer.salesforce.com/docs/atlas.en-us.platform_events.meta/platform_events/platform_event_limits.htm), especially: Event Delivery: maximum number of delivered event notifications in the last 24 hours, shared by all clients. (Applies to CometD and Pub/Sub API clients, empApi Lightning components, and event relays only.) = 50,000 in Performance and Unlimited Editions.

Before making any decisions about implementing PE should be preceded by a meaningful analysis of how many PEs per day you want to publish; and keep in mind that it's a shared limit for the whole org, not only for your package, so consuming too much can make serious issues.

The second issue is what Phil already mentioned, that sometimes PE may not be delivered, so there is a need to use some ESB or extra code to retrieve events. In the Managed package, you cannot make assumptions that your customers always use ESB, so anyway it will require extra development.

In managed package I would by default consider building Apex REST endpoints.