Even Driven Enterprise External Integration with Salesforce and External Applications

Question

I am asking about Salesforce Event Driven notifications and similar notification frameworks.

We are trying to integrate our CRM (Salesforce) with our ERP (External Applications located on Private Network).

I have gone through the following:

• Trailhead Module - Platform Events Basics Trailhead module.
• Integration Patterns and Practices - Remote Process Invocation—Request and Reply
• Integration Patterns and Practices - Remote Process Invocation—Fire and Forget
• Developer Guide - Salesforce Streaming API
• Documentation - Outbound Message Notifications

One thing that we have noticed is Message Durability is limited to 24 hours in all approaches.

• Message Durability
  Salesforce stores events for 24 hours. With API version 37.0 and later, you can retrieve events that are within the retention window. The Streaming API event framework decouples event producers from event consumers. A subscriber can retrieve events at any time and isn’t restricted to listening to events at the time they’re sent.

How can we meet following requirements from Salesforce Event Driven Notification Platforms (Streaming API, Outbound Notification, etc.)?

1. We need to have real time update.
2. We should able to support 50000 transactions (as per the data uploader tool) per second.
3. Sometime external systems (except Salesforce Org) will shut down for more than 24 hours. Once these systems are back up, information should flow to downstream.

Answer 1

Sounds like a direct connection between Salesforce streams and system subscribers would make for a very challenging integration.

I suggest you don't connect those end systems to Salesforce. Instead do the following:

1. Set up the publishers for the Salesforce data
2. Create a cloud script that subscribes to the feed, stays awake, reconnects and ultimately pushes the data it collects to a cloud based pub/sub system that provides more durability and flexibility.
3. That system i.e. Kafka can then retain and re-publish the data for these 3rd party systems
4. Set up the other systems to subscribe to the non sf pub/sub system.

By decoupling from Salesforce for pub/sub you can have your streaming cake and consume it too.