Change Data Capture vs Platform Events vs Replication API vs Streaming API vs Batch Data Synchronization [closed]

Question

I have a requirement to sync Salesforce data from 2 objects to Datawarehouse and insert/update data from Datawarehouse to Salesforce.We want this data sync to be realtime for this application.Data volume should be around 10000-15000 records per day.What is the best integration pattern for this use case.

1. Change Data Capture(pilot)
2. Platform Events or
3. Replication API or
4. Streaming API or
5. Third party ETL tool(Batch data synchronization) or
6. ESB (use ESB in between Datawarehouse and Salesforce)

My production org has 60000 API requests limit per 24 hours.How does this API requests effect in each use case.Currently, we don't have any integrations setup with this org and other external systems.

In future,we are also planning to setup daily bidirectional datasync between salesforce and datawarehouse for other Salesforce applications.

Please suggest best integration pattern.

Answer 1

The replication API is designed to have synchronization windows of at least five minutes, so would not be suitable for synchronization of shorter periods. Most ETL tools also don't support windows of less than five minutes, since they tend to use the replication API or SOAP API. ETL stands for "Extract-Translate-Load". There's nothing in there about synchronizing data. While some tools have this capacity to some extent, ETL tools are usually a poor choice for this business purpose.

Of the remaining choices, Streaming API, Platform Events, and Change Data Capture all use the same basic infrastructure, so there's no particular benefit to choosing one over the other for performance or limits reasons. Note that the 60k calls/day limit doesn't matter, because they're streaming, which means that very few API calls are used because the connection remains open for minutes or even hours at a time.

Using an ESB is just an already built solution that would use one of the above APIs. It's no better or worse than using the above APIs. ESBs are typically used to to keep data flowing between a bunch of different systems, so purchasing one just to connect two sources might be overkill. If you plan on connecting more systems later, an ESB might make sense, otherwise you might want to go with a simpler solution.

Aside from ESBs, there are also just plain "connectors" you might be able to find. A connector is a specialized tool that connects some particular type of system to Salesforce. Generally speaking, these tend to be less expensive than ESBs, but there's no hard rule saying that this will always be true.