1

I'm building a data replication system using SF Rest API. I see quite a few different ways to reach my goal. Can somebody highlight the differences between some of these possibilities and provide insight into what's recommened:

  1. SObject Get Updated (and deleted) (Understood that this won't work past 30 days)
  2. Query All
  3. If Modified Since header
Improve this question
1

2 Answers 2

Reset to default
3

Data replication systems need two use cases:

  1. Initial loads or syncs, where you obtain all data for some object to persist.
  2. Incremental or real-time updates during use.

You've identified some of the features that suit those use cases.

sObject Get Updated and Get Deleted are solutions for (2). It is their design purpose to be used for polling in data replication. If you want to use real-time (push) replication instead, look at Change Data Capture.

Query All is a potential solution for (1), but at larger data volume you should definitely be looking at the Bulk API rather than the REST API. The (asynchronous) Bulk API is designed for extracting and loading data efficiently at massive scale.

The If-Modified-Since header isn't really going to help you on large-scale data replication. It would only provide notification that a specific, relatively small set of sObject Ids had not been modified; that is, it's an inefficient form of polling.

Improve this answer
6
  • Correct about the 2 use cases. I was thinking that maybe it would be smarter to look for a solution that could support both. Is Query All not recommended for Option 2 as well? (ie query with LastModified > timestamp)?
    – FizzBuzz
    Feb 10, 2020 at 3:55
  • Hmm. I do not have benchmarks to hand but I would naively expect that would be less performant. And I believe the semantics differ relative to hard-deleted records.
    – David Reed ♦
    Feb 10, 2020 at 4:13
  • After getting a list of updatedIds, I need to get the updated records by list of ids - is Query the right goto there?
    – FizzBuzz
    Feb 11, 2020 at 4:02
  • No, you don't need to execute a query - you can use Sobject Collections Retrieve to get record data in batches by id.
    – David Reed ♦
    Feb 11, 2020 at 4:29
  • 1
    I think you'll probably have to do some benchmarking and/or evaluate the expected patterns and data volume for your real application to answer that. It would make sense to me that if you exceed some threshold count of Ids relative to the total table size that you might want to re-sync the whole table via Bulk API. Remember there are limits on how many Ids you can include in a query and it's not that high (~200, as I recall, although it may have gone up in a recent release).
    – David Reed ♦
    Feb 14, 2020 at 19:11
0

If API limits are a concern, sObject Get Updated and Deleted may be the right way to go since only 5 QueryAll requests can be joined together in a composite call and 25 getUpdated requests can be joined.

Improve this answer
1
  • @DavidReed Any reason not to group these calls in a composite request?
    – FizzBuzz
    Feb 11, 2020 at 4:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .