2

We are implementing SFMC for a client without Salesforce CRM but with a legacy loyalty database where all the 1M+ contacts live. My proposal is to do an hourly job to send the changes in a csv file to the FTP as the volume can be quite significant. The client is proposing to send these changes over API. My opinion is that this is a lot of work, expensive in API calls and is prone to errors. Am I missing some obvious benefit here?

7

This depends on how often these contacts are updated. You can easily have 1M+ contacts, but only 10.000 of them have monthly activity, keeping the number of data update calls to SFMC at a minimum.

Some considerations, when it comes to API integrations for large data volumes:

Even when grouping multiple record updates into a single payload, will result in significant numbers of calls. Marketing Cloud has a courtesy limit of 50,000 REST API calls per day, and payload limit of 5MB. This goes for all calls, and not only data updates. So please ensure to calculate the estimated call volumes to see how much "margin" you have with regards to these.

When it comes to peak loads, using a Tenant Specific Endpoint for making an API calls, you can do approximately 3500 concurrent requests per sec, so this also depends on how scalable your source system is.

When it comes to error handling, you might see some interesting cases when using SYNC REST API. Below example shows an error returned when trying to upsert 10 records in a single call. One of them has invalid data, however the message does not provide any info on which record it is, making it difficult to build a robust integration:

{
    "message": "Unable to save rows for data extension ID 43e12d47-6d04-e911-9493-78e3b50b7f0c",
    "errorcode": 10006,
    "documentation": "",
    "additionalErrors": [
        {
            "message": "email: The value for column [email] is not a valid email address. Parse error [InvalidEmailAddress]",
            "errorcode": 10000,
            "documentation": ""
        }
    ]
}

Generally, however, my recommendation for bulk updates is always via SFTP. This will allow you to do a more fire-and-forget approach to integration. Also, using file upload trigger in Automation Studio can also trigger a number of activities e.g. running a query activity refreshing related data extensions.

To summarise: I agree with you

4
  • So well written and very useful, thanks Lukas! – zuzannamj Jan 29 '20 at 17:34
  • Thank you, @zuzannamj - happy to share some good insights – Lukas Lunow Jan 29 '20 at 21:41
  • Thanks Lukas! That SYNC REST API you mention, are you referring to DataEvents for the hub/v1 endpoint? If I I understand correctly that's the only way to upsert more than 1 contact per request over API right? – Arthur Noort Jan 30 '20 at 6:06
  • 1
    You're welcome @ArthurNoort . All the REST DE endpoints support multiple records in one single payload, e.g. the newer ASYNC route as well – Lukas Lunow Jan 30 '20 at 8:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.