How to improve email delivery speed for highly dynamic email (fetching product info)

Question

We're looking to improve the send time of a recurrent email campaign.

The campaign has for objective to push a given number of products in an e-mail (max 20 products). The email contains quite some AMPscript of course, for standard personalisation, pref. center, some dynamic content here and there and finally this heavy product content block. Currently, when sending this email to an audience of 800k, the platform takes between 6h and 9h to execute the send. And this is way too long.

What consumes a lot of time in our opinion is the products lookup in the product catalog. The catalog has around 15k records. And for each subscriber of our entry source, AMPscript looks up this product catalog to match one of the 10 productID's defined as variables in the email body and return some attributes (Name, Price, ImageURL,...).

We have 2 potential solutions and I'd like to get the community's opinion on which one would perform best:

1. We thought about adding a step to the campaign process which would be: define the 20 productID's in a data extension rather than AMPscript variables. Using this data extension, we could run an automation that would query the product catalog and isolate the products' attributes beforehand. We then modify our AMPscript to look up this new data extension of 20 records rather than 15k records data extension.

2. Radically shift the computing power from SFMC to Heroku using a custom content block. Instead of having AMPscript logic in the e-mail, user would have to input productID's in the custom content block when creating the e-mail. At send time, the productID's would be sent to the App on Heroku which would lookup either its own dB or call SFMC product catalog to retrieve records (to be decided but local storage of the products will be quicker to query). The output would be a piece of HTML, ready to be displayed by the email itself.

I personally think that solution 1 can already improve a lot and could even be improved on user-friendliness side.

Solution 2 is much more user-friendly but is also more blurry to us as API calls will take time to execute. This being said, the product lookups and HTML construction will happen outside any other side process that could be running in // of a journey execution (automations, other email sends, S50 on friday, ...)

Therefore I would like to the community's opinion on this matter in order to solve this in the best way possible.

Many thanks for your help,

Answer 1

2 is absolutely worth investing time into.

Having done both, the second approach is much stronger because if a mail creator actually enters the same product ID for all users, you can do one API Call per product at editing time, instead of repeating the same lookup 800k at send time. This will be obviously beneficial to performance, but maybe even as importantly, it opens the door to much better usability:

When you have a Custom Content Block that can pull in data when the creator enters a product ID, this also means that they can preview the final content block already when editing.

And when you invest some more time, you can also construct logics where you pull the data in by product ID, and then allow changes after the lookup - e.g. the mail creator pulls in the product data, spots an error in the description and then can fix that.

Do not overlook that you control the UI of a Custom Block, so you can even tailor this to specific fields only, and make e.g. prices non-editable but descriptions editable after pulling them out of your DB.

Answer 2

There are couple of additional items to consider.

The first thing is to consider Burst Sending - it's a paid add-on meant to send an Email to large audience as fast as possible. This will lock down the audience and pre-build the Emails to be ready for the scheduled send time.

It is very useful in case your Emails will contain a lot of variables and dynamic content and are sent to a large audience.

In terms of Email design, lookups are indeed the main source of slowdowns. Especially fuzzy matching or looking up from multiple DE's. Ideally the pre-built target group will have all the variables available in the target Data Extension unless there's a possibility the data may change before the send.

It won't hurt to look through the Email Sending Speed Optimization trailhead as well.