1

In Marketing Cloud I have 79.131 Email addresses, and 10.939 of these have an Assigned Personalized Send Time.

In Marketing Cloud Journey Builder I want to split my audience into two:
I want those with an Assigned Personal Send Time to go along one path.
I want those without an Assigned Personal Send Time to go along another path.

I want to do this to compare open rates for the audiences and hopefully be able to prove the value of Einstein Send Time Optimization.

How would I accomplish this?
Thank you very much :)

3

I'm the Product Manager for Einstein Send Time Optimization, happy to add a few clarifications and comments here:

  • Einstein STO doesn't load tables of 'Open Likelihood Scores' into Data Extensions, that is correct: instead the Journey Builder activity asks a service for best timings in close to real time when contacts enter the activity in the Journey. This more flexible architecture prevents any syncing issues with DEs, and will let us power the same scoring feature in different systems, such as Pardot later this year.
  • I would recommend thinking of the Einstein STO activity as trying to optimize timings for all contacts to raise Open Rates, not just for those who have a Personalized Model. Contacts who do not have a personalized model are sent in accordance with a generic model which is constructed using all the engagement data that is in your instance. I would advise testing the Einstein STO activity for all contacts going through it, against a regular send at your normal time. Einstein should achieve a little lift with your unmodeled contacts too.
  • You can also test against a pathway which sends at a Randomly selected time; there is an option for the Einstein STO activity to operate in that mode if you want a completely fair control group to compare against.
  • Also make sure that you use the analytics that are provided on the Einstein STO activity itself: if you click it when the Journey is active, you will see lots of rich data including when contacts have exited, the count of contacts who are being held currently and a view of when those contacts will exit the activity. This also includes sub-counts breaking down whether contacts were Personalized or not. That should remove the need to create extracts and files to understand what the activity is doing.

For more detail on how to test Einstein Send Time Optimization's performance with Path Optimizer, feel free to watch a video I recorded on that for Trailhead Live last year here.

1
  • 1
    Thank you Sullivan. Delighted to have a Product Manager assist me in these endeavors. For now I'll go for the Path Optimizer approach in the trailhead video in another business unit in which almost everyone having a Personalized Model, and when I've collected more Personalized Models for my Business Unit with 80K subscribers, I'll do the same test here for confirming the result, but with a much greater audience. Good stuff, have a great day Feb 10 '21 at 10:48
2

It does indeed seem like this information is not exposed in a data extension or public data view.

So by default, you cannot split like this. Einstein does not provide you a record by record view of the times it calculates at this time.

You would have to do an ex post analysis. It's less clean, but since you have no data available at the start, it's what we are left with. At least you would't even need to split for that. Just observe actual sendout times.

Assumption: Set the Einstein setting for "insufficient data" to "send immediately".

Send an email with STO, then look at sendout times the system tracks in _sent, or in a sendlog you set up. Now you can just compare opening or click rates of those whose sent date is leaning towards "immediately" vs. those who got delayed.

Hypothesis would be that you see the "delayed" group perform better. Obvious problem: those whose "optimal sendout time" is the same as "immediately" blur the result somewhat. Sending at a time that is as far away from what Einstein classifies as currently best would fix that, however, this in itself will of course skew your results, those badly timed emails would likely perform worse.

So better accept some blur than to knowingly skew.

If your whole plan is a sort of Proof Of Concept, I would go with this approach. It would be good enough to see if Einstein STO absolutely DOESN'T work.

===

You could also engineer it more, and try and approximate it in two steps, but this will be much more effort, and isn't perfect either:

If you anyway want to lead "unclassified" data down a separate path, you would have to prepare a sort of trial run with a very similar approach: i.e. "observe an STO sendout closely, then split a second sendout, based on the observation" - the sendout times of the "observation" sendout would give you a snapshot of when mails were actually sent to which user.

This will also not be ultra clean as there is a time lag between your observation sendout and the "real" sendout, during which recipients who had no optimal time in the observation sendout get their time calculated.

At any rate: For this you would need a sendout to all your addresses using STO, say in an optimization timeframe of 24 hours. This needs to be more or less shortly before your real sendout, so the data doesn't change too much.

Again, Set the Einstein setting for "insufficient data" to "send immediately". Send an email that includes a sendlog so that you track the sendout time, or use the _sent data view.

In your sendlog (or data view extract), you should see roundabout 69k with a sendout close to the planned timeslot (they were "sent immediately", as fast as journey builder could process), and a lot of others at a different time - those were held back because Einstein knows their "optimal time". Put their Subscriber Keys into a data extension as "STO_optimized; true / false" records, connect the DE to the Contact Model, insert a decision split based on that DE.

Yes, this has business implications and you would need to fit it somehow in your sendout plans. Yes, it is only an approximation, and it's a one time snapshot. And the obvious problem of "people whose optimal date is close to immediately" remains. So really... for a POC, just try STO and look at the results after the fact.

1
  • Thank you, Jonas. Though not exactly what I was looking for and not totally clean, I get both approaches, the uses of data views and really value your suggestions. I'll keep them both in mind for testing purposes. Have a great day Feb 10 '21 at 10:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.