0

After a lot of wrestling with the Script Activities containing SSJS for removing unwanted rows from my DEs and noticing that there are limits on the number of rows they can delete in a for-loop (2500) and how slow that process is, I have decided to delete rows using SQL Query activities and staging (temporary) DEs.

To this end, I will have to rewrite the DEs that are connected to my contact model on a daily basis. I still use delta (diff) files for updating but at some point, I would have to empty my main DE and insert and adjust the correct data in a temp DE (with new rows added, current ones updated and the deleted ones removed) back to the main DE.

I am wondering if there are any consequences to this approach, knowing that a journey in Journey Builder can have an entry point DE that updates its data based on the contact model. Does disappearing/reappearing of records in a DE affect my journeys (or anything else)? Would it be new journey's triggered? Would existing data re-entered every day be treated as new rows?

Improve this question

1 Answer 1

Reset to default
0

If processes touch the data extensions when they are overwritten, issues could arise. The data that it needs to do a decision split may not be there at the time of the decision split.

When you perform a query with an overwrite setting, it drops the table first and then populates the information. Therefore leaving a period of time in which no data is available for the journeys.

If you are using it for entry events, then you need to look at your Entry Event's settings such as high water mark and how an update or an overwrite relates to the high water mark setup.

If using the highwater mark and only want future rows recognized, you would likely use append. If not using highwatermark and want to evaluate all records each time the automation for the journey is ran, you would likely overwrite so that it runs through all of the records on the DE you are targeting with your queries.

NOTE on this comment: "knowing that a journey in Journey Builder can have an entry point DE that updates its data based on the contact model." Be aware that the Journey data for a contact injected into a Journey does not change based upon the entry DE.

https://help.salesforce.com/articleView?id=mc_jb_journey_contact_data.htm&type=5

If the entry DE is linked to your contact model, then the data contained in the Entry DE will affect decision splits that are using the Contact data in the applicable attribute group.

Improve this answer
5
  • What do you mean by High Watermark?
    – Disasterkid
    Aug 22, 2019 at 12:00
  • The entry DE is not directly linked to the Contact Model. It uses SQL to fetch that information. I update the information in my data extension to keep up with the latest info (e.g. whether a flight is cancelled or a booking is updated). So that during the journeys, we recheck before an email send if the data still holds.
    – Disasterkid
    Aug 22, 2019 at 12:03
  • Also, the update of the contact model is done at midnight and the journeys are currently based on a scheduled run of automations.
    – Disasterkid
    Aug 22, 2019 at 12:05
  • May not be called high water mark any more, but you will need to select a processing option. Read under heading "When you set a schedule that repeats, you select a processing option." help.salesforce.com/… If the update is done at midnight, and the journeys are scheduled outside of that time, then there would not be this conflict. If a contact hits a decision split at midnight, that could be an issue then. It is about timing and settings in your scenario.
    – Mark_F
    Aug 22, 2019 at 12:59
  • High water mark. It's more of a conceptual term in this case. Using a DE entry, when the journey entry runs, it only processes records that are new since the last time it ran, or where the high water mark was previously. After this process, a new high water mark is set.
    – Josh de Blank
    Aug 23, 2019 at 0:09

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .