Can we Run An Automation for 24 Hours by adding repetitive activities to ensure we are processing data we captured in Data Extension

Question

We need to continuously process data(need to make an API call using SSJS script activity to external system which is time sensitive) which we are capturing in Marketing Cloud Data Extension via SFMC Collect code hosted on our website.

And we need to ensure we are processing the data collected in SFMC DE 24*7

We are having 4 Activities including - 3 SQL activities and 1 SSJS activity in an automation to process the records. Can we schedule an Automation and add these 4 activities repetitive to continuously run for 24 hours ?

We cant think any other alternative solution to achieve the task in marketing Cloud as SFMC collect code capturing customer details in Data Extension.

Answer 1

Running a single automation for 24 hours is theoretically possible, but far from recommended. Main reason: should your current automation fail to finish within 24 hours, the next run will be skipped. Leaving you without any processing for close to 24 hours.

Another approach is to add a Script Activity in the last step of your automation, which will trigger same automation once current run has finished. There is a catch, however. You will need to duplicate your automation, as an automation cannot trigger itself (you will receive an error, as it is already running). Instead, you will have two automations, A and B. A will have a script triggering B and vice versa. Here Adam Spriggs provides some advice and SSJS code for triggering automations.

Depending on how often they will end up triggering each other, you might end up with a high number of automation runs - more than 100.000 annual runs, just for this process (with an estimated run time of 5 minutes). You could repeat the 4 activities - 3 SQL activities and 1 SSJS a number of times, in your A and B, hence reducing the number of times you need to switch automations. E.g. having them all occur 10 times, will reduce the number of runs to 10.000. Indeed a more tolerable number.

Mind you, this will either way not be real time, but near real time. Since you will need to wait for the SQL to run, before the SSJS can do the processing stuff.