0

I am facing an issue that my automation will throw out a runtime error if it already ran about 30 mins and hadn't been finished yet. I checked and only a part of the data (a few hundred records in the data extension) was processed.

enter image description here

How do I handle this issue? My data extension will be renewed every next morning and I am using activity script (SSJS) to process it.
Another thing I wonder is If I use SSJS via WSProxy, will it be faster? Thanks.

2
  • You might find this useful sfmarketing.cloud/2020/06/05/… by SFMC MVP Zuzanna Oct 8, 2021 at 3:33
  • It's a nice article but I already used the try catch statement in the script. Therefore I think the reason for this is that the number of records to process
    – Duc Le
    Oct 8, 2021 at 4:18

1 Answer 1

2

Script activities do have timeouts.

Large data volume, especially stuff like refreshing data extensions, is generally something that you want to process with SQL, not ssjs.

The problem is in the general approach, looking into WSProxy will probably just change details (a few hundred / thousand records more or less until it times out).

==

With that disclaimer out of the way - it probably is the answer to 90% of usecases:

If you REALLY REALLY need to iterate over a ton of records with script - because you do NOT simply process data, but want something that SQL cannot technically do, say, send API calls. Then you need to limit the number of records per script activity iteration.

I've laid it out here when I had to make SSJS based Bulk API calls with batches of 10000 records each:

Account Activity or Object Activity in Journey Builder has limits to update like Data Loader?

Similarly, you will need a batching logic that caps the number of records each SSJS script activity processes.

Essentially, you need to loop through batches.

As automations run in a linear way, activity by activity, from start to finish, an automation cannot "decide" to just return to a previous activity and repeat.

In other words: a running automation cannot restart itself.

In other words: a single automation cannot loop.

An SSJS script activity could loop / restart itself, but only for so long - it will time out. That's exactly what you have now.

Automations can decide to start other automations though. So you can write connected Automations that form a loop together. The loop should stop once no more records are there.

Diclaimer part 2: Mind you that there theoretically is a contractual limit to Automation runs, so this is not a solution you throw at any problem that you can solve with SQL. Plus it is a lot of time and effort.

In the basic setup, you need three Automations, in the linked solution it had to be four.

One "scheduler" that runs at a predefined time, two "batch processors" that get called or call each other. Again - These automations should perform whatever data operation with SQL where possible (checking number of records, marking records), not script.

Automation 1

  • runs on a schedule.
  • Checks if there are more than 0 unprocessed records in your DE
  • If number of records is > 0: Call Automation 2. Else: End.

Automation 2:

  • has no schedule, so it can be started by other automations through an API call.

  • processes a defined max number of records (SELECT TOP...)

  • then marks them in the source as processed in some field you set up.

  • if there are more than 0 records unmarked, it calls Automation 3. Else: End

Automation 3

  • same as 2
  • same as 2
  • same as 2
  • if there are more than 0 records unmarked, it calls Automation 3. Else: End

Bonus disclaimer: Sure you can't do it with SQL?

3
  • Great post, Jonas! My main data was from an SQL query. I don't think my data will be up to 10000 records so may be WSProxy seems like a promising solution. As I understand, I may need to segment my main data extension to smaller ones in order to use them in connected automations, right? Basically, there's a script (SSJS?) where it can be able to call automation from there? Can you refer me the source where I can read about connecting up Automations? I am not aware I know about it
    – Duc Le
    Oct 8, 2021 at 7:58
  • click the link in my post, I referenced the source for calling an automation through api and also had more details about batching. 10000 records was just a random anecdote (they are the SF Bulk API record limit) , that has nothing to do with WSProxy or SSJS. I do not think that wsproxy is anymore promising, give or take a few seconds or rows processed. If your script runs 30 minutes, you have a more fundamental problem than the ssjs / wsproxy choice. Oct 8, 2021 at 8:26
  • 1
    Also as an alternative, what I do instead of calling multiple automation is make the script activity run for ~25 minute via a do/while loop and then have two inside an automation scheduled to run hourly. I then add a sql query and validation step before it to stop the auto if there is no more data to process. This removes the risk of the API call to start the next automation failing and adds a break point to stop needless processing. Oct 8, 2021 at 11:47

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .