I am facing an issue that my automation will throw out a runtime error if it already ran about 30 mins and hadn't been finished yet. I checked and only a part of the data (a few hundred records in the data extension) was processed.
How do I handle this issue? My data extension will be renewed every next morning and I am using activity script (SSJS) to process it.
Another thing I wonder is If I use SSJS via WSProxy, will it be faster? Thanks.
Script activities do have timeouts.
Large data volume, especially stuff like refreshing data extensions, is generally something that you want to process with SQL, not ssjs.
The problem is in the general approach, looking into WSProxy will probably just change details (a few hundred / thousand records more or less until it times out).
==
With that disclaimer out of the way - it probably is the answer to 90% of usecases:
If you REALLY REALLY need to iterate over a ton of records with script - because you do NOT simply process data, but want something that SQL cannot technically do, say, send API calls. Then you need to limit the number of records per script activity iteration.
I've laid it out here when I had to make SSJS based Bulk API calls with batches of 10000 records each:
Account Activity or Object Activity in Journey Builder has limits to update like Data Loader?
Similarly, you will need a batching logic that caps the number of records each SSJS script activity processes.
Essentially, you need to loop through batches.
As automations run in a linear way, activity by activity, from start to finish, an automation cannot "decide" to just return to a previous activity and repeat.
In other words: a running automation cannot restart itself.
In other words: a single automation cannot loop.
An SSJS script activity could loop / restart itself, but only for so long - it will time out. That's exactly what you have now.
Automations can decide to start other automations though. So you can write connected Automations that form a loop together. The loop should stop once no more records are there.
Diclaimer part 2: Mind you that there theoretically is a contractual limit to Automation runs, so this is not a solution you throw at any problem that you can solve with SQL. Plus it is a lot of time and effort.
In the basic setup, you need three Automations, in the linked solution it had to be four.
One "scheduler" that runs at a predefined time, two "batch processors" that get called or call each other. Again - These automations should perform whatever data operation with SQL where possible (checking number of records, marking records), not script.
Automation 1
Automation 2:
has no schedule, so it can be started by other automations through an API call.
processes a defined max number of records (SELECT TOP...)
then marks them in the source as processed in some field you set up.
if there are more than 0 records unmarked, it calls Automation 3. Else: End
Automation 3
Bonus disclaimer: Sure you can't do it with SQL?
