0

I have scrolled through these pages but can't seem to find a solution. Here's hoping someone can help me out :)

I am using the below SSJS in an Automation to remove rows with IsDeleted = 'True' from a DE that has over 45 millions rows in total.

The scrip works successfully with my test DE which I have been able to get up to 24 million rows, but as soon as I run the same script, the same way on a DE with 25M rows or higher, the process does not complete as expected.

Basically, the deletion action is performed, and the IsDeleted = 'True' rows are deleted (and quite quickly, in about 2 minutes I can delete over 12M rows) but the automation seems to hang for a couple more minutes. The automation then is marked green as being successful, but a couple of minutes later the changes are reverted in the DE and the deletion process is undone.

Does anyone have an idea of how I can perform this same action on my live DE with 45M rows? Or an easy solution to remove the unwanted rows?

Thanks, Mitch

<script runat="server" > 

          Platform.Load("Core","1.1.1");

          try 
          {
            var DeleteIsDeleted = Variable.GetValue("@IsDeleted");
            var DeletedResult = Platform.Function.DeleteData('LargeRowDeTest',['IsDeleted'],['1']);
          } 
          
          catch (e) 
          {

            Write("<br><br>Error: " + Stringify(e));

          }
       
</script>
| improve this question | |
  • Are you sure, there is no source that adds the data back to De after deletion? May be synchronized sources? – Swati Mishra Nov 13 at 11:07
  • Hey Swati. Yes, it's a brand new DE with no integrations or syncing. – mitch-w Nov 13 at 11:16
  • Have you tried any other method of deleting? What happens if you manually delete? – Swati Mishra Nov 13 at 11:19
  • As part of my testing I clear out the DE and then add new information. I am able to add 24M rows of data, run the script and the deletion works and it's all good. But then when I add 25M rows or more, this is when I hit issues. I can also manually select 1 row and delete, and it works. – mitch-w Nov 13 at 11:21
  • 1
    I would advice using SQL if it somehow fits the usecase. You could just do an automation with multiple sql queries run after each other and therefore also make some changes. Example: Run query on Master Dataextension and only select those you dont want to delete -> store them in a temp dataextension with overwrite. afterwards take the temp dataextension and overwrite the master dataextension. Could this be a possible solution for you? – Johannes Schapdick Nov 13 at 11:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.