Marketing Cloud - SSJS Activity timing out. Not sure of error

Question

I have an SSJS Activity I am trying to get to work wherein we are calling an API endpoint using basic authentication (obviously the user and pass are obscured in production). But I am having issues with handling the size of response. In QA, everything works fine because the response payload is manageable. But once I go to production, the API call is taking around 2 minutes to get a response and I am getting a timeout error in Automation Studio. Can't troubleshoot in Cloud Pages, as the page just never loads. No error, even using try catch.

Wondering if I am just hitting runtime errors that I don't know how to handle either in the HTTP request or in Cloud Pages. I know the runtime for SSJS Activity is 30 minutes in theory, but it is erroring much sooner than that.

Error in automation studio: "A runtime error occurred while the SSJS Activity executed the SSJS script. Please review the SSJS activity to ensure that errors are handled properly."

Here is my script:

    <script runat="server">
    Platform.Load("Core", "1");
    
     var url = "https://endpoint.x";
     var user = "user";
     var password = "pass";
    var auth = 'Basic ' + Platform.Function.Base64Encode(user+':'+password);
    
    var headerNames = ["contentType", "Authorization"];
    var headerValues = ["application/json", auth];  
    
    var req = new Script.Util.HttpRequest(url);
    req.emptyContentHandling = 0;
      req.retries = 2;
      req.continueOnError = true;
      req.contentType = "application/json"
      req.setHeader("Authorization", auth);
      req.method = "GET";
      var resp = req.send();
      var authRequest = resp.content;
      
      
    var json = Platform.Function.ParseJSON(authRequest);
    var AccountCartsDE = DataExtension.Init("xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx");
      
    var insertresult = AccountCartsDE.Rows.Add(json);
      
    </script>

Answer 1

maybe not the answer that is easiest to change to, but probably will address load based problem. Likely the parser cannot handle the data. Try to avoid script when you cant control size, try to shift as much to sql as you can.

so:

• isolate that the HTTPGET works without the parseJSON function, using production data (or data that is comparable in size). If yes, the rest can make sense. If the GET alone already times out, forget whats next, it will obviously be besides the point.

• build a data extension with a field "data" without maxlength (use API for this, or your maxlength will be 4000).

• build an automation that uses sql to parse data from that field into your targetDE

How to parse json using SQL in Marketing Cloud

• in your original script, execute the HTTP GET and store the JSON in the data field, then start the above automation using its objectID. This will shift the heavy lifting to sql and out of the ssjs, avoiding the timeout (assuming its not already in the GET)

var prox = new Script.Util.WSProxy();

function startAutomation(objectId) {
    var action = "Start";
    var props = {
        ObjectID: objectId
    };
    var opts = {};
    var data = prox.performItem("Automation", props, action, opts);
    return data
}

var performedAutomation = startAutomation(myAutomationObjectId);