I have to implement a Marketing Cloud Solution that will have millions of records in a single Data Extension (DE). Most DEs will have 20-30 million records with some DEs going over 100 M. I know how to write a query following the best practices, but I'm not sure that MC can work with so much data.
Taking a simple example with DE A having 20M records and DE B having 30M records. A flag (checkbox field) is updated in DE A by API calls for all records and all records from DE A are in DE B. DE B has the same checkbox and needs to be updated daily based on data from DE A. Can a simple Update query process all those records and update DE B before reaching the 30min limit?
It's even possible to process this many records? To do a simple update on a 30M record DE?
- If the answer is 'Yes', what happens if I need to do a simple Join between DE A and DE B and update the result in DE C? It's still safe to assume the limit will not be reached?
- If the answer is 'No', what is a way to process so many records in MC? I know it should involve multiple DEs and Queries, but I have no idea how the data can/should be split and put back together. Can you please provide some information?
Similar question for SSJS activities. From what I know it's not possible to process this many records using SSJS, but please let me know if I'm wrong or if you have any idea about how I can split the data to be able to process it in SSJS.
And before you ask, yes I tried talking to Salesforce about this, but to no avail. Salesforce is avoiding a clear response on this subject, all they could tell me was to follow best practice and everything will work, but I'm not so sure 'best practice' will be enough taking into consideration the volume.
If you ever faced this kind of problem or if you know how to work with large volumes of data in MC, please tell me how you did it and what was the approach.