We're working with somewhat large sets of data, around 15M - 35M records per DE, each row with 150+ attributes.
What are some ways to optimize SQL queries when handling many records?
- For an exaple, using intermediate DE's and SQL queries selecting TOP 50% from a DE with 14M records will still take approximately 21-27 minutes to run, meaning with any major data increase these queries will time out.
- Data flow is Synchonized DE -> Shared DE -> sub-BU DE per object.
Salesforce's own documentation on optimizing Queries is really basic and not of use this time around.
Here is as example Query
SELECT TOP 50 PERCENT // attribute list // FROM Synchronized DE WHERE // conditions // ORDER BY // condition //