0

I'm looking for a way to mirror a synchronised data extension into a local data extension.

My SDE is currently counting +- 6mio records and 100 columns (and yes, we need them!). I feed the local DE with a query automation like:

SELECT *
FROM ent.SDE
WHERE ...

currently, it takes up to 30min (if I don't get a timeout error :-) ) Is there a way to speed up the process??

Thanks!

1
  • If my answer helped you solve your issue, please mark it as accepted so others can see this has been resolved. Thanks! Oct 21, 2020 at 6:50

1 Answer 1

0

You could break down your query into multiple ones where the first uses the "Overwrite" option and the following append the further data.

To do so you need a field that you can use to segment the data, so the separation is possible without missing some data.

For example if your DE contains customers and you have a numeric customer number, you could use that field and prepare your sqls the following way:

  1. Customer number <= 50000
  2. Customer number > 50000 AND <= 100000
  3. Customer number > 100000 AND <= 150000
  4. ...

Another option would be only updating records that have been changed since the last run of your automation. (But with that option you'd keep the deleted records in your Sync DE in your local one.)

Further resources on SQL optimization in Marketing Cloud:

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.