I do remember seeing documentation articles defining some soft limits for the number of records and attributes for Data Extensions but could not find it now. Can you please provide some references for any existing limits/performance impact based on amount of DEs, their sizes etc.
-
So far, I've found only this document not mentioning number of attributes specifically. [link] (developer.salesforce.com/docs/atlas.en-us.mc-apis.meta/mc-apis/…) – Artur Krotkov Jun 11 '20 at 14:14
Add a comment
|
Here is the closest to Official Best Practices that I have found. It does not really list anything around fields or volume though.
To my knowledge (and verified by @EliotHarper here) there is no actual limit to the number of fields you can have, nor the total volume of records you can have in a DE. But, the larger and more significant the storage required for the DE, the slower and less performant it will be. Eventually making the entire DE completely useless as all related activities will time-out or error. Also, the processing and storage that is being used on that DE will also affect all your other operations, meaning you will experience a slow down overall in your Account. (I am talking about significant size. E.g. the DE is multiple gigabytes in size).
In general, I would recommend if you are going to have a ton of fields in the DE, that you limit the volume and vice versa. There is no real definitive guidelines as each case is unique, but overall size - I would make sure to limit your DEs to under 1 gigabyte total size.
I know that is not always possible, but (from my experience) anything over 1 gig will have a slight slow down, anything over 5 will experience a significant slowdown and anything over 10 will be relatively unusable in platform. If your DE is required to be 10 gigs or bigger in size, I would look at ways to turn this into a relation system to separate out attributes or records into chunks instead of all inside a single location.
answered Jun 11 '20 at 15:12
-
1Great overview. Also, worth mentioning is following general SQL best practices: Use indexes where applicable (primary key DE fields automatically become indexes), join on integers rather than strings, keep total width below 4000 chars (which implicitly means no NVARCHAR(MAX) fields) – Lukas Lunow Jun 11 '20 at 19:34
-
1Unusable anywhere approaching this, but here'd be a hard limit of 1,024 fields as per SQL Server's limitation docs.microsoft.com/en-us/sql/sql-server/… – Macca Jun 12 '20 at 9:59
