0

I want to load about a million records into a Data Extension from an external system. Wondering what would be the most efficient way to achieve that?

If I use SOAP/ REST API, how many records should I pass to the API call every time?

Any other suggestions?

1 Answer 1

4

Your best bet, due to the large audience is to export the records into a csv and then compress it and send it to the FTP associated with your SFMC account. You can then set up a program in automation studio that is triggered based on this file drop that decompresses it and imports it into your account. From everything I have read, this is the best/easiest way to handle massive bulk imports.

If you are determined to use API, you will likely want to do a bunch of asynchronous upserts using the REST API. I believe the limit is at around 10k per call though.


Similar questions/answers:

ExactTarget: Common data loading strategies?

Adding Bulk data to Data Extension Object in Exact target

How to bulk (import /export) to (data extensions / lists) with (soap / api / sdk)?

2
  • 1
    +1 for a File Import from FTP. Depends on the cleanliness of your data, but you could save yourself some trouble by using a tab-delimited file instead CSV. Oct 5, 2016 at 19:14
  • +1 for SFTP, If you don't want to setup automations via WebUI, or have a dynamic columns definition, I suggest to create ImportDefinition programmatically by mapping the fields , then perform the import definition. I've done it for List Import, DE should be very similar.
    – Bo Hu
    Oct 5, 2016 at 23:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .