I want to load about a million records into a Data Extension from an external system. Wondering what would be the most efficient way to achieve that?

If I use SOAP/ REST API, how many records should I pass to the API call every time?

Any other suggestions?


Your best bet, due to the large audience is to export the records into a csv and then compress it and send it to the FTP associated with your SFMC account. You can then set up a program in automation studio that is triggered based on this file drop that decompresses it and imports it into your account. From everything I have read, this is the best/easiest way to handle massive bulk imports.

If you are determined to use API, you will likely want to do a bunch of asynchronous upserts using the REST API. I believe the limit is at around 10k per call though.

Similar questions/answers:

ExactTarget: Common data loading strategies?

Adding Bulk data to Data Extension Object in Exact target

How to bulk (import /export) to (data extensions / lists) with (soap / api / sdk)?

  • 1
    +1 for a File Import from FTP. Depends on the cleanliness of your data, but you could save yourself some trouble by using a tab-delimited file instead CSV. Oct 5 '16 at 19:14
  • +1 for SFTP, If you don't want to setup automations via WebUI, or have a dynamic columns definition, I suggest to create ImportDefinition programmatically by mapping the fields , then perform the import definition. I've done it for List Import, DE should be very similar.
    – Bo Hu
    Oct 5 '16 at 23:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.