1

Dear ExactTarget specialists,

I am trying to get a hang of how the ExactTarget Marketing Cloud works, and I am struggling in finding information on a seemingly basic task: Loading data into the platform.

These questions may be awkward and n00b-like, but I hope you don't mind me asking:

  • I have gathered that I need to create "Data Extensions" for a customer record which in turn store any kind of data I need to be able to segment my customers; is this correct?
  • Our data sources consist of different systems, like SAP, Remedy, and an Oracle database containing various kinds of information I need to be able to use in segmentations; would I design Data Extensions based around the data sources, or in some other way? Would I use multiple Data Extensions if I need to get large amounts of data from a single source?
  • How does the import process into the platform work? Would I normally use the "Import Activity" from an "Enhanced FTP directory" in ExactTarget? Or would I invoke the REST API to update the records in batches?

To set the scene a little: I am facing 5-10 million records with a daily change rate of about 1000-5000 (new orders, updates,...).

I know my questions are very general, perhaps there is some kind of documentation (I was not able to find) to give me some pointers?

Best regards,

Martin

2

1) I have gathered that I need to create "Data Extensions" for a customer record which in turn store any kind of data I need to be able to segment my customers; is this correct?

Yes. Data extensions might work better based off of the type of data you have described.

2) Our data sources consist of different systems, like SAP, Remedy, and an Oracle database containing various kinds of information I need to be able to use in segmentations; would I design Data Extensions based around the data sources, or in some other way? Would I use multiple Data Extensions if I need to get large amounts of data from a single source?

Without looking at the data itself, I wouldn't be able to describe how you would do this in complete detail. However, data extensions would allow you the most flexibility in the terms of importing your information and also allow multiple ways to segment your data. I would design the data extensions around the data that you are importing. Depending on your use case for your data, you may want to use multiple data extensions.

3) How does the import process into the platform work? Would I normally use the "Import Activity" from an "Enhanced FTP directory" in ExactTarget? Or would I invoke the REST API to update the records in batches?

You could do either, but REST API may be slightly burdensome in the terms of the amount of calls you may need to make to the system. I would highly recommend that the files go to the FTP and then load into the system via an Import Activity. If you have the Automation Studio, you can create a triggered process for the files when they're dropped into the FTP. You may want to be careful about the size of the files, however - anything over 80 mb can experience issues. Feel free to check out the Import Activity documentation on the ET help website for more info: https://help.exacttarget.com/en-US/documentation/exacttarget/interactions/activities/import_activity/

  • 1
    For large files you can also compress them first, then upload them zipped to ftp, run a file transfer task to unzip, then import the unzipped text file using the import activity. Then can be handled simply by creating a program and run the first then second task. – james Sep 26 '14 at 0:19
  • Note to james: I selected this answer as the most comprehensive one; if I had had a better "reputation", I would also have upvoted your answers and comments, as they also were very helpful. Thanks! – donmartin Sep 26 '14 at 6:15
1

In terms of adding that amount of data to ET data extensions and FTP/CSV's are the only way to really go.

What you could do is when you send up the file you can call the api remotely to start a program that then has the import task in it (and any other steps you may need).

Are you using C#, PHP, Java? Examples could probably be supplied by other readers if they know what type of backend you are using.

  • Now you've got me really confused. Backend? I was planning to use an ETL tool (like IBM DataStage or similar) to wrap up the data from the sources as I need and then "upload" them to ET. As to what I understand here now, I would normally create CSVs and FTP them to ExactTarget. Where am I sidetracked? What would I normally do in the "backend", and where does that run? To me, "backend" means my backend systems, like ERP and such, and that's SAP. That's not what you mean, right? – donmartin Sep 25 '14 at 17:52
  • I wasn't sure what you were using to load the data. If the ETL tool is what you are using then yes you would send it up using an FTP connection and then have the program that imports run at a normal interval. So for instance if you upload the CSV's to ET at 4pm and 11pm you could schedule programs to run every day at say an hour beyond that 5pm and midnight or so. By backend I just meant if you were not using an ETL tool but doing a custom process by code you could upload the file and then call the program start through code upon completion of upload. – james Sep 25 '14 at 18:05
  • But either way it is the same process. Upload through FTP and import to a data extension. That is the most efficient for large amounts of data. – james Sep 25 '14 at 18:05
  • Thank you for your clarifications! Now I start to get a grip on how it works. – donmartin Sep 26 '14 at 6:14
0

for the initial big chunk of 10M records you are forced to use FTP, however for the modification volume 5-10K you can use API calls loading your tables/data extensions.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.