0

I need to fetch DataExtensions page-by-page but I didn't find anything useful for SOAP in SFMC

The only I found is how to limit batch size for the response

Can I load the next batch, or is there another way to walk through pages?

2
2

When you make a RetrieveRequest where the response indicates there are further pages of results that may be retrieved, you'll receive an OverallStatus of "MoreDataAvailable" similar to this response:

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/08/addressing" xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">
    <soap:Header>
        <wsa:Action>RetrieveResponse</wsa:Action>
        <wsa:MessageID>urn:uuid:153f237f-aee1-4681-981d-73574e5d49ef</wsa:MessageID>
        <wsa:RelatesTo>urn:uuid:9f08a3e1-507a-4bb0-a955-66687ed0c2f6</wsa:RelatesTo>
        <wsa:To>http://schemas.xmlsoap.org/ws/2004/08/addressing/role/anonymous</wsa:To>
        <wsse:Security>
            <wsu:Timestamp wsu:Id="Timestamp-a2156239-7103-4b72-bc7a-004c5d61224c">
                <wsu:Created>2020-09-30T01:21:11Z</wsu:Created>
                <wsu:Expires>2020-09-30T01:26:11Z</wsu:Expires>
            </wsu:Timestamp>
        </wsse:Security>
    </soap:Header>
    <soap:Body>
        <RetrieveResponseMsg xmlns="http://exacttarget.com/wsdl/partnerAPI">
            <OverallStatus>MoreDataAvailable</OverallStatus>
            <RequestID>b35a269a-eaf6-4d8c-9697-dae288d88531</RequestID>
            <Results xsi:type="DataExtensionObject">
            ...
            </Results>
        </RetrieveResponseMsg>
    </soap:Body>
</soap:Envelope>

To get the subsequent page, provide the RequestID from the previous response in the ContinueRequest property of the following RetrieveRequest:

<Envelope xmlns="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <Header>
        <h:fueloauth xmlns="http://exacttarget.com" xmlns:h="http://exacttarget.com">{{_token}}</h:fueloauth>
    </Header>
    <Body>
        <RetrieveRequestMsg xmlns="http://exacttarget.com/wsdl/partnerAPI">
            <RetrieveRequest>
                <ContinueRequest>b35a269a-eaf6-4d8c-9697-dae288d88531</ContinueRequest>
                <ObjectType>DataExtensionObject[Contacts]</ObjectType>
                <Properties>uuid</Properties>
            </RetrieveRequest>
        </RetrieveRequestMsg>
    </Body>
</Envelope>

Sample PHP and DotNet code is provided in online documentation in Retrieve More than 2500 Records Using the ContinueRequest Property.

Before going down the route of attempting to extract large amounts of data from Marketing Cloud Data Extensions, consider carefully using Automation Studio and Data Extension Extracts. This approach is far more scalable than using APIs for bulk extraction of data. Also keep in mind that, although not actively metered by Salesforce, you'll have a quota of API calls at your disposal that is finite and could become the object of Salesforce's monetization efforts in the future.

0

Pagination with the custom page size is possible by combining ContinueRequest and BatchSize properties

In example I paginate over DataExtensionObjects which have a useful auto-incremented identifier _CustomObjectKey

In the first request, we specify list of Properties and BatchSize which limits the number or entries in the response:

<RetrieveRequest>
    <ObjectType>DataExtensionObject[SampleDataExtensionKey]</ObjectType>
    <Properties>_CustomObjectKey</Properties>
    <Properties>ColumnA</Properties>
    <Options>
        <BatchSize>10</BatchSize>
    </Options>
</RetrieveRequest>

In the response, we get the list of entries with indexes from 1 to 10

<RetrieveResponseMsg xmlns="http://exacttarget.com/wsdl/partnerAPI">
    <OverallStatus>MoreDataAvailable</OverallStatus>
    <RequestID>2f5db341-212a-47dd-a2f5-9ec41f2e8c5f</RequestID>
    <Results xsi:type="DataExtensionObject">
        <PartnerKey xsi:nil="true"/>
        <ObjectID xsi:nil="true"/>
        <Type>DataExtensionObject</Type>
        <Properties>
            <Property>
                <Name>_CustomObjectKey</Name>
                <Value>1</Value>
            </Property>
        </Properties>
    </Results>
    ...
    <Results xsi:type="DataExtensionObject">
        <PartnerKey xsi:nil="true"/>
        <ObjectID xsi:nil="true"/>
        <Type>DataExtensionObject</Type>
        <Properties>
            <Property>
                <Name>_CustomObjectKey</Name>
                <Value>10</Value>
            </Property>
        </Properties>
    </Results>
</RetrieveResponseMsg>

When MoreDataAvailable is specified in the response we can take a RequestID which now refers to the stored request with Properties

Now we can request the next page with only specifying RequestID and BatchSize

ObjectType and Properties no more needed for further pagination. Actually, they are just ignored - you can not change properties between pagination steps

We can use different BatchSize on each iteration to increase or decrease the limit dynamically - it works consistently

<RetrieveRequest>
    <ContinueRequest>2f5db341-212a-47dd-a2f5-9ec41f2e8c5f</ContinueRequest>
    <Options>
        <BatchSize>12</BatchSize>
    </Options>
</RetrieveRequest>

Response:

<RetrieveResponseMsg xmlns="http://exacttarget.com/wsdl/partnerAPI">
    <OverallStatus>MoreDataAvailable</OverallStatus>
    <RequestID>2f5db341-212a-47dd-a2f5-9ec41f2e8c5f</RequestID>
    <Results xsi:type="DataExtensionObject">
        <PartnerKey xsi:nil="true"/>
        <ObjectID xsi:nil="true"/>
        <Type>DataExtensionObject</Type>
        <Properties>
            <Property>
                <Name>_CustomObjectKey</Name>
                <Value>11</Value>
            </Property>
        </Properties>
    </Results>
    ...
    <Results xsi:type="DataExtensionObject">
        <PartnerKey xsi:nil="true"/>
        <ObjectID xsi:nil="true"/>
        <Type>DataExtensionObject</Type>
        <Properties>
            <Property>
                <Name>_CustomObjectKey</Name>
                <Value>22</Value>
            </Property>
        </Properties>
    </Results>
</RetrieveResponseMsg>

The RequestID returned for each page is always the same, it looks like it is stored statically and is not regenerated for every new response. There is nothing in the documentation stating that it is always the same for the consequent requests, so I would recommend to always pick a returned one.

This also means that ContinueRequests are not idempotent - every request changes the internal state of the RequestID - it holds (at least) the number of already returned objects, so you can not re-request the same page one more time.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.