1

I have the Marketing Cloud Connector working fine and I have a little issue to start to use the Journey Builder to update the records in CRM.

In first moment i want to udpate 1 million records to subscribers that became "hard bounce" and udpate the field "PersonHasOptedOutOfEmail" in CRM to this records not come back for Marketing Cloud. My Data Extension only has the ID and PersonHasOptedOutOfEmail ('true' for all).

How the Dataloader(by bulk api) is time consuming as it would take hours to complete this action. I wonder if there are batch limits for udpates in journey builder. The documentation miss this part.

  • 2
    Using JB to update 1 million records through the update activity would be very time consuming and could possibly slow down your other Journeys – EazyE Dec 3 '19 at 2:30
  • I am guessing, that JB would take way longer than DataLoader. Nevertheless I am not aware of any limits there. – DonL Dec 3 '19 at 7:37
2

I have had this challenge (mass updates from marketing cloud) before and could solve it with the SFSC Bulk API. Journey Builder does not leverage the bulk API, which means JB is considerably slower than the speeds you could achieve with Bulk API.

I tested the JB throughput once with Campaign Members, and it was somewhere around 200 - 400 records per minute. To my knowledge there is nothing you can do within JB to speed this up. Generally, JB is NOT a tool intended for Bulk usage, so I would anyway advise against the approach.

The Bulk API on the other hand is meant for bulk jobs like "update 1 million records". You can use the Bulk API with custom scripts from Automation Studio. The speed I reached with this approach was 100k Campaign Member Inserts in 18 Minutes.

It is a bit of an intricate process - as you need to have 4 automations that call each other given a certain condition. https://sprignaturemoves.com/retrieving-and-starting-an-automation-with-ssjs/

basically:

Automation 1 opens a Bulk API job and then checks how many records are left to be sent to Sales Cloud:

IF number of records is > 0: Call Automation two. Else: Call Automation 4.

Automation 2 sends a batch of up to 10k via Bulk API, then marks the records as "sent".

If there are still records not marked as "sent" after that: Automation 2 calls Automation 3. Else: call automation 4.

Automation 3 is a copy of 2. Only difference - if there are still records not marked as "sent" - it calls Automation 2 again. Else: call automation 4.

Automation 4 closes the bulk API job.

This may look fishy, but it is just another way of assembling a continuous loop with automation Studio that breaks when no more records are available, and has proved very stable for me.


In Automation one:

As a precursor, you will get API responses in XML format, which you cannot natively parse, but this has been solved here: Parse XML in Server Side Javascript

/* parse XML */
function trim(x) {
  return x.replace(/^[\s\uFEFF\xA0]+|[\s\uFEFF\xA0]+$/g, '');
};

function dataFromAttr(data, attr, notNull){
  if (data){
    var reg = new RegExp("<"+attr+">", "g");
    var result = data.match(reg);
    if(result.length == 0){
      var str = '';
      return trim(str);
    }
    if(result.length == 1){
      var str = data.split("<"+attr+">")[1];
      str = str.split("</"+attr+">")[0];
      return trim(str);
    }
    if(result.length > 1){
      var str = data.split("</"+attr+"> <"+attr+">");
      return str;
    }
    var nullElementValue = new RegExp("<"+attr+" />");
    if(data.match(nullElementValue).length) return "";
    if(notNull) return "";
  }
  else{
    return null;
  }
};

var sessionId = dataFromAttr(authresponse, "sessionId");
var serverUrl = dataFromAttr(authresponse, "serverUrl");

authenticate against Sales Cloud from an SSJS script, then, open a Bulk API Job in which you also specify your object and intended operation.

var create_service = "services/async/43.0/job";
var create_url = base_url.concat(create_service);
var contentType = 'application/xml; charset=UTF-8;';
var headerNames = ["X-SFDC-Session"];
var headerValues = [sessionId];
var  payload = '<?xml version="1.0" encoding="UTF-8"?>';
payload += '<jobInfo xmlns="http://www.force.com/2009/06/asyncapi/dataload">';

/* DELETE Job - permission required: https://help.salesforce.com/articleView?id=000171306&amp;type=1 

payload += '    <operation>delete</operation>';
payload += '    <object>Contact</object>';
 */
/*  INSERT CAMPAIGN MEMBER Job */
/*
payload += '    <operation>insert</operation>';
payload += '    <object>CampaignMember</object>';
*/

/*  Update CAMPAIGN MEMBER Job */

payload += '    <operation>update</operation>';
payload += '    <object>CampaignMember</object>';


payload += '    <contentType>JSON</contentType>';
payload += '</jobInfo>'

result = HTTP.Post(create_url,contentType,payload,headerNames,headerValues);
var statusCode = result["StatusCode"];
var createresponse = result["Response"][0];
createresponse = Stringify(createresponse).replace(/[\n\r]/g, '');

var jobId = dataFromAttr(createresponse, "id");

This returns a job Id. Write it into a data extension, you'll need it for all other automations.

You are ready to send batches. Now you would be going forward to call Automation Two.

Automation 2 and 3:

Within the 24 hour timeframe of the Job being "open", you would have to loop through retrieving records in DE, in batches of up to 10000 records - which is the limit for batchsize in Sales Cloud. Note: This is beyond the limit of a Marketing Cloud Data Extension retrieve - which does 2500 per run, but this you can workaround with a do /while loop x4. I have tested this to be stable without timing out.

The payload you are thus assembling will be a long list of up to 10k, in this example, campaignMembers, being updated to e.g. status "sent":

[
 {
    "Id": 701xxxxxxxxxxxxxxxx,
    "Status": "Sent" 
 },
{
    "Id": 701xxxxxxxxxxxxxxxy,
    "Status": "Sent" 
 }
]

This payload is one batch. For each of these 10000 record batches, you send one API call until no more records are left.

like so:

var sendbatch_service = "services/async/43.0/job/";
var sendbatch_url = base_url.concat(sendbatch_service,jobId,'/batch');

var contentType = 'application/json';
var headerNames = ["X-SFDC-Session"];
var headerValues = [sessionId];

result = HTTP.Post(sendbatch_url,contentType,payload,headerNames,headerValues);
var statusCode = result["StatusCode"];

Note: For Campaign members, the Job processing had to be serial and could not run in parallel due to Record Lock Contention. This could be different for other objects. Mark the records you sent in your source DE. Repeat until there are no more unmarked records in that DE.

Then you can call Automation 4: send another call to close the Bulk API job.

Closing the bulk api Job by JobId:

/* Close Job */

var close_service = "services/async/43.0/job/"
var close_url = base_url.concat(close_service,jobId);
var contentType = 'application/xml; charset=UTF-8;';
var headerNames = ["X-SFDC-Session"];
var headerValues = [sessionId];
var  payload = '<?xml version="1.0" encoding="UTF-8"?>';
payload += '<jobInfo xmlns="http://www.force.com/2009/06/asyncapi/dataload">';
payload += '    <state>Closed</state>';
payload += '</jobInfo>'

result = HTTP.Post(close_url,contentType,payload,headerNames,headerValues);
var statusCode = result["StatusCode"];
var closeresponse = result["Response"][0];
closeresponse = Stringify(closeresponse).replace(/[\n\r]/g, '');

var jobId = dataFromAttr(closeresponse, "id");

Monitoring of your Bulk Job is native. https://developer.salesforce.com/docs/atlas.en-us.api_asynch.meta/api_asynch/asynch_api_batches_monitor.htm

So, this does consume API calls outside of the Connector based Connected App Authentication, these are not of a threateningly high number. For one million updates it would be 100 calls + auth + open + close = 103 API calls.

| improve this answer | |
1

As EazyE pointed out, updating all your records with JB can be very time consuming and will certainly affect your other running Journeys.

To answer your question, the API calls coming from SFMC through the connector will not count towards the API Limits.

Having done several hundreds of thousands of updates using Salesforce Activities, I can tell you that once you have triggers or custom apex processing that are triggered after the insert/update of your records on the CRM, you will encounter very low processing.

Here are some best practices that can help you optimize your journeys:

When a journey has no entry filter, you can admit up to 2 million contacts per hour. When a journey has an entry filter, you can admit up to 250,000 contacts per hour. Following these general guidelines promotes efficient hourly processing.

Avoid using wait activities with a duration of less than one hour.

PS:I invite you to read more on journeys best practices on the link below.

Reference: Optimize Journey Builder Performance

| improve this answer | |
  • I am in the same situation. Your reference doesn't cover anything regarding Salesforce Activities. There should be some best practice out there regarding this. For example "Don't enter more than 100 000 contacts per hour if you are using Salesforce Activities". – user3274635 Dec 3 '19 at 10:46
  • Also, this statement is wrong "To answer your question, the API calls coming from SFMC through the connector will not count towards the APEX Governor Limits." It will count towards Apex Govern Limits. But not the API Limits. – user3274635 Dec 3 '19 at 10:48
  • Like it is written in my answer, the reference covers Journey Builder best practices. Regarding the Apex Govern Limits, you are right, corrected that on the answer (cons of trying to answer questions fast).. – Rachid Mamai Dec 3 '19 at 11:00
  • Yes, and if you follow these best practices you can essentially put in 2 million records an hour. With a Salesforce Activty this would encounter for 2 million API calls to Sales/Service Cloud. Which I guess isn't best practices. I have raised this question in Salesforce's own channels. But no feedback unfortunately. – user3274635 Dec 3 '19 at 11:56
  • Yeah, and not following these best practices would mean more problems and greater processing time. I think the doc should give some feedback regarding Salesforce activities use and limitations though.. Don't hesitate to post it in here if you get an answer from SF ;) – Rachid Mamai Dec 3 '19 at 12:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.