I have had this challenge (mass updates from marketing cloud) before and could solve it with the SFSC Bulk API.
Journey Builder does not leverage the bulk API, which means JB is considerably slower than the speeds you could achieve with Bulk API.
I tested the JB throughput once with Campaign Members, and it was somewhere around 200 - 400 records per minute. To my knowledge there is nothing you can do within JB to speed this up. Generally, JB is NOT a tool intended for Bulk usage, so I would anyway advise against the approach.
The Bulk API on the other hand is meant for bulk jobs like "update 1 million records". You can use the Bulk API with custom scripts from Automation Studio.
The speed I reached with this approach was 100k Campaign Member Inserts in 18 Minutes.
It is a bit of an intricate process - as you need to have 4 automations that call each other given a certain condition.
https://sprignaturemoves.com/retrieving-and-starting-an-automation-with-ssjs/
basically:
Automation 1 opens a Bulk API job and then checks how many records are left to be sent to Sales Cloud:
IF number of records is > 0:
Call Automation two. Else: Call Automation 4.
Automation 2 sends a batch of up to 10k via Bulk API, then marks the records as "sent".
If there are still records not marked as "sent" after that:
Automation 2 calls Automation 3. Else: call automation 4.
Automation 3 is a copy of 2.
Only difference - if there are still records not marked as "sent" - it calls Automation 2 again. Else: call automation 4.
Automation 4 closes the bulk API job.
This may look fishy, but it is just another way of assembling a continuous loop with automation Studio that breaks when no more records are available, and has proved very stable for me.
In Automation one:
As a precursor, you will get API responses in XML format, which you cannot natively parse,
but this has been solved here: Parse XML in Server Side Javascript
/* parse XML */
function trim(x) {
return x.replace(/^[\s\uFEFF\xA0]+|[\s\uFEFF\xA0]+$/g, '');
};
function dataFromAttr(data, attr, notNull){
if (data){
var reg = new RegExp("<"+attr+">", "g");
var result = data.match(reg);
if(result.length == 0){
var str = '';
return trim(str);
}
if(result.length == 1){
var str = data.split("<"+attr+">")[1];
str = str.split("</"+attr+">")[0];
return trim(str);
}
if(result.length > 1){
var str = data.split("</"+attr+"> <"+attr+">");
return str;
}
var nullElementValue = new RegExp("<"+attr+" />");
if(data.match(nullElementValue).length) return "";
if(notNull) return "";
}
else{
return null;
}
};
var sessionId = dataFromAttr(authresponse, "sessionId");
var serverUrl = dataFromAttr(authresponse, "serverUrl");
authenticate against Sales Cloud from an SSJS script, then, open a Bulk API Job in which you also specify your object and intended operation.
var create_service = "services/async/43.0/job";
var create_url = base_url.concat(create_service);
var contentType = 'application/xml; charset=UTF-8;';
var headerNames = ["X-SFDC-Session"];
var headerValues = [sessionId];
var payload = '<?xml version="1.0" encoding="UTF-8"?>';
payload += '<jobInfo xmlns="http://www.force.com/2009/06/asyncapi/dataload">';
/* DELETE Job - permission required: https://help.salesforce.com/articleView?id=000171306&type=1
payload += ' <operation>delete</operation>';
payload += ' <object>Contact</object>';
*/
/* INSERT CAMPAIGN MEMBER Job */
/*
payload += ' <operation>insert</operation>';
payload += ' <object>CampaignMember</object>';
*/
/* Update CAMPAIGN MEMBER Job */
payload += ' <operation>update</operation>';
payload += ' <object>CampaignMember</object>';
payload += ' <contentType>JSON</contentType>';
payload += '</jobInfo>'
result = HTTP.Post(create_url,contentType,payload,headerNames,headerValues);
var statusCode = result["StatusCode"];
var createresponse = result["Response"][0];
createresponse = Stringify(createresponse).replace(/[\n\r]/g, '');
var jobId = dataFromAttr(createresponse, "id");
This returns a job Id. Write it into a data extension, you'll need it for all other automations.
You are ready to send batches.
Now you would be going forward to call Automation Two.
Automation 2 and 3:
Within the 24 hour timeframe of the Job being "open", you would have to loop through retrieving records in DE, in batches of up to 10000 records - which is the limit for batchsize in Sales Cloud.
Note: This is beyond the limit of a Marketing Cloud Data Extension retrieve - which does 2500 per run, but this you can workaround with a do /while loop x4.
I have tested this to be stable without timing out.
The payload you are thus assembling will be a long list of up to 10k, in this example, campaignMembers, being updated to e.g. status "sent":
[
{
"Id": 701xxxxxxxxxxxxxxxx,
"Status": "Sent"
},
{
"Id": 701xxxxxxxxxxxxxxxy,
"Status": "Sent"
}
]
This payload is one batch.
For each of these 10000 record batches, you send one API call until no more records are left.
like so:
var sendbatch_service = "services/async/43.0/job/";
var sendbatch_url = base_url.concat(sendbatch_service,jobId,'/batch');
var contentType = 'application/json';
var headerNames = ["X-SFDC-Session"];
var headerValues = [sessionId];
result = HTTP.Post(sendbatch_url,contentType,payload,headerNames,headerValues);
var statusCode = result["StatusCode"];
Note:
For Campaign members, the Job processing had to be serial and could not run in parallel due to Record Lock Contention. This could be different for other objects.
Mark the records you sent in your source DE.
Repeat until there are no more unmarked records in that DE.
Then you can call Automation 4:
send another call to close the Bulk API job.
Closing the bulk api Job by JobId:
/* Close Job */
var close_service = "services/async/43.0/job/"
var close_url = base_url.concat(close_service,jobId);
var contentType = 'application/xml; charset=UTF-8;';
var headerNames = ["X-SFDC-Session"];
var headerValues = [sessionId];
var payload = '<?xml version="1.0" encoding="UTF-8"?>';
payload += '<jobInfo xmlns="http://www.force.com/2009/06/asyncapi/dataload">';
payload += ' <state>Closed</state>';
payload += '</jobInfo>'
result = HTTP.Post(close_url,contentType,payload,headerNames,headerValues);
var statusCode = result["StatusCode"];
var closeresponse = result["Response"][0];
closeresponse = Stringify(closeresponse).replace(/[\n\r]/g, '');
var jobId = dataFromAttr(closeresponse, "id");
Monitoring of your Bulk Job is native.
https://developer.salesforce.com/docs/atlas.en-us.api_asynch.meta/api_asynch/asynch_api_batches_monitor.htm
So, this does consume API calls outside of the Connector based Connected App Authentication, these are not of a threateningly high number. For one million updates it would be 100 calls + auth + open + close = 103 API calls.
