1

We need to update a field in around 8 million records of an object and we cannot process them using Apex Batch because it will be taking around a day or two for completion of this task.

So is there any alternative to update large number of records without consuming a lot of time?

5
  • Have you considered using the Bulk API via Data Loader, after manipulating the data outside of Salesforce?
    – David Reed
    May 3, 2018 at 12:47
  • 2
    github.com/danieljpeter/HyperBatch
    – kurunve
    May 3, 2018 at 12:59
  • For 8 million, I would go with Bulk Api V2. I dont wanna ruin my Daily Async jobs.. with batches. May 3, 2018 at 13:08
  • @DavidReed @ Pranay. Thanks for your response. The field which we are updating in those 8 million records are to be updated with reference to other field in another object; and hence they cannot be set by us in a csv. For ex. ObjectA.field1__c will hold corresponding ObjectB.field1__c value. How to proceed with bulk api in this situation? May 3, 2018 at 14:11
  • Possible duplicate
    – Adrian Larson
    May 3, 2018 at 14:36

3 Answers 3

2

If you're looking for faster processing, Daniel Peter (Salesforce MVP) built out a tool called HyperBatch which can massively speed up execution time when you're dealing with millions of records.

You can see a slide deck here, where he gives a little more detail about what the tool is and why to use it. For example, he takes a job which creates Contact records from 45 minutes to just over 2. For the update job in his example, processing time goes from 10 minutes down to 38 seconds. And the delete job goes from 33 minutes (had to run twice) down to a little over a minute.

Down-side is you can't schedule it, and you have to keep the UI open the whole time you're running the job. Upshot is your jobs will run ten to twenty times faster.

4
  • There is a huge risk with using hyper batch. Each organization is allowed 10 concurrent long-running requests. If the limit is reached, any new synchronous Apex request results in a runtime exception. developer.salesforce.com/blogs/engineering/2013/05/… once we get concurrent-limit exceeded exception normal platform users/community users will see a Blue Screen Of death. I try to avoid it at all cost. May 3, 2018 at 14:39
  • @PranayJaiswal Just one reason among many a data load of this scale should be performed during an outage, with no other users around logging into the system.
    – Adrian Larson
    May 3, 2018 at 14:41
  • Yeah you are right.. I personally felt hyper batch was a hacky way of getting this done plus there was this bug which kinda discouraged me from using hyper batch. success.salesforce.com/issues_view?id=a1p300000008ZE5AAM . May 3, 2018 at 14:44
  • Fair enough. I haven't used it but if someone is worried about processing time it seems like a reasonable avenue to explore.
    – Adrian Larson
    May 3, 2018 at 14:45
0

Data loader will be easiest solution. How long such an update will take is slightly dependant on state of Salesforce servers, internet connection and settings but it will probably take around 7 hours or less

0

It sounds like this is a one-time data update. If that is not the case, you may want to spend more time on this and consider an enterprise ETL solution and/or changes to your data model.

For a one-off update, I would use the Data Loader over Bulk API to pull down the entire tables for ObjectA and ObjectB.

Because 8 million rows is too much for either Excel or LibreOffice to handle, I would write a simple Python script using the csv module to populate the required field on ObjectA from ObjectB and generate a new CSV file with the appropriate field populated. You could achieve the same workflow with your preferred scripting language.

If the volume of ObjectB is significantly less than ObjectA (< 1 million records), it would be feasible to use a programmer's editor to chunk the 8 million CSV records into smaller files and then populate the field in each file with an INDEX/MATCH construct using Excel or LibreOffice, avoiding the need to write any code at all.

I'd then uploaded that generated file back to Salesforce via Data Loader/Bulk API.

2
  • Talend Open Studio for Data Integration is a free Enterprise grade ETL tool which supports bulk API v2, sounds better than writing python scripts or using excel sheets. May 3, 2018 at 14:49
  • I need to try out Talend. I use Python for everything, so there's basically no overhead on that for me. Thanks for the pointer.
    – David Reed
    May 3, 2018 at 14:51

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .