5

We have an object that may have millions of records (for simplicity, we can assume <50m). We need to process those millions of records to do some simple calculation on one of the fields.

One obvious way to do this is to use aggregate queries. However, we cannot do that because the fields that we would want to aggregate on may have more than 50k records in one aggregated result, which I believe will break the query. Please let me know if there is a way around it.

On the other hand, processing the records by batches of 2k will take 1,000,000/2 = 5k batches, which will take a long time, even if it is possible.

So the question is: What is the best practice to optimize our start/execute methods to handle basic processing for millions of records in batch while minimizing the number of batches and runtime when the processing is minimal (e.g., summing MyObject.Amount__c on the records)?

|improve this question
  • 4
    You should look at HyperBatch. – Adrian Larson♦ Mar 20 '18 at 19:18
  • Or, look at DLRS, which is designed to handle these sorts of tasks. – sfdcfox Mar 20 '18 at 20:12
  • @sfdcfox It may not be able to handle this kind of data volume though. – Adrian Larson♦ Mar 20 '18 at 20:24
  • @AdrianLarson, is there a way to bypass the 50k limit? I know with "at" readonly we can increase the limit to 1m but not sure if that's possible with Batch – Jorjani Mar 21 '18 at 13:29
  • 1
    No there is no equivalent to @readonly. You might want to just try writing the batch. That framework was built to handle serious volume. Converting to HyperBatch if you decide you need to go that route is quite fast from what I understand. – Adrian Larson♦ Mar 21 '18 at 13:32
5

Eh I guess this tool might stand on its own as an answer, since you've basically got the exact use case. If you're looking for faster processing, Daniel Peter (Salesforce MVP) built out a tool called HyperBatch which can massively speed up execution time when you're dealing with hundreds of thousands or millions of records.

You can see a slide deck here, where he gives a little more detail about what the tool is and why to use it. For example, he takes a job which creates Contact records from 45 minutes to just over 2. For the update job in his example, processing time goes from 10 minutes down to 38 seconds. And the delete job goes from 33 minutes (had to run twice) down to a little over a minute.

Down-side is you can't schedule it, and you have to keep the UI open the whole time you're running the job. Upshot is your jobs will run ten to twenty times faster.

|improve this answer
  • very interesting on how this tool has been built . very creative! – RedDevil Mar 21 '18 at 8:32
  • Thanks @adrian. This is a very interesting tool. However, I do need to be able to schedule the job and will need to rerun it as a scheduled job multiple times during the day. – Jorjani Mar 21 '18 at 13:19
  • 1
    I certainly take no credit for its conception or realization. @Jorjani you could probably piece something together with Selenium if you can dedicate a machine to the cause. Or maybe get something running on AWS or Heroku. You do need an actual browser to do the heavy lifting. – Adrian Larson♦ Mar 21 '18 at 13:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.