We have an object that may have millions of records (for simplicity, we can assume <50m). We need to process those millions of records to do some simple calculation on one of the fields.
One obvious way to do this is to use aggregate queries. However, we cannot do that because the fields that we would want to aggregate on may have more than 50k records in one aggregated result, which I believe will break the query. Please let me know if there is a way around it.
On the other hand, processing the records by batches of 2k will take 1,000,000/2 = 5k batches, which will take a long time, even if it is possible.
So the question is: What is the best practice to optimize our start/execute methods to handle basic processing for millions of records in batch while minimizing the number of batches and runtime when the processing is minimal (e.g., summing MyObject.Amount__c on the records)?