How to estimate how long a given batch will run?

Question

Are there any reliable numbers, heuristics, experience reports to estimate (and display it to the triggering user) how long a given batch will run?

Sure the time will mainly depend on the number of processed records. But...

1. Will the time per record vary depending on the Server and its load
2. What is the main influence on the time a Batch is preparing (in start())
3. Do single batch jobs take the same time? So if 50% of all jobs are done will the other 50% take the same amount of time?

EDIT: My idea was to measure how long the batch time and store time/no of records in a custom setting. Every batch call would improve the average stored in that field. I would hope this somewhat would converge...

Answer 1

1. It primarily depends on server load. You'll typically notice a one to three second delay between each execute call, so using larger batch sizes usually results in faster overall completion time. The actual time for each call depends on what the execute function does, but assuming each call performs the same amount of work, each execution should take approximately the same amount of time. The start function may fluctuate not only on server load, but also the server's database cache. Recently loaded records in the cache may cause a shorter startup time.

2. The same as any other function. The complexity of the query is usually the primary factor, but if you're using an iterator, then the time will be based on the complexity of the iterator. The time is roughly linear, or O(n), meaning the start function's execution time should increase approximately linearly with the number of iterations or records involved.

3. From simple observations, this is generally true, although due to the interlaced behavior of the queue, the time delay between each batch may fluctuate, although I've rarely noticed any observable difference from the time start finishes executing and the first execute call and the difference in time between each execute call.

In summary, the time depends mostly on server load, but should remain fairly constant throughout the life of the transaction.

Answer 2

This fairly old Salesforce document Asynchronous Processing in Force.com describes concepts like "extended delay" that are part of the queuing process and so impact execution time.

Answer 3

I think you could approach this along the lines of the way many software install or render programs run; particularly since you have no control over Server Load.

Your initial instincts to keep historical data to provide an initial estimate seems like a reasonable approach to me as a first display. However, until the first group of records is run, you don't have any kind of context to know what the current server load is. Once the first group of records runs, then you can recalculate and give a more accurate estimate of the time for the job to run.

You may want to collect data that includes time for first group of records to complete for a given type of batch job and total time/total number of records (again for a given type of batch job). I think using this methodology would give you the ability to provide some heuristics that account for server load. As an experiment, you may want to collect data to see if this changes or becomes a more accurate predictor if you also record time for first 3 groups of records as well. Its all a matter of how complex you want to make your algorithm.

Whether or not the complexity of the query or the work the batch job does is significant, I can't say and that's why I've said for a given type of batch job. That would allow you to group certain jobs you run that you feel are very similar into the same data groups should you choose to do so. If you initially keep all the data separate, you can later combine some of it should you find it similar in nature and "close enough" for your needs.