Executing Batch Class In For Loop - Flex Queue Limit?

Question

We have a requirement of operating on data in present in different objects. The total record count (of all objects) ~ 50 millions (some objects holding ~20 millions records alone). So we have considered the option of handling it via batch apex.

The logic is:

• Create a List (say objectsList) of String which will hold the object names.
• A method will start batch execution from For loop. Batch apex will identify the object by the integer passed and start operating on that object.

Something as follows:

for(integer i = 0; i< objectList.size(); i++){
RandomBatch bch = new RandomBatch(i);
Database.executeBatch(bch,2000);
}

Is this feasible and practical? From what I read, flex queue has capacity of holding 100 jobs in queued state. We don't have more than 20 objects to cover (so 20 batches will be executed). Can flex queue hold jobs for over 2 days (since objects holding ~20 millions records are going to take time to complete). And is it even acceptable to execute batch apex from For loop? There's nothing going to be in finish() method (apart from mailing us the error messages if any).

Answer 1

Personally, I have not seen Batches to be queued that long, even if they are it doesn't seem SF will kill them after some time if they are in limit.

That being said, You can follow another approach. We know at max 5 batch jobs can run at a time. Let's utilize this and start the next batch at the end of each batch.

RandomBatch bch = new RandomBatch(0);
Database.executeBatch(bch,2000);
bch = new RandomBatch(1);
Database.executeBatch(bch,2000);
bch = new RandomBatch(2);
Database.executeBatch(bch,2000);
bch = new RandomBatch(3);
Database.executeBatch(bch,2000);
bch = new RandomBatch(4);
Database.executeBatch(bch,2000);

And in the finishMethod of each batch, you will add 5 to your Index and run next batch till the index reaches the max value of 20.

public void finish(BatchableContext bc){
   Integer nextBatchIndex = currentIndex + 5;
   if(nextBatch < 20){
      RandomBatch bch = new RandomBatch(nextBatchIndex );
      Database.executeBatch(bch,2000);
   }
}

Also you need to ensure that your batch won't exceed the Async Apex Limit , because if that happens you wont be able to run any future queuable schduler apex till the limit resets.