Can we avoid 'Attempted to schedule too many concurrent batch jobs in this org' error using Apex flex queue?

Question

We have a batch class AccountDeactivationBatch from whose finish() method we run 3 more batches, including AccountDeactivationBatch again, but with new set of Ids. These new set of Ids are all children account IDs of all records that were processed in this batch. This goes on till no eligible child records are fetched from database.querylocator().

Following is sample code:

public class AccountDeactivationBatch implements Database.Batchable<sObject>, Database.Stateful {
    public Set<Id> accountIdSet = new Set<Id>();

    public Set<Id> newAccountIdSet = new Set<Id>(); //to maintain next level children acc id.


    public AccountDeactivationBatch(Set<Id> accountIdSet){
        this.accountIdSet = accountIdSet;
    }

    public Database.QueryLocator start(Database.BatchableContext bc) {
        String query = 'select Active__c, from Account ' +
        'where Send_To_ABC__c = true ' +
        'and Active__c = true and ParentId in : accountIdSet';
        return Database.getQueryLocator(query);
    }

    public void execute(Database.BatchableContext bc, List<Account> records)

    {
      public List<Account> accountList = new List<Account>();
      newAccountIdSet.clear();
      for(Account acc : records){
        acc.Active__c = false;
        accountList.add(acc);
        newAccountIdSet.add(acc.id);    //Maintaining children account ids 
                                        that were fetched from query. To be used in query of next batch.
       }
       update records;
    } 

    public void finish(Database.BatchableContext bc){
        //Run Contact batch. 
        ContactDeactivationBatch contactDeactivationBatch = new ContactDeactivationBatch(accountIdSet); 
        Id contactBatchId = Database.executeBatch(contactDeactivationBatch);

        //Run User batch.
        UserDeactivationBatch userDeactivationBatch = new UserDeactivationBatch(accountIdSet); 
        Id userBatchId = Database.executeBatch(userDeactivationBatch);


        //Calling this same batch again, but with next level child account IDs
        if(newAccountIdSet.size() > 0){         //Need this check to avoid recrusion.
            AccountDeactivationBatch accountDeactivationBatch = new AccountDeactivationBatch(newAccountIdSet); 
            Id accountBatchId = Database.executeBatch(accountDeactivationBatch); 
        }
    }   
}

Following are the queries in this case: 1. We may run into 'Attempted to schedule too many concurrent batch jobs in this org' error. Can this error be avoided using Apex flex queues. 2. Is the query for AsyncApexJob table required to fetch Status='Queued' jobs or flex queues will resolve this? 3. If we write SOQL on AsyncApexJob, is there a chance to hit SOQL exception? 4. What are other possible limits that could be exceeded in this case?

Appreciate any help here. Thanks in advance.

I had referred this help article regarding this issue.

Answer 1

Your design very likely to hit the above mentioned exception where you have more accounts to process.

Flex queue might be able to help you but to only some extent.(100 jobs holding.) If the Apex flex queue has the maximum number of 100 jobs, Database.executeBatch throws a LimitException and doesn’t add the job to the queue.Submit More Batch Jobs with Apex Flex Queue (Generally Available)

As far as checking the AsyncApexJob query not sure about any exceptions at this point but you should be able to query job status whether to go forward with calling another execute here.

One other workaround for this I can think of is changing the design itself.

Now according to your code your'e processing 3 things,

1. Account
2. Contact
3. User

You can think of a chain that has a 3 links(each of the above three is a link) and your chain should be called again and again until your'e out of accountIds to process. By doing this your'e technically keeping number of active batches to 1. The idea here is to run them in series rather than in parallel.

You need to maintain an order then your Account batch will look like below

public class AccountDeactivationBatch implements Database.Batchable<sObject>, Database.Stateful {
    public Set<Id> accountIdSet = new Set<Id>();

    public Set<Id> newAccountIdSet = new Set<Id>(); //to maintain next level children acc id.

    public AccountDeactivationBatch(Set<Id> accountIdSet){
        this.accountIdSet = accountIdSet;
        currentIdToProcess ='';
    }

    public Database.QueryLocator start(Database.BatchableContext bc) {

        String query = 'select Active__c, from Account ' +
        'where Send_To_ABC__c = true ' +
        'and Active__c = true and ParentId in : accountIdSet';
        return Database.getQueryLocator(query);
    }

    public void execute(Database.BatchableContext bc, List<Account> records)

    {
      public List<Account> accountList = new List<Account>();
      newAccountIdSet.clear();
      for(Account acc : records){
        acc.Active__c = false;
        accountList.add(acc);
        newAccountIdSet.add(acc.id);    //Maintaining children account ids 
                                        that were fetched from query. To be used in query of next batch.
       }
       update records;
    } 

    public void finish(Database.BatchableContext bc){

    if(accountIdSet.size()>0){
                //Run Contact batch. 
        ContactDeactivationBatch contactDeactivationBatch = new ContactDeactivationBatch(accountIdSet, newAccountIdSet);// your contact batch should accept the child Ids to process
        Id contactBatchId = Database.executeBatch(contactDeactivationBatch);
    } 
    else{// call this only if the batch wont go through the chain
            AccountDeactivationBatch accountDeactivationBatch = new AccountDeactivationBatch(newAccountIdSet); 
            Id accountBatchId = Database.executeBatch(accountDeactivationBatch); 
    }

    }

}

Now your Contact batch looks like below accepting two parameters one to process currently one to pass on to next batch

public class ContactDeactivationBatch implements Database.Batchable<sObject>, Database.Stateful {
    public Set<Id> accountIdSet = new Set<Id>();

    public Set<Id> newAccountIdSet = new Set<Id>(); //to maintain next level children untill next account batch.


    public ContactDeactivationBatch(Set<Id> accountIdSet,Set<Id> newAccIdSet){
        this.accountIdSet = accountIdSet;
        this.newAccountIdSet = newAccIdSet;
    }

    public Database.QueryLocator start(Database.BatchableContext bc) {
        String query = 'querry something'
        return Database.getQueryLocator(query);
    }

    public void execute(Database.BatchableContext bc, List<Account> records)
//do something
    } 

    public void finish(Database.BatchableContext bc){

        //Run User batch.
        UserDeactivationBatch userDeactivationBatch = new UserDeactivationBatch(accountIdSet,newAccountIdSet); // your user batch should again look for the child records to carry over 
        Id userBatchId = Database.executeBatch(userDeactivationBatch);
    }   
}

Now you user batch will be similar to contact batch looks for extra parameter but here if there are child records then you will call the chain to restart again.

public class UserDeactivationBatch implements Database.Batchable<sObject>, Database.Stateful {
    public Set<Id> accountIdSet = new Set<Id>();

    public Set<Id> newAccountIdSet = new Set<Id>(); //to maintain next level children untill next account batch.


    public UserDeactivationBatch(Set<Id> accountIdSet,Set<Id> newAccIdSet){
        this.accountIdSet = accountIdSet;
        this.newAccountIdSet = newAccIdSet;
    }

    public Database.QueryLocator start(Database.BatchableContext bc) {
        String query = 'querry something'
        return Database.getQueryLocator(query);
    }

    public void execute(Database.BatchableContext bc, List<Account> records)
//do something
    } 

    public void finish(Database.BatchableContext bc){

       //Calling this chain again,with next level child account IDs
       // ALSO EMPTY accountIdSet HERE AT THIS POINT 
        if(newAccountIdSet.size() > 0){         //Need this check to avoid recrusion.
            AccountDeactivationBatch accountDeactivationBatch = new AccountDeactivationBatch(newAccountIdSet); 
            Id accountBatchId = Database.executeBatch(accountDeactivationBatch); 
        }
    }   
}

One addition to this would be doing some error handling at the end of each batch class to send you emails etc.

Note: As I haven't seen what's going on contact and user batch I only gave an idea for design implementation you can tweak this approach to fit your needs.