Batch Job Governor Limits

Question

We have two custom objects - obj1 and obj2. These are related by a custom Id. We need to iterate through obj1, get its relevant records from obj2 based on Id and further query for permission of each object in the user permission table and make external call outs for them. Obj1 has around 5 M records and each record can have around 10K records in obj2. I will be executing this through a batch job. Is there a possibility of hitting any governor limits in this scenario? Following is pseudo-code for execute method of the batch job.

  global Database.Querylocator start (Database.BatchableContext BC) {
        return Database.getQueryLocator('select id from obj1'); 
    }

global void execute (object1Scope) {
   For every object in object1Scope
      Fetch all relevant objects in object2
         For each object retrieved from object2 
                  fetch permissions from user permission table
      Multiple external callouts
 update 
}

Answer 1

Yes, you have a high likelihood - more like a certainty, really - of hitting a variety of limits in this scenario.

We need to iterate through obj1, get its relevant records from obj2 based on Id and further query for permission of each object in the user permission table and make [multiple] external call outs for them. Obj1 has around 5 M records and each record can have around 10K records in obj2

This certainly implicates

• The callout limit, which is 100 per transaction.
• The DML rows limit (10,000), if you make any updates on Obj2 and process more than one Obj1's related records per execute().
• The SOQL rows limit (50,000), if you process more than five Obj1's related records per execute().
• Possibly the SOQL queries limit (200 in batch) depending on how you structure your execute().
• The heap size limit.
• The CPU time limit.

Solutions involving spawning other asynchronous processes from your batch, in execute() or in finish(), risk a variety of other limits depending on the architecture, including the Apex flex queue limit, heap size, Asynchronous Apex invocations, and so on.

This process will require very careful architecting and is definitely not solvable in the scope of an SFSE answer. Consider consulting with an experienced Salesforce architect, and/or moving this process off-platform to a middleware solution.

If you're in a situation where you have fifty billion records (is that actually correct?) in Obj2, you need to be working with LDV (large data volume) and architecture experts.