8

Just assume you had an app that would have to process huge amounts of records in an org in multiple ways each day. For each "way" you have a separate Batchable class and those are triggered independently by user actions. Depending on how many users and how much data you have you can easily run into one of the Batch related limits.

My ideas was to in any way centralize batch processing and by that reducing the risk of e.g. too many batches running in parallel.

I thought of having a ContinuousWorkerBatch class that checks in his start() method if work needs to be done (which type of work on which record). If so it passes the Ids of the records to process to his execute method otherwise skips the execute. In his finish method it would calls a new instance of itself. With Summer '13 we can use the new method to delay the new batch by scheduling it in a minute.

global class ContinuousWorkerBatch implements Database.Batchable<SObject>, Database.Stateful {

    private BatchQueueItem__c currentWork;
    private List<Id> recordIds;
    private BatchWorkerCommand command;

    public ContinuousWorkerBatch() {
        this.currentWork = [SELECT txt_CommandClassName__c, ltxt_IdsToProcess__c
                            FROM BatchQueueItem__c
                            ORDER BY CreatedDate ASC
                            LIMIT 1];                                  
        this.command = newFromName(this.currentWork.txt_CommandClassName__c);  
        this.recordIds = parseToList(this.currentWork.ltxt_IdsToProcess__c);                       
    }

    global List<SObject> start(Database.BatchableContext context) {
       return queryRecordsForIds(this.recordIds);
    }

    global void execute(Database.BatchableContext context, List<SObject> scope) {
        this.command.execute(scope);
        updateRecordIds(scope);
    }


    global void finish(Database.BatchableContext context) {
        ContinuousWorkerBatch same = new ContinuousWorkerBatch();

        if(this.recordIds.isEmpty()) {
            delete this.currentWork;
            Database.scheduleBatch(same, 1);
        }
        else {
            this.currentWork.ltxt_IdsToProcess__c = String.join(this.recordIds, ',');
            update this.currentWork;
            Database.executeBatch(same);
        }
    }
        ...
}

But the basic idea is that only a single batch is constantly running and not many are independently triggered and fighting against each other.

Work could be registered by adding record Ids and the name of a Command pattern like class in the database. Maybe just a long text field to be able to store thousands of pairs.

public interface BatchWorkerCommand {
    void execute(List<SObject> recordsToBeProcessed);
}

Order of execution could be respected and we would avoid race conditions between batches.

  1. Is this a stupid idea after a few glasses of wine ;) or a feasible solution?
  2. Have you done something similar and can share some lessons learned?
  3. What what you change improve or take into account?

Any feedback is appreciated.

  • 1
    Great idea Robert! – Andrew Fawcett May 21 '13 at 6:40
  • Thanks Andrew! Have you ever used/seen/thought of such a custom centralized batch queue?! – Robert Sösemann May 21 '13 at 9:55
  • 1
    Yes its been a consideration on and off, i kind of keep hoping Salesforce will someday address the concurrent batch job limit vs investing development of such a framework. There is also the challenge of where to put the queue object if you have multiple products/packages. Dependent packages are not always desirable depending on the level of coupling, but is an option. – Andrew Fawcett May 22 '13 at 7:04
4

I've done this a few times. It works well. It can take a while to get the thing running end-to-end, since debugging can be arduous.

A few suggestions:

  • in your command table, add a "log" long text field and write back relevant status/debugging info there.
  • catch all exceptions from your batch manager and write them out to the log rather than blowing up.
  • you can use the Apex dynamic instantiation (reflection) to allow your command table to hold actual class names if you want to maximize flexibility
  • I use a number of common base classes for my batch jobs, that do certain common stuff like logging, recording start/stop times and record counts, self-rescheduling (via command table) for recurring jobs, etc.

Hope that helps.

  • Jkraybill in which fieldtype and format did you store the ids. If i store to many of them in a single record i run out of chars. If i store too less i reproduce many limit problems in the infrastructure... – Robert Sösemann May 21 '13 at 5:42
  • 1
    How many jobs are you queuing at any given time? We do one record per job, with fields for class name, parameters, and other data like start/stop time, queue time, duration, records processed, etc. We can potentially generate thousands of records per day, but we also run a scheduled job that deletes completed job records over N days old (a configuration value). – jkraybill May 21 '13 at 6:12
  • During a day we might update millions of records and cannot use the one id per queue records approach, as adding record there is as expensive (limit wise) as just performing our original DML. I though of adding thousands of Ids to a long text field. But then I would have to find a good format. – Robert Sösemann May 21 '13 at 7:29
  • What triggers a record being put in the queue? Can you not use some range-based proxy like last modified date, or a checkbox field that gets set on each target record by a workflow or trigger? – jkraybill May 21 '13 at 7:50
  • 1
    My pattern was pretty different from yours, but it was done before batches could schedule themselves. I have a scheduled job manager that wakes up, finds a job (which is itself a batch job), then calls execute on the job and goes to sleep. Then the job does its work and reschedules the job manager. But your code looks just fine. I also ended up writing a lot of debug logging (to data rows) because it's incredibly difficult to figure out what went wrong after the fact when you hit an exception etc. – jkraybill May 22 '13 at 2:17
2

Robert, the general gist of what you're suggesting is much better described in great detail in Dan Appleman's book Advanced Apex Programming. In his book, he details how to have all triggers for a given object go through a "central dispatcher" to call other batchable classes to perform the necessary work AND keep track of where those batches are at in terms of iterations.

He also has a means for including a custom diagnostics and error handling code class in his "central dispatcher". With his methodology, one can easily control the order of trigger execution. The flow he describes also has the ability to implement the kind of scaling that you're speaking of. You might also be interested to know that the last Chapter of the book is specific to Designing for Packages.

I highly recommend this book to anyone who wants to advance their understanding of programming in Apex for the Salesforce platform. Dan really has a good grasp of the pertinent issues and does an excellent job of explaining them. I've never seen a better coverage of unit testing, governors, limits using real world examples; including a case where the total lines of code became more of an issue than the number queries! Dan genuinely teaches you how to "think in Apex" as applies to Salesforce.

BTW, you can download the sample code from the book's website which can be installed as an unmanaged packaged at http://www.AdvancedApex.com/samplecode. Without the book to put it in context, it may not be nearly as useful as it otherwise would be. I should also mention that much of the code is commented out for experimentation and study purposes.

  • I own this book but had a really hard time understanding its terribly formatted code examples. Where exactely do you see my idea implemented? – Robert Sösemann May 18 '13 at 22:03
2

To solve my problem and invite other to provide feedback and collaborate on such a custom Apex queue I created the GitHub project SObject Work Queue which had the following design goals:

  • Must prevent Max 5 batch in parallel limit - We should never run into this limit with work that is processed over the queue.
  • The queue needs to be so generic that "work" on any type of database object needs to be enqueued.
  • Any type of modification of database objects need to be possible. This should be transparently handled by the queue.
  • Provides better error diagnostics like Batch. Knows last successful Id, full stacktrace and sends email to developers.
  • Secures data integrity like Batch or better. Failures should not leave data in inconsistent state or user of the infrastructure should be able to handle them.
  • Optimistic locking : Instead of locking all many records we do not to process work on Ids that have other work already scheduled.
  • Work that can be run synchronously, should not be queued and processed asynch.

Here is an overview sequence diagram that shows how work is defined, enqueued and processed in such a custom Apex Queue:

enter image description here

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.