Scheduled batch jobs and durability

Question

I have a process that runs scheduled Apex quite regularly as a batch job manager. This process runs on a one-time schedule, and looks into a batch job table (custom object). If it finds work to do, it kicks off a batch job.

All my batch jobs re-schedule the main batch manager to run a short time in the future after they complete. Similarly, if the main batch manager finds no jobs to do, it schedules itself to run again a short time in the future. Since these jobs are serialized in that way, there's only ever a maximum of one batch job running at a time, and only ever a maximum of one scheduled apex job.

This works great, except that very occasionally (seems to be once every few months), the scheduled job "disappears" and has to be rescheduled manually.

My batch jobs maintain thorough logs of their activity, and have very conservative exception handling - other than LimitExceptions (which I am 100% certain I am not throwing), all Exceptions are caught and logged, and the normal execution flow is then followed.

I'm not sure why the job sometimes dies, but I'm fairly certain it is not code-related. I don't have enough data yet, but it appears that the times this has happened in the past coincide with major upgrade windows from Salesforce. The last time I saw it was on NA14 on June 16 at exactly 04:00 UTC. That wasn't the stated major update window (which was listed as 06:00 UTC on June 15) but is suspiciously similar to the update times cited by Salesforce for that weekend.

• has anyone seen similar behaviour, where in-process batch jobs and/or scheduled Apex jobs get summarily killed during upgrade windows?
• has anyone addressed this issue, and if so, what was the solution? My only thought is to try a second regularly scheduled job that runs, say, once per day at a time not associated with maintenance windows, and is purely a sentinel process to make sure the batch job manager is properly scheduled and/or running. However, that will cost a very valuable open scheduled job slot, which I'd prefer to avoid. I suppose I could also add similar sentinel code into the UI of my app, but it's not always guaranteed that users will be hitting any part of the UI every day.

Any help appreciated!

Answer 1

Rather than scheduling a one-time job, schedule a recurring job.

Schedule the job to run on an hourly interval (every hour). As part of the finishing phase of your job, cancel this hourly schedule and replace it with another similar hourly schedule where the first execution is set to be a short period (let's say 5 minutes) from the finish of the job.

This works in a very similar way to using a "one off" schedule (as per your existing implementation) - in both of these implementations the job is rescheduled in the finish phase, but by using a recurring schedule you have the added benefit that if for any reason the job does not execute, the platform will attempt to run it again an hour later, and every hour until it succeeds.

Note that we don't know why the job may fail to execute - but we're assuming that it relates to platform maintenance. Chaining one-off scheduled jobs together relies on the successful start and completion of each job for the integrity of the chain, whereas using a recurring scheduled job provides "auto-resume" behaviour regardless of the successful start / completion of an individual job.

Example process flow:

(1) at 12:00 we schedule a job to run every every hour, at 5 minutes past the hour: 12:05,13:05,14:05...etc...

(2) at 12:05 the batch manager job is started according to the hourly schedule, and this checks your custom batch job object records to see if there is any work currently running or waiting.

It finds that there are no jobs running but there is a job waiting: "Foo". The batch manager therefore starts the batch process for Foo.

(3) at 13:05 the batch manager job is started according to the hourly schedule.

On this occasion it finds that job Foo is in progress and so quits taking no action.

(4) at 13:35 job Foo finishes.

In the finish phase, the existing hourly scheduled job is cancelled, and another new hourly job is scheduled, this time to run at 40 minutes past the hour: 13:40, 14:40, 15:40...etc…

(5) at 13:40 the batch manager job is due to start according to the hourly schedule, but this fails (we assume because of platform maintenance)

(6) at 14:40 the batch manager job is started according to the hourly schedule.

It finds that there are no jobs running but there is a job waiting: "Bar". The batch manager therefore starts the batch process for Bar.

etc.

Answer 2

I have seen this type of behaviour on a number of occasions. It seems to be related to (in my experience) what you've said above :

if the main batch manager finds no jobs to do, it schedules itself to run again a short time in the future.

And on occasions that 'short time' passes before the job has had chance to successfully execute, either as a result of high load or as you say above a maintenance window.

Have you considered a second 'keep-alive' scheduled job that say runs once an hour (less likely to be impacted by aforementioned issues) that checks things are in order and if not reschedules your initial job?

Edit: As it happens, I am doing some testing in a Sandbox right now where we have increased the scheduled job to run every minute (just while users are doing some hefty testing) and I'm getting this very issue, in this situation the scheduled job ends up not having a Next Start time and hangs in limbo. So, the Keep Alive job is going to looks something like:

global class KeepAlive implements schedulable
{    
    global void execute( SchedulableContext SC )
    {        
        // have the worker job store its job id in custom setting
        JobIdState__c jobIdState = JobIdState__c.getInstance();
        Id workerId = (Id)jobIdState.JobId__c;

        CronTrigger dead = [ select Id From CronTrigger where Id = :workerId And NextFireTime = null ];

        // abort the dead job and start a new one
        if( dead != null )
        {
            System.abortJob( workerId );

            // start new one again in a minute
            Datetime sysTime = System.now().addSeconds( 60 );      
            String chronExpression = '' + sysTime.second() + ' ' + sysTime.minute() + ' ' + sysTime.hour() + ' ' + sysTime.day() + ' ' + sysTime.month() + ' ? ' + sysTime.year();

           System.schedule( 'Worker Scheduler ' + sysTime, chronExpression, new WorkerScheduler() );
        }       

        // abort me and start again
        System.abortJob( SC.getTriggerId() );

        KeepAlive.start();

    }

    public static void start()
    {
        // start keepalive again in 5 mins
        Datetime sysTime = System.now().addSeconds( 300 );      
        String chronExpression = '' + sysTime.second() + ' ' + sysTime.minute() + ' ' + sysTime.hour() + ' ' + sysTime.day() + ' ' + sysTime.month() + ' ? ' + sysTime.year();
        System.schedule( 'KeepAlive ' + sysTime, chronExpression, new KeepAlive() );
    }

}

I've just implemented this now, and will obviously have to wait until the next failure to determine whether it has worked or not but thought it might be useful.

Edit 2: Jobs scheduled in code do count towards limits, however spent one's without a NextFireTime don't seem to.

Answer 3

How about a "dead man's switch" using a "sentinel" custom object, time-based workflow rule and a trigger?:

• You have a sentinel object, where a single record represents your batch job manager
• Your batch job manager updates the sentinel record with a datetime field to say "I'm alive now"
• Your time-based workflow fires a period of time after "I'm alive now", and sets a "reboot" field on the record
• A trigger on your sentinel object reschedules the batch job manager when it sees a "reboot"

NB: Completely untried :-)

Answer 4

I've come up with what I think is a fairly durable solution for Near Realtime batches:

global class NearRealTimeBatch implements Database.Batchable<sObject>, Schedulable  {

    private String qry = 'SELECT Id FROM Account WHERE Needs_Processing__c = true';

    private transient Database.QueryLocator locator;

    global NearRealTimeBatch(){}

    global NearRealTimeBatch(Database.QueryLocator locator) {
        this.locator = locator;
    }

    //create a new query locator, unless one has been passed in
    global Database.QueryLocator start(Database.BatchableContext BC) {
        if(locator == null){
            locator = Database.getQueryLocator(qry);
        }
        return locator;
    }


    global void execute(Database.BatchableContext BC, List<sObject> scope) {
        //do work

        //mark "needs processing" as false
        for(Account acc : (List<Account>) scope){
            account.Needs_Processing__c = false;
        }

        update scope;
    }

    /**
     * Batch Finish method
     * Check to see if any records have been created since we first started this process.
     * if so, run Sync again.
     * Otherwise, we'll schedule it to run in 2 minutes
     */
    global void finish(Database.BatchableContext BC) {
        locator = Database.getQueryLocator(qry);
        if(locator.iterator().hasNext()){
            System.debug('More rows found.  Run again!');
            Database.executeBatch(new NearRealTimeBatch(locator));
            return;
        }

        //otherwise schedule it to run in 2 minutes
        System.scheduleBatch(new NearRealTimeBatch(), 'Chained Sync Handle', 2);
    }

    /**
     * Scheduable execute method. Used to restart process if for somereason it is stopped.  
     * Should be scheduled to run as often as possible.  
     * Kicks off batch process if it is not Queued, or running.
     */
    private static final String[] RUNNING_STATUS = new String[]{'Queued', 'Preparing', 'Processing'};
    global void execute(SchedulableContext SC) {
        //make sure we are not already executing
        List<AsyncApexJob> activeJobs = [SELECT Id
                                        FROM AsyncApexJob 
                                        WHERE ApexClass.Name = :NearRealTimeBatch.Class.getName()
                                        AND Status IN:RUNNING_STATUS];
        if(activeJobs.size() > 0){
            return;
        }
        Database.executeBatch(this);
    }

}

Here's how it works:

1. When a batch is done executing, it will recreate the query locator and check to see if any new eligible records have been created. This assumes there is a field marking which records need to be processed. If you do not need this ability, ignore this step.
2. If #1 does not trigger a batch execution schedule it to run again in x minutes.
3. Set up a Schedulable to check and see if the process is queued or running. If its not, start it again. This helps handle any scenario where the batch somehow exits without scheduling itself again.

Also worth noting... If you do implement part one, it's possible to end in an endless batch loop if a record fails to update. Make sure that you have some means for catching this and excluding the record.

Answer 5

No I did never see this behavior but I have recently built something similar to circumvent the max 5 batch limitation.

A ContinuousWorkerBatch class that checks in its start() method if work needs to be done and processes it. In his finish method it would calls a new instance of itself.

public class ContinuousWorkerBatch implements Database.Batchable<SObject>, Database.Stateful {


    global List<SObject> start(Database.BatchableContext context) {
       return getNewWork();
    }

    global void execute(Database.BatchableContext context, List<SObject> scope) {
        ...
    }


    global void finish(Database.BatchableContext context) {
        ContinuousWorkerBatch same = new ContinuousWorkerBatch();
            Database.executeBatch(same);
    }
        ...
}

Didn't you even answer on my question?!

Answer 6

Not sure if this will help you here @jkraybill. I read this in Apex language reference that whenever some maintenance operation is running, we can detect that using ApplicationReadWriteMode. Probably using similar code like the following to check if ApplicationMaintenance is running in Salesforce and applying to your scenario might work. Probably, you might just use this code to check if SF is under maintenance and some more code to check if your schedule is stopped or not when any job is entered in your custom object. am not sure though!! Just guessing :)

To check if 5 Minute Upgrade from Salesforce is running (as part of maintenance) :- Example code :-

public class myClass 
    {
        public static void execute() 
        {
            ApplicationReadWriteMode mode = System.getApplicationReadWriteMode();
            if (mode == ApplicationReadWriteMode.READ_ONLY) 
            {
            // Do nothing. If DML operaton is attempted in readonly mode,
            // InvalidReadOnlyUserDmlException will be thrown.
            } 
            else if (mode == ApplicationReadWriteMode.DEFAULT) 
            {
            Account account = new Account(name = 'my account');
            insert account;
            }
        }
    }