25

I have a process that runs scheduled Apex quite regularly as a batch job manager. This process runs on a one-time schedule, and looks into a batch job table (custom object). If it finds work to do, it kicks off a batch job.

All my batch jobs re-schedule the main batch manager to run a short time in the future after they complete. Similarly, if the main batch manager finds no jobs to do, it schedules itself to run again a short time in the future. Since these jobs are serialized in that way, there's only ever a maximum of one batch job running at a time, and only ever a maximum of one scheduled apex job.

This works great, except that very occasionally (seems to be once every few months), the scheduled job "disappears" and has to be rescheduled manually.

My batch jobs maintain thorough logs of their activity, and have very conservative exception handling - other than LimitExceptions (which I am 100% certain I am not throwing), all Exceptions are caught and logged, and the normal execution flow is then followed.

I'm not sure why the job sometimes dies, but I'm fairly certain it is not code-related. I don't have enough data yet, but it appears that the times this has happened in the past coincide with major upgrade windows from Salesforce. The last time I saw it was on NA14 on June 16 at exactly 04:00 UTC. That wasn't the stated major update window (which was listed as 06:00 UTC on June 15) but is suspiciously similar to the update times cited by Salesforce for that weekend.

  • has anyone seen similar behaviour, where in-process batch jobs and/or scheduled Apex jobs get summarily killed during upgrade windows?
  • has anyone addressed this issue, and if so, what was the solution? My only thought is to try a second regularly scheduled job that runs, say, once per day at a time not associated with maintenance windows, and is purely a sentinel process to make sure the batch job manager is properly scheduled and/or running. However, that will cost a very valuable open scheduled job slot, which I'd prefer to avoid. I suppose I could also add similar sentinel code into the UI of my app, but it's not always guaranteed that users will be hitting any part of the UI every day.

Any help appreciated!

  • 3
    A small bit of good news: there's now 100 scheduled job slots on Summer13 orgs. A sentinel process should be much less "expensive" in terms of resources now. – ca_peterson Jun 25 '13 at 20:04
  • Whoa I missed that awesome! That actually greatly eases the pain of a sentinel process, although Stephen Willcock's idea is still more elegant if it works. – jkraybill Jun 25 '13 at 23:44
  • Yeah @jkraybill am following the same process and my master batchjob execute every 1 minute and if there is an entry in the custom job object, it sees how many batch jobs can fit in the queue (based on the limits) and runs those and whenever a slot is free it pulls the next batchjob and executes it. I had the same problem, during the last SF maintenance window, the schedule stopped and I had to reschedule it through the console. Sometimes it stops during some of my deployments as well. So I just ensure manually that this job is running after my deployments and SF maintenance. :( – Sathya Jun 28 '13 at 20:03
20
+100

Rather than scheduling a one-time job, schedule a recurring job.

Schedule the job to run on an hourly interval (every hour). As part of the finishing phase of your job, cancel this hourly schedule and replace it with another similar hourly schedule where the first execution is set to be a short period (let's say 5 minutes) from the finish of the job.

This works in a very similar way to using a "one off" schedule (as per your existing implementation) - in both of these implementations the job is rescheduled in the finish phase, but by using a recurring schedule you have the added benefit that if for any reason the job does not execute, the platform will attempt to run it again an hour later, and every hour until it succeeds.

Note that we don't know why the job may fail to execute - but we're assuming that it relates to platform maintenance. Chaining one-off scheduled jobs together relies on the successful start and completion of each job for the integrity of the chain, whereas using a recurring scheduled job provides "auto-resume" behaviour regardless of the successful start / completion of an individual job.

Example process flow:

(1) at 12:00 we schedule a job to run every every hour, at 5 minutes past the hour: 12:05,13:05,14:05...etc...

(2) at 12:05 the batch manager job is started according to the hourly schedule, and this checks your custom batch job object records to see if there is any work currently running or waiting.

It finds that there are no jobs running but there is a job waiting: "Foo". The batch manager therefore starts the batch process for Foo.

(3) at 13:05 the batch manager job is started according to the hourly schedule.

On this occasion it finds that job Foo is in progress and so quits taking no action.

(4) at 13:35 job Foo finishes.

In the finish phase, the existing hourly scheduled job is cancelled, and another new hourly job is scheduled, this time to run at 40 minutes past the hour: 13:40, 14:40, 15:40...etc…

(5) at 13:40 the batch manager job is due to start according to the hourly schedule, but this fails (we assume because of platform maintenance)

(6) at 14:40 the batch manager job is started according to the hourly schedule.

It finds that there are no jobs running but there is a job waiting: "Bar". The batch manager therefore starts the batch process for Bar.

etc.

  • (deleted my earlier comment because I misunderstood what you were saying.) This seems like a fine idea actually. So the job, rather than being scheduled one-time (say 12:05am today), gets scheduled today on a recurrent schedule (12:05am, every hour on the hour)? And when it runs, it cancels its own job and reschedules itself (to say 12:15am every hour on the hour)? This sounds like a great solution. Definitely bounty-worthy if it works, thanks! – jkraybill Jun 25 '13 at 13:14
  • 2
    Yep you got it. Probably should have included an example (such as you have provided) in the answer :-) – Stephen Willcock Jun 25 '13 at 13:16
  • 2
    Please do, it will make your answer so much better. I'm still wrapping my head around it, and I love examples! – Daniel Blackhall Jun 26 '13 at 21:52
  • I have tired to clarify the answer, and have provided an example flow. Sorry about that, meant to expand the answer sooner! – Stephen Willcock Jun 27 '13 at 9:29
  • Haven't implemented this yet but it seems like the best option, don't spend your bounty all at once! :) – jkraybill Jul 1 '13 at 2:49
11

I have seen this type of behaviour on a number of occasions. It seems to be related to (in my experience) what you've said above :

if the main batch manager finds no jobs to do, it schedules itself to run again a short time in the future.

And on occasions that 'short time' passes before the job has had chance to successfully execute, either as a result of high load or as you say above a maintenance window.

Have you considered a second 'keep-alive' scheduled job that say runs once an hour (less likely to be impacted by aforementioned issues) that checks things are in order and if not reschedules your initial job?

Edit: As it happens, I am doing some testing in a Sandbox right now where we have increased the scheduled job to run every minute (just while users are doing some hefty testing) and I'm getting this very issue, in this situation the scheduled job ends up not having a Next Start time and hangs in limbo. So, the Keep Alive job is going to looks something like:

global class KeepAlive implements schedulable
{    
    global void execute( SchedulableContext SC )
    {        
        // have the worker job store its job id in custom setting
        JobIdState__c jobIdState = JobIdState__c.getInstance();
        Id workerId = (Id)jobIdState.JobId__c;

        CronTrigger dead = [ select Id From CronTrigger where Id = :workerId And NextFireTime = null ];

        // abort the dead job and start a new one
        if( dead != null )
        {
            System.abortJob( workerId );

            // start new one again in a minute
            Datetime sysTime = System.now().addSeconds( 60 );      
            String chronExpression = '' + sysTime.second() + ' ' + sysTime.minute() + ' ' + sysTime.hour() + ' ' + sysTime.day() + ' ' + sysTime.month() + ' ? ' + sysTime.year();

           System.schedule( 'Worker Scheduler ' + sysTime, chronExpression, new WorkerScheduler() );
        }       

        // abort me and start again
        System.abortJob( SC.getTriggerId() );

        KeepAlive.start();

    }

    public static void start()
    {
        // start keepalive again in 5 mins
        Datetime sysTime = System.now().addSeconds( 300 );      
        String chronExpression = '' + sysTime.second() + ' ' + sysTime.minute() + ' ' + sysTime.hour() + ' ' + sysTime.day() + ' ' + sysTime.month() + ' ? ' + sysTime.year();
        System.schedule( 'KeepAlive ' + sysTime, chronExpression, new KeepAlive() );
    }

}

I've just implemented this now, and will obviously have to wait until the next failure to determine whether it has worked or not but thought it might be useful.

Edit 2: Jobs scheduled in code do count towards limits, however spent one's without a NextFireTime don't seem to.

  • Yeah that's my only idea right now, but I would really like to avoid having to take up two scheduled job slots if I can avoid it. – jkraybill Jun 22 '13 at 1:57
  • @jkraybill As Stephen points out, I think only jobs scheduled using the UI (as opposed to programatically) will use one of your slots so if you start your keepalive job programatically you won't lose the second slot. – Phil Hawthorn Jun 25 '13 at 8:52
  • Hey Phil, I didn't mean to imply that - I would have thought all scheduled jobs would count toward the limit however they are started – Stephen Willcock Jun 26 '13 at 6:15
  • Just checked docs: "Note that the 100 concurrent scheduled classes limit doesn’t apply to scheduled batch jobs started using the System.scheduleBatch method" - i.e. "to schedule a batch job to run once at a specified time in the future" – Stephen Willcock Jun 26 '13 at 6:25
  • I probably misunderstood :). That said, I still don't think that scheduled jobs submitted like this (i.e. to run once at a specified time in the future) count towards the limit but like I say I'm not 100% sure. Also, see my post about scheduleBatch, it seems unreliable to me salesforce.stackexchange.com/questions/12794/… – Phil Hawthorn Jun 26 '13 at 6:36
4

How about a "dead man's switch" using a "sentinel" custom object, time-based workflow rule and a trigger?:

  • You have a sentinel object, where a single record represents your batch job manager
  • Your batch job manager updates the sentinel record with a datetime field to say "I'm alive now"
  • Your time-based workflow fires a period of time after "I'm alive now", and sets a "reboot" field on the record
  • A trigger on your sentinel object reschedules the batch job manager when it sees a "reboot"

NB: Completely untried :-)

  • I think this is a solid concept. Will try it out and report back. – jkraybill Jun 25 '13 at 0:46
  • 2
    Hmmm.... time-based workflows aren't able to be packaged in managed packages still, are they? Argh. – jkraybill Jun 25 '13 at 0:48
  • 3
    Anecdotal and old, but: I once tried something like this (chaining time based workflows) before the invent of scheduled apex and it ended up being spectacularly unreliable. Often failed with cryptic messages like "No FileForce server available" if you ran too often. – ca_peterson Jun 26 '13 at 0:24
2

I've come up with what I think is a fairly durable solution for Near Realtime batches:

global class NearRealTimeBatch implements Database.Batchable<sObject>, Schedulable  {

    private String qry = 'SELECT Id FROM Account WHERE Needs_Processing__c = true';

    private transient Database.QueryLocator locator;

    global NearRealTimeBatch(){}

    global NearRealTimeBatch(Database.QueryLocator locator) {
        this.locator = locator;
    }

    //create a new query locator, unless one has been passed in
    global Database.QueryLocator start(Database.BatchableContext BC) {
        if(locator == null){
            locator = Database.getQueryLocator(qry);
        }
        return locator;
    }


    global void execute(Database.BatchableContext BC, List<sObject> scope) {
        //do work

        //mark "needs processing" as false
        for(Account acc : (List<Account>) scope){
            account.Needs_Processing__c = false;
        }

        update scope;
    }

    /**
     * Batch Finish method
     * Check to see if any records have been created since we first started this process.
     * if so, run Sync again.
     * Otherwise, we'll schedule it to run in 2 minutes
     */
    global void finish(Database.BatchableContext BC) {
        locator = Database.getQueryLocator(qry);
        if(locator.iterator().hasNext()){
            System.debug('More rows found.  Run again!');
            Database.executeBatch(new NearRealTimeBatch(locator));
            return;
        }

        //otherwise schedule it to run in 2 minutes
        System.scheduleBatch(new NearRealTimeBatch(), 'Chained Sync Handle', 2);
    }

    /**
     * Scheduable execute method. Used to restart process if for somereason it is stopped.  
     * Should be scheduled to run as often as possible.  
     * Kicks off batch process if it is not Queued, or running.
     */
    private static final String[] RUNNING_STATUS = new String[]{'Queued', 'Preparing', 'Processing'};
    global void execute(SchedulableContext SC) {
        //make sure we are not already executing
        List<AsyncApexJob> activeJobs = [SELECT Id
                                        FROM AsyncApexJob 
                                        WHERE ApexClass.Name = :NearRealTimeBatch.Class.getName()
                                        AND Status IN:RUNNING_STATUS];
        if(activeJobs.size() > 0){
            return;
        }
        Database.executeBatch(this);
    }

}

Here's how it works:

  1. When a batch is done executing, it will recreate the query locator and check to see if any new eligible records have been created. This assumes there is a field marking which records need to be processed. If you do not need this ability, ignore this step.
  2. If #1 does not trigger a batch execution schedule it to run again in x minutes.
  3. Set up a Schedulable to check and see if the process is queued or running. If its not, start it again. This helps handle any scenario where the batch somehow exits without scheduling itself again.

Also worth noting... If you do implement part one, it's possible to end in an endless batch loop if a record fails to update. Make sure that you have some means for catching this and excluding the record.

1

No I did never see this behavior but I have recently built something similar to circumvent the max 5 batch limitation.

A ContinuousWorkerBatch class that checks in its start() method if work needs to be done and processes it. In his finish method it would calls a new instance of itself.

public class ContinuousWorkerBatch implements Database.Batchable<SObject>, Database.Stateful {


    global List<SObject> start(Database.BatchableContext context) {
       return getNewWork();
    }

    global void execute(Database.BatchableContext context, List<SObject> scope) {
        ...
    }


    global void finish(Database.BatchableContext context) {
        ContinuousWorkerBatch same = new ContinuousWorkerBatch();
            Database.executeBatch(same);
    }
        ...
}

Didn't you even answer on my question?!

  • Could you not possibly fall foul of "The maximum number of batch Apex method executions per a 24-hour period is 250,000 or the number of user licenses in your organization multiplied by 200, whichever is greater. Method executions include executions of the start, execute, and finish methods." if there are reasonable periods of time where there is no NewWork? – Phil Hawthorn Jun 26 '13 at 21:09
  • Yes you are right. But then just replace Database.executeBatch with the new scheduleBatch wit param set to 1 min and you schould be safe. – Robert Sösemann Jun 26 '13 at 22:24
  • I did answer on your question, and in fact this is the job that is sometimes stopping for me -- my issue is not the batch management, it's that occasionally the job just dies (I believe around SFDC maint windows). Stephen Willcock's idea of using a recurrent cron expression, and then cancelling/rescheduling in one transaction, seems the best option to prevent this. – jkraybill Jun 27 '13 at 0:08
  • 2
    Good luck using System.scheduleBatch :) – Phil Hawthorn Jun 27 '13 at 6:28
0

Not sure if this will help you here @jkraybill. I read this in Apex language reference that whenever some maintenance operation is running, we can detect that using ApplicationReadWriteMode. Probably using similar code like the following to check if ApplicationMaintenance is running in Salesforce and applying to your scenario might work. Probably, you might just use this code to check if SF is under maintenance and some more code to check if your schedule is stopped or not when any job is entered in your custom object. am not sure though!! Just guessing :)

To check if 5 Minute Upgrade from Salesforce is running (as part of maintenance) :- Example code :-

public class myClass 
    {
        public static void execute() 
        {
            ApplicationReadWriteMode mode = System.getApplicationReadWriteMode();
            if (mode == ApplicationReadWriteMode.READ_ONLY) 
            {
            // Do nothing. If DML operaton is attempted in readonly mode,
            // InvalidReadOnlyUserDmlException will be thrown.
            } 
            else if (mode == ApplicationReadWriteMode.DEFAULT) 
            {
            Account account = new Account(name = 'my account');
            insert account;
            }
        }
    }

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.