Chaining Scheduled Apex to process callouts every 30 seconds

Question

We wanted to create an outbound message queue system that would process requests individually and in order to any external web service. The frequency that these requests should be processed is variable but as an example, it might be that we would like the process to run once every 30 seconds.

It is not possible or practical to schedule the same job 120 times for every 30 seconds within the hour.

The approach I finally settled on was to chain schedule apex (wait until scheduled time, call future method and delete (abort) itself) -> future call (process request if there is one and reschedule class to rerun in 30 seconds) -> schedule apex (call future method in 30 seconds, abort itself) -> future call (process request and reschedule) and so on indefinitely. The end result has been running on production for 2 weeks now but has just failed to reschedule itself and I do not understand why.

I believe what is happening is that I am breaching some kind of governor limit but I don't understand how. I have been having similar issues more frequently on my sandbox and have received a limit exception stating that the limit for future requests has been exceeded.

From the point that the service failed, I queried for all asynchronous jobs (not just future) from the previous 24 hours and it returned just 2200. I believe this figure should be 4400 because I think the schedule jobs are not showing up (The scheduled jobs abort themselves after they call the future method).

Even so, Salesforce states that we are allowed 250,000 asynchronous apex executions every 24 hours so I don't understand what is going on. Does anyone have any experience trying to chain scheduled jobs in this way? Is there a better way to achieve this? Is there a limit that I am not aware of or am I measuring asynchronous executions in the wrong way?

Any help or advice would be greatly appreciated.

Answer 1

I've used this pattern in a number of projects to achieve this, by calling this from the finish method of the Batch Apex class:

private void reSubmitJob( )
{
        // try again in a minute
        Datetime sysTime = System.now().addSeconds( 60 );

        String chronExpression = '' + sysTime.second() + ' ' + sysTime.minute() + ' ' + sysTime.hour() + ' ' + sysTime.day() + ' ' + sysTime.month() + ' ? ' + sysTime.year();

        System.schedule( 'Scheduled Apex ' + sysTime, chronExpression, this );
}

Then, in your Schedule execute call Database.executeBatch() and abort the scheduled job System.abortJob(SC.getJobTriggerId()) to remove the 'used' job from the Scheduled Job list.

Note, Summer '13 gives you the System.scheduleBatch method which may also be useful for what you are trying to achieve. Though I have noticed some potential issues with that myself.

EDIT: one of the things to be wary of is the number of seconds you specify for your re-schedule. Sometimes, if this is too low (e.g. 30 seconds) the system takes longer than that to schedule it and the time passes and your job never runs. I've found 1 minute to be generally acceptable but have also been caught out with that on some occasions when I guess there is a lot of activity.

EDIT 2: this is not deemed best practice, see post Is there a simple way to serialize execution of Batchable class?. It's called 'suicidal apex' :D and should be used with caution.