7

We have a number of Scheduled Jobs that are scheduled with repeat and these are all working fine. We have a handful of Scheduled Jobs that are dynamically scheduled and re-scheduled, either as one-time executions of Schedulables or as System.scheduleBatch executions of Batchables. These were also working fine until the last few days. Now we find that the latter types of Scheduled Job are having problems executing on some orgs.

For example, the following was captured from a customer org around 18:14 today:

enter image description here

The two highlighted "Next Scheduled Run" items have completely stalled, yet some others (like to top most one) are still able to work. As time goes by, more of these stall.

By "stalled" I mean the scheduled time passes but the job does not get executed (even 30 minutes [or indeed way more] after that scheduled time, as shown in the above screenshot).

By "dynamically scheduled and re-scheduled" I mean that, for example, the Batchable calls System.scheduleBatch(this, '*Batch Name*', delay); in its own finish method, or that the Schedulable is first scheduled using:

Datetime next = System.now().addSeconds(Integer.valueOf(delay));
String schedule = next.format('ss mm HH dd MM ? yyyy');
System.schedule('*Schedulable Name*', schedule, schedulable);

And at the end of execute calls System.abortJob(context.getTriggerId()); then effectively repeats its initial scheduling.

This inability to run these Scheduled Jobs is having a detrimental impact on our customers. Our only workaround is to periodically kill and re-start these stalled jobs which basically means having an admin watch the scheduled jobs and undertake manual interventions as needed.

Has anyone else seen this issue, perhaps with migration of a given org to Winter '21? Have you found an alternative workaround?

1
  • Stalled how? Haven't seen it in our orgs, fwiw. Would be interesting to see the skeleton of code for dynamically scheduled and re-scheduled so we're all on the same page. – identigral Oct 19 '20 at 18:30
8

FURTHER UPDATE:

It seems that, whilst there is some sympathy from within Salesforce, the Salesforce PM and R&D team responsible believe this is a "feature enhancement", not the reporting of a bug, because they "already did the analysis" (some 6 years ago).

Their main argument is that "there is an existing backlog of features, fixes and improvements that each item needs to be weighed up against and, in this particular case, the view is that there is a workaround. What's more, a change of this nature requires a level of due diligence to avoid a knock-on impact to our customers and to our own platform."

My response to that latter point (the "knock-on impact") is that, I think it fair to say, no one could possibly be relying on the system failing to honour an at-the-time future scheduling request. Also, given this problem infrequently happens (we've seen it for one customer on one [production] org, albeit for 24hrs, but only immediately after the Winter '21 upgrade), the actual delta in processing load on the Salesforce Platform for honouring such requests is negligible. The true knock-on here, for me, is that a fix would reinforce trust in the Salesforce Platform, unlike the current behaviour which corrosively damages that trust.

The view that there is a workaround is, for us, unfortunately naïve; our customers rely on relatively short delays in the data processing and to extend this from one or two minutes to a minimum of 15 isn't tenable for them.

The journey continues, however, and we will continue to discuss this situation with Salesforce in the new year. I'll post another update then.


UPDATE:

Long story short, Salesforce support asked me to raise an "idea" about this bug to gauge interest in a fix. Please please please upvote it!


ORIGINAL POSTING:

Having followed this up with Salesforce support, we finally got the following response after several weeks going round and round with them:

We use Quartz (http://www.quartz-scheduler.org/) for this piece of functionality. If the "system.schedule" is called at 2:13:17 (next to Fire Time) and takes more time to call Quartz, due to resource unavailability, then the time specified in the schedule will be in the past, and Quartz will never fire the trigger.

The resource availability dependency is the CRUX of Async processing. Thus, if due to reasons, the server gets busy and resource is not available, and it surpasses the NextFireTime of the crontrigger, it will cease the execution of scheduled job and status will be set to 'Waiting'.

As a long-term solution and avoid the late realization of schedule jobs ceasing, we suggest you to ensure the exception handling(if job ceases due to an exception) and retry mechanism (checking the Cron job status and rescheduling).

Also it is recommended to have a difference of at least 15 minutes for each execution.

I don't know about you, but to me this is nuts.

The official documentation doesn't mention any recommendation or requirement to use a minimum offset in the future of 15 minutes. Indeed, the documentation for batch apex and specifically use of System.scheduleBatch says:

This example schedules a batch job to run one minute from now by calling System.scheduleBatch.

And goes on to show the example doing:

String cronID = System.scheduleBatch(reassign, 'job example', 1);

Further, the KA for running a schedulable once only uses an example of 10 minutes.

It seems clear to me that if any async process is requested before the required execution time then that async processing request must be fulfilled, even if resources only become available some time after that process was intended to execute. To simply omit the execution because it took time to get round to looking at the execution requests is a massive hole in the entire async processing infrastructure and basically says that execution is not guaranteed. That, in turn, means that implementers cannot be sure that any timed processing request will be fulfilled, potentially completely breaking the processing chain in an org.

The fob off about writing our own re-try mechanism is somewhat ludicrous; to do so implies having another async job just to see if other jobs are working (which itself would need to run less frequently than the "recommended minimum" 15 minutes), but worse we would have to engineer our own means to track the state for the async processing so we can use that same state in the re-scheduled execution. This is a potentially huge undertaking.

To me this is a bug in the infrastructure and really needs to be resolved. We'll carry on pushing to have it viewed in this way instead of some acceptable edge case (which it isn't).

11
  • +1 for this is nuts and The fob off about writing our own re-try mechanism is somewhat ludicrous – cropredy Nov 16 '20 at 2:18
  • Appreciate you effort on this detailed and useful post. I agree this really is a problem in the platform and developers shouldn't ideally have to write all the additional code to get around this problem. By any chance, is there a known issue created and exposed via public URL by Salesforce? – arut Nov 16 '20 at 5:50
  • Not yet that I am aware of. – Phil W Nov 16 '20 at 7:11
  • 1
    DONE; via Twitter – cropredy Nov 27 '20 at 15:23
  • 1
    Ah, thank you, I misunderstood the context. I'll take a peek on Monday and see if I can draw any attention to that issue. – David Reed Nov 27 '20 at 15:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.