It seems that, whilst there is some sympathy from within Salesforce, the Salesforce PM and R&D team responsible believe this is a "feature enhancement", not the reporting of a bug, because they "already did the analysis" (some 6 years ago).
Their main argument is that "there is an existing backlog of features, fixes and improvements that each item needs to be weighed up against and, in this particular case, the view is that there is a workaround. What's more, a change of this nature requires a level of due diligence to avoid a knock-on impact to our customers and to our own platform."
My response to that latter point (the "knock-on impact") is that, I think it fair to say, no one could possibly be relying on the system failing to honour an at-the-time future scheduling request. Also, given this problem infrequently happens (we've seen it for one customer on one [production] org, albeit for 24hrs, but only immediately after the Winter '21 upgrade), the actual delta in processing load on the Salesforce Platform for honouring such requests is negligible. The true knock-on here, for me, is that a fix would reinforce trust in the Salesforce Platform, unlike the current behaviour which corrosively damages that trust.
The view that there is a workaround is, for us, unfortunately naïve; our customers rely on relatively short delays in the data processing and to extend this from one or two minutes to a minimum of 15 isn't tenable for them.
The journey continues, however, and we will continue to discuss this situation with Salesforce in the new year. I'll post another update then.
Long story short, Salesforce support asked me to raise an "idea" about this bug to gauge interest in a fix. Please please please upvote it!
Having followed this up with Salesforce support, we finally got the following response after several weeks going round and round with them:
We use Quartz (http://www.quartz-scheduler.org/) for this piece of functionality. If the "system.schedule" is called at 2:13:17 (next to Fire Time) and takes more time to call Quartz, due to resource unavailability, then the time specified in the schedule will be in the past, and Quartz will never fire the trigger.
The resource availability dependency is the CRUX of Async processing. Thus, if due to reasons, the server gets busy and resource is not available, and it surpasses the NextFireTime of the crontrigger, it will cease the execution of scheduled job and status will be set to 'Waiting'.
As a long-term solution and avoid the late realization of schedule jobs ceasing, we suggest you to ensure the exception handling(if job ceases due to an exception) and retry mechanism (checking the Cron job status and rescheduling).
Also it is recommended to have a difference of at least 15 minutes for each execution.
I don't know about you, but to me this is nuts.
The official documentation doesn't mention any recommendation or requirement to use a minimum offset in the future of 15 minutes. Indeed, the documentation for batch apex and specifically use of
This example schedules a batch job to run one minute from now by calling System.scheduleBatch.
And goes on to show the example doing:
String cronID = System.scheduleBatch(reassign, 'job example', 1);
Further, the KA for running a schedulable once only uses an example of 10 minutes.
It seems clear to me that if any async process is requested before the required execution time then that async processing request must be fulfilled, even if resources only become available some time after that process was intended to execute. To simply omit the execution because it took time to get round to looking at the execution requests is a massive hole in the entire async processing infrastructure and basically says that execution is not guaranteed. That, in turn, means that implementers cannot be sure that any timed processing request will be fulfilled, potentially completely breaking the processing chain in an org.
The fob off about writing our own re-try mechanism is somewhat ludicrous; to do so implies having another async job just to see if other jobs are working (which itself would need to run less frequently than the "recommended minimum" 15 minutes), but worse we would have to engineer our own means to track the state for the async processing so we can use that same state in the re-scheduled execution. This is a potentially huge undertaking.
To me this is a bug in the infrastructure and really needs to be resolved. We'll carry on pushing to have it viewed in this way instead of some acceptable edge case (which it isn't).