When you say "schedule a task", I'm going to assume you're talking about a batch class that performs work. If that's the case, I believe your issue is related to how asynchronous processing works on the platform overall when you have to share resources with other orgs on the same pod.
During times of heavy load, when thresholds are exceeded for server memory or CPU usage on the pod, asynchronous processing is reduced in favor of synchronous requests to ensure the latter are being supported for customer responsiveness. When too many asynchronous calls are being requested at once from a single org on a pod, those jobs can wind up experiencing multiple extended delays. I believe that is what you're seeing happening to your org which is a result of the pattern you're using.
Below is a lengthy explanation taken from excerpts of the documentation, Asynchronous Processing in Force.com:
Asynchronous processing, in a multi-tenant environment, presents some challenges:
- Ensure fairness of processing – Make sure every customer gets a fair chance at processing resources in a multi-tenant architecture.
- Ensure fault tolerance - Make sure no asynchronous requests are lost due to equipment or software failures.
Here's a high level overview:
Salesforce uses an asynchronous queue to manage asynchronous requests from multiple organizations within each instance. Here's an example to show what it looks like:
Notice that the queue contains requests from multiple organizations. Each request can be associated with a job that could vary in complexity and running time. In the diagram, longer running jobs are represented with larger boxes.
As one request is completed, another request is removed from the queue and processed. Error handling and failure recovery is built in (via request persistence) so the requests are not lost if a queue failure or handler failure occurs.
Fair Request Handling
...a single organization could queue 250,000 @future Apex requests in a 24-hour period, depending on Salesforce license type. If one organization adds a large number of requests to the queue, it could prevent other customers from getting access to the worker threads. To avoid this, the queuing framework implements flow control which prevents a single customer from using all of the available threads.
the queuing framework will determine if the maximum number of worker threads (as determined by the handler) is being used by a single organization. If so, the framework will "peek" into the queue to see if other organizations have requests waiting. The set of requests is called the peek set and is limited to a fixed number of requests at the front of the queue (currently set at 2,000 requests).
For this example, assume that a maximum of 12 threads can process requests from a single organization, and that our peek set size is 15 requests.
- Assume organization 1 creates 13 @future requests.
- Organization 2 adds two @future requests to the queue.
- Then, two more organization 1 @future requests are en-queued.
If 13 total threads are available and no other requests are being processed for organization 1 or organization 2, the processing will be as follows:
- 12 threads will take the first 12 requests from organization 1.
- The 13th thread will not process a request from organization 1 although it is the next one in the queue. This is because organization 1 has taken its allotted amount of threads. This request will remain in the queue at its current position until one of the 12 threads becomes available. This request is delayed.
- The framework will scan for requests from other organizations within the peek set of 15 requests. It will find the first @future request from organization 2 and begin processing this request, skipping the 13th request for organization 1.
Here's how it looks graphically:
What happens when requests for a particular organization occupy the entire peek set when the queue is scanned in step 3 above?
This time, organization 1 has 15 requests remaining in the queue and organization 2 has two requests in the queue as shown in this diagram:
Since all of the requests in the peek set are from a single organization (organization 1), those 15 requests will be moved to the back of the queue with a specific delay. This is called an extended delay.
The delay is different for each message. For example, for @future requests, the delay is 5 minutes. That means a minimum of 5 minutes must elapse before those requests are eligible for processing.
When delayed requests become eligible for processing, it's possible for these requests to be acted upon by flow control and again get moved to the back of the queue and delayed.
...the queuing framework will monitor system resources such as server memory and CPU usage and reduce asynchronous processing when thresholds are exceeded. If necessary, under heavy load, Salesforce will delay long running jobs in the queue to give resource priority to synchronous requests.
Best Practices for Asynchronous Apex
Apex supports batch Apex and @future Apex methods. Both of these features add requests to the asynchronous queue. Keep the following best practices in mind when planning out development work that will use asynchronous Apex.
Future Apex
Every @future invocation adds one request to the asynchronous queue. Design patterns that would add large numbers of @future requests over a short period of time should be avoided unless absolutely needed. Best practices include:
Avoid adding large numbers of @future methods to the asynchronous queue, if possible. If more than 2,000 unprocessed requests from a single organization are in the queue, any additional requests from the same organization will be delayed while the queue handles requests from other organizations.
Ensure that the @future requests execute as fast as possible. To ensure fast execution of batch jobs, minimize Web service call out times and tune queries used in your @future methods. The longer the @future method executes, the more likely other queued requests are delayed when there are a large number of requests in the queue.
Test your @future methods at scale. Where possible, test using an environment that generates the maximum number of @future methods you’d expect to handle. This will help determine if delays will occur.
Consider using batch Apex instead of @future methods to process large number of records asynchronously. This will be more efficient then creating a @future request for each record.
Extended delay time is 5 minutes.
Edit: in response to comments...
Summary
As you'll below, there's a LOT of additional information that applies to this subject which can impact your comments. Of most importance is that while there's a Flex Queue that can be enabled, there's also the original Batch Queue. Any @future job is an asynchronous job and ultimately gets processed through the Flex or Batch Queue into the Pod's Queue. That Queue monitors the load on the Pod as a whole and the sequential requests from each org.
Your requests will get processed by your org's queue into the Pod's queue. In times of heavy load in your org or on the pod, you may observe that they continue to be processed by your queue without results being returned. This can happen to the point where the number of your org's submitted jobs can exceed 2000. This has NOTHING to do with the size of your Flex Queue. The Flex Queue size limit is only for jobs with a status of "hold".
Once jobs are submitted into the Pod's queue, they may get delayed numerous times (even before reaching the 2000 jobs mark - see original explanation above). You'll have no control over that. The best thing you can do for yourself is to minimize the number and frequency of your asynchronous jobs while optimizing them to run as fast as possible.
See supporting reference material below, including Batch Apex Best Practices.
The following comes from the current Apex Governor Limits in the Salesforce Developer Limits Quick Reference.
The maximum number of asynchronous Apex method executions (batch Apex, future methods, Queueable Apex, and scheduled Apex) per a 24-hour period(1)
- 250,000 or the number of user licenses in your org multiplied by 200, whichever is greater
Number of synchronous concurrent requests for long-running requests that last longer than 5 seconds for each org.(2) : 10
- Maximum number of Apex classes scheduled concurrently: 100
- Maximum number of batch Apex jobs in the Apex flex queue that are in Holding status: 100
- Maximum number of batch Apex jobs queued or active concurrently(3): 5
- Maximum number of batch Apex job start method concurrent executions(4): 1
- Maximum number of batch jobs that can be submitted in a running test: 5
Maximum number of test classes that can be queued per 24-hour period (production orgs other than Developer Edition)(5)
- The greater of 500 or 10 multiplied by the number of test classes in the org
Maximum number of query cursors open concurrently per user(6): 50
- Maximum number of query cursors open concurrently per user for the Batch Apex start method: 15
- Maximum number of query cursors open concurrently per user for the Batch Apex execute and finish methods: 5
Footnotes
(1) For Batch Apex, method executions include executions of the start, execute, and finish methods. This limit is for your entire org and is shared with all asynchronous Apex: Batch Apex, Queueable Apex, scheduled Apex, and future methods. To check how many asynchronous Apex executions are available, make a request to the REST API limits resource. See List Organization Limits in the Force.com REST API Developer Guide. The licenses that count toward this limit are full Salesforce user licenses or Force.com App Subscription user licenses. Chatter Free, Chatter customer users, Customer Portal User, and partner portal User licenses aren’t included.
(2) If more requests are made while the 10 long-running requests are still running, they’re denied.
(3) When batch jobs are submitted, they’re held in the flex queue before the system queues them for processing.
(4) Batch jobs that haven’t started yet remain in the queue until they’re started. If more than one job is running, this limit doesn’t cause any batch job to fail and execute methods of batch Apex jobs still run in parallel.
(5) This limit applies to tests running asynchronously. This group of tests includes tests started through the Salesforce user interface including the Developer Console or by inserting ApexTestQueueItem objects using SOAP API.
(6) For example, if 50 cursors are open and a client application still logged in as the same user attempts to open a new one, the oldest of the 50 cursors is released. Cursor limits for different Force.com features are tracked separately. For example, you can have 50 Apex query cursors, 15 cursors for the Batch Apex start method, 5 cursors each for the Batch Apex execute and finish methods, and 5 Visualforce cursors open at the same time.
Additional material from Using Batch Apex in the Apex Developer's Guide:
Each execution of a batch Apex job is considered a discrete transaction. For example, a batch Apex job that contains 1,000 records and is executed without the optional scope parameter from Database.executeBatch is considered five transactions of 200 records each. The Apex governor limits are reset for each transaction. If the first transaction succeeds but the second fails, the database updates made in the first transaction are not rolled back.
With the Apex flex queue, you can submit up to 100 batch jobs.
The outcome of Database.executeBatch is as follows.
- The batch job is placed in the Apex flex queue, and its status is set to Holding.
- If the Apex flex queue has the maximum number of 100 jobs, Database.executeBatch throws a LimitException and doesn’t add the job to the queue.
Note: If your org doesn’t have Apex flex queue enabled, Database.executeBatch adds the batch job to the batch job queue with the Queued status. If the concurrent limit of queued or active batch job has been reached, a LimitException is thrown, and the job isn’t queued.
Some things to note about System.scheduleBatch:
- When you call System.scheduleBatch, Salesforce schedules the job for execution at the specified time. Actual execution occurs at or after that time, depending on service availability.
- The scheduler runs as system—all classes are executed, whether or not the user has permission to execute the class.
- When the job’s schedule is triggered, the system queues the batch job for processing. If Apex flex queue is enabled in your org, the batch job is added at the end of the flex queue. For more information, see Holding Batch Jobs in the Apex Flex Queue.
- All scheduled Apex limits apply for batch jobs scheduled using System.scheduleBatch. After the batch job is queued (with a status of Holding or Queued), all batch job limits apply and the job no longer counts toward scheduled Apex limits.
- After calling this method and before the batch job starts, you can use the returned scheduled job ID to abort the scheduled job using the System.abortJob method.
Batch Apex Best Practices
- Use extreme care if you are planning to invoke a batch job from a trigger. You must be able to guarantee that the trigger will not add more batch jobs than the limit. In particular, consider API bulk updates, import wizards, mass record changes through the user interface, and all cases where more than one record can be updated at a time.
- When you call Database.executeBatch, Salesforce only places the job in the queue. Actual execution can be delayed based on service availability.
- When testing your batch Apex, you can test only one execution of the execute method. Use the scope parameter of the executeBatch method to limit the number of records passed into the execute method to ensure that you aren’t running into governor limits.
- The executeBatch method starts an asynchronous process. When you test batch Apex, make certain that the asynchronously processed batch job is finished before testing against the results. Use the Test methods startTest and stopTest around the executeBatch method to ensure that it finishes before continuing your test.
- Use Database.Stateful with the class definition if you want to share instance member variables or data across job transactions. Otherwise, all member variables are reset to their initial state at the start of each transaction.
Methods declared as future aren’t allowed in classes that implement the Database.Batchable interface.
- Methods declared as future can’t be called from a batch Apex class.
When a batch Apex job is run, email notifications are sent to the user who submitted the batch job. If the code is included in a managed package and the subscribing org is running the batch job, notifications are sent to the recipient listed in the Apex Exception Notification Recipient field.
Each method execution uses the standard governor limits anonymous block, Visualforce controller, or WSDL method.
- Each batch Apex invocation creates an AsyncApexJob record. To construct a SOQL query to retrieve the job’s status, number of errors, progress, and submitter, use the AsyncApexJob record’s ID. For more information about the AsyncApexJob object, see AsyncApexJob in the Object Reference for Salesforce and Force.com.
- For each 10,000 AsyncApexJob records, Apex creates an AsyncApexJob record of type BatchApexWorker for internal use. When querying for all AsyncApexJob records, we recommend that you filter out records of type BatchApexWorker using the JobType field. Otherwise, the query returns one more record for every 10,000 AsyncApexJob records. For more information about the AsyncApexJob object, see AsyncApexJob in the Object Reference for Salesforce and Force.com.
- All methods in the class must be defined as global or public.
For a sharing recalculation, we recommend that the execute method delete and then re-create all Apex managed sharing for the records in the batch. This process ensures that sharing is accurate and complete.
- Batch jobs queued before a Salesforce service maintenance downtime remain in the queue. After service downtime ends and when system resources become available, the queued batch jobs are executed. If a batch job was running when downtime occurred, the batch execution is rolled back and restarted after the service comes back up.
- Minimize the number of batches, if possible. Salesforce uses a queue-based framework to handle asynchronous processes from such sources as future methods and batch Apex. This queue is used to balance request workload across organizations. If more than 2,000 unprocessed requests from a single organization are in the queue, any additional requests from the same organization will be delayed while the queue handles requests from other organizations.
- Ensure that batch jobs execute as fast as possible. To ensure fast execution of batch jobs, minimize Web service callout times and tune queries used in your batch Apex code. The longer the batch job executes, the more likely other queued jobs are delayed when many jobs are in the queue.
