17

Problem

Salesforce is a multi-tenant platform and as a consequence the time for an asynchronous request to be dequeued may vary. This is not a problem for us, as long as the delays stay within reasonable limits. Reasonable limits are a relative notion of course, but we regularly experience execution delays that are more than 10 times longer than usual. These inter-request delays lead to a pileup of tasks that are time critical, like onboarding customers and requests for time-critical information.

Analysis

The graph below depicts the number of pending asynchronous batch jobsover time in our org. Number of processed async jobs over time

As you can see there is a large variation in the speed at which jobs are processed. The sharp spikes correspond to 'normal' execution times during which the delay between job items isn't longer than a few seconds. However, there are also periods, for example between 2017-12-02 and 2017-12-03 when performance dramatically decreased. Further inspection showed that during such periods, a batch of 50 jobs would take 82 minutes to be processed, of which 76 minutes are spent waiting to be dequeued.

These periods increased latency in the dequeuing of asynchronous jobs are a problematic for us for two reasons:

  1. They can last for a long time. The longest period we have observed now is more than two days. A delay of an hour in onboarding a customer is not problematic for our business, but a delay of two days is!
  2. They come without warning. If we would know in advance that a period period of slowness is to be expected, we could potentially reschedule some of our less time-critical activities.

Questions

As our company grows, the asynchronous processes are expected increase both in number and in volume. This makes us wonder if SF is a reliable platform for the development of our business processes. To make an informed decision we would need an answer to the following questions:

  1. Is our situation an anomaly, or are other orgs experiencing similar problems?
  2. Can anything be done reduce the frequency or duration of periods with dramatically increased dequeue latency?
  3. Is it posible to be notified about such periods in advance, or at least when they are happening?
  • 1
    Can you tell us more about how these jobs are being queued? Is this primarily related to one object or jobs being fired using a pattern like an @future method from a trigger (or other pattern) or are these "all your jobs" that are scheduled & executed via any means? Does it included queueables? Do they schedule themselves? Anything that narrows the scope of your question would be helpful. – crmprogdev Mar 27 '17 at 15:34
  • Thanks for your helping out @crmprogdev! All our jobs are represented by an Event custom object. We schedule a task that checks if there are new events to be processed every minute. These jobs are Database.Batchable Apex jobs that are run through Database.executeBatch. Each job is some Apex code that performs a task. An example of such a task is a bimonthly report that we need to send to all our customers. This entails: fetching info from our ERP through a call-out, sending an email and logging the action in a case. Hope this helps! – Sam Theisens Mar 27 '17 at 20:47
  • The graph is a bit misleading as it is not displaying your stated issue. It shows the number of items in queue and could simply be spiking due to items added to the queue at that time. While the queue time may play a factor, why not show ad the avg execution time as a third axis and it would make the graph more meaningful. - I would also not that deals of 2 days would NOT be considered normal in any case. What Pod are you on? NA35? Have you reached out to SF support? Also, I know SF can place specific orgs at higher priority but I have only seen it done for some large well paying clients – Eric Apr 28 '17 at 14:49
12

When you say "schedule a task", I'm going to assume you're talking about a batch class that performs work. If that's the case, I believe your issue is related to how asynchronous processing works on the platform overall when you have to share resources with other orgs on the same pod.

During times of heavy load, when thresholds are exceeded for server memory or CPU usage on the pod, asynchronous processing is reduced in favor of synchronous requests to ensure the latter are being supported for customer responsiveness. When too many asynchronous calls are being requested at once from a single org on a pod, those jobs can wind up experiencing multiple extended delays. I believe that is what you're seeing happening to your org which is a result of the pattern you're using.

Below is a lengthy explanation taken from excerpts of the documentation, Asynchronous Processing in Force.com:

Asynchronous processing, in a multi-tenant environment, presents some challenges:

  • Ensure fairness of processing – Make sure every customer gets a fair chance at processing resources in a multi-tenant architecture.
  • Ensure fault tolerance - Make sure no asynchronous requests are lost due to equipment or software failures.

Here's a high level overview:

high level overview of Force.com's asynchronous processing technology

Salesforce uses an asynchronous queue to manage asynchronous requests from multiple organizations within each instance. Here's an example to show what it looks like:

Asynchronous Queue

Notice that the queue contains requests from multiple organizations. Each request can be associated with a job that could vary in complexity and running time. In the diagram, longer running jobs are represented with larger boxes.

As one request is completed, another request is removed from the queue and processed. Error handling and failure recovery is built in (via request persistence) so the requests are not lost if a queue failure or handler failure occurs.


Fair Request Handling

...a single organization could queue 250,000 @future Apex requests in a 24-hour period, depending on Salesforce license type. If one organization adds a large number of requests to the queue, it could prevent other customers from getting access to the worker threads. To avoid this, the queuing framework implements flow control which prevents a single customer from using all of the available threads.

the queuing framework will determine if the maximum number of worker threads (as determined by the handler) is being used by a single organization. If so, the framework will "peek" into the queue to see if other organizations have requests waiting. The set of requests is called the peek set and is limited to a fixed number of requests at the front of the queue (currently set at 2,000 requests).

For this example, assume that a maximum of 12 threads can process requests from a single organization, and that our peek set size is 15 requests.

  • Assume organization 1 creates 13 @future requests.
  • Organization 2 adds two @future requests to the queue.
  • Then, two more organization 1 @future requests are en-queued.

If 13 total threads are available and no other requests are being processed for organization 1 or organization 2, the processing will be as follows:

  1. 12 threads will take the first 12 requests from organization 1.
  2. The 13th thread will not process a request from organization 1 although it is the next one in the queue. This is because organization 1 has taken its allotted amount of threads. This request will remain in the queue at its current position until one of the 12 threads becomes available. This request is delayed.
  3. The framework will scan for requests from other organizations within the peek set of 15 requests. It will find the first @future request from organization 2 and begin processing this request, skipping the 13th request for organization 1.

Here's how it looks graphically:

enter image description here

What happens when requests for a particular organization occupy the entire peek set when the queue is scanned in step 3 above?

This time, organization 1 has 15 requests remaining in the queue and organization 2 has two requests in the queue as shown in this diagram:

Extended delay

Since all of the requests in the peek set are from a single organization (organization 1), those 15 requests will be moved to the back of the queue with a specific delay. This is called an extended delay.

The delay is different for each message. For example, for @future requests, the delay is 5 minutes. That means a minimum of 5 minutes must elapse before those requests are eligible for processing.

When delayed requests become eligible for processing, it's possible for these requests to be acted upon by flow control and again get moved to the back of the queue and delayed.


...the queuing framework will monitor system resources such as server memory and CPU usage and reduce asynchronous processing when thresholds are exceeded. If necessary, under heavy load, Salesforce will delay long running jobs in the queue to give resource priority to synchronous requests.


Best Practices for Asynchronous Apex

Apex supports batch Apex and @future Apex methods. Both of these features add requests to the asynchronous queue. Keep the following best practices in mind when planning out development work that will use asynchronous Apex.

Future Apex

Every @future invocation adds one request to the asynchronous queue. Design patterns that would add large numbers of @future requests over a short period of time should be avoided unless absolutely needed. Best practices include:

  • Avoid adding large numbers of @future methods to the asynchronous queue, if possible. If more than 2,000 unprocessed requests from a single organization are in the queue, any additional requests from the same organization will be delayed while the queue handles requests from other organizations.

  • Ensure that the @future requests execute as fast as possible. To ensure fast execution of batch jobs, minimize Web service call out times and tune queries used in your @future methods. The longer the @future method executes, the more likely other queued requests are delayed when there are a large number of requests in the queue.

  • Test your @future methods at scale. Where possible, test using an environment that generates the maximum number of @future methods you’d expect to handle. This will help determine if delays will occur.

  • Consider using batch Apex instead of @future methods to process large number of records asynchronously. This will be more efficient then creating a @future request for each record.

Extended delay time is 5 minutes.


Edit: in response to comments...

Summary

As you'll below, there's a LOT of additional information that applies to this subject which can impact your comments. Of most importance is that while there's a Flex Queue that can be enabled, there's also the original Batch Queue. Any @future job is an asynchronous job and ultimately gets processed through the Flex or Batch Queue into the Pod's Queue. That Queue monitors the load on the Pod as a whole and the sequential requests from each org.

Your requests will get processed by your org's queue into the Pod's queue. In times of heavy load in your org or on the pod, you may observe that they continue to be processed by your queue without results being returned. This can happen to the point where the number of your org's submitted jobs can exceed 2000. This has NOTHING to do with the size of your Flex Queue. The Flex Queue size limit is only for jobs with a status of "hold".

Once jobs are submitted into the Pod's queue, they may get delayed numerous times (even before reaching the 2000 jobs mark - see original explanation above). You'll have no control over that. The best thing you can do for yourself is to minimize the number and frequency of your asynchronous jobs while optimizing them to run as fast as possible.

See supporting reference material below, including Batch Apex Best Practices.


The following comes from the current Apex Governor Limits in the Salesforce Developer Limits Quick Reference.

  • The maximum number of asynchronous Apex method executions (batch Apex, future methods, Queueable Apex, and scheduled Apex) per a 24-hour period(1)

    • 250,000 or the number of user licenses in your org multiplied by 200, whichever is greater
  • Number of synchronous concurrent requests for long-running requests that last longer than 5 seconds for each org.(2) : 10

  • Maximum number of Apex classes scheduled concurrently: 100
  • Maximum number of batch Apex jobs in the Apex flex queue that are in Holding status: 100
  • Maximum number of batch Apex jobs queued or active concurrently(3): 5
  • Maximum number of batch Apex job start method concurrent executions(4): 1
  • Maximum number of batch jobs that can be submitted in a running test: 5
  • Maximum number of test classes that can be queued per 24-hour period (production orgs other than Developer Edition)(5)

    • The greater of 500 or 10 multiplied by the number of test classes in the org
  • Maximum number of query cursors open concurrently per user(6): 50

  • Maximum number of query cursors open concurrently per user for the Batch Apex start method: 15
  • Maximum number of query cursors open concurrently per user for the Batch Apex execute and finish methods: 5

Footnotes

(1) For Batch Apex, method executions include executions of the start, execute, and finish methods. This limit is for your entire org and is shared with all asynchronous Apex: Batch Apex, Queueable Apex, scheduled Apex, and future methods. To check how many asynchronous Apex executions are available, make a request to the REST API limits resource. See List Organization Limits in the Force.com REST API Developer Guide. The licenses that count toward this limit are full Salesforce user licenses or Force.com App Subscription user licenses. Chatter Free, Chatter customer users, Customer Portal User, and partner portal User licenses aren’t included.

(2) If more requests are made while the 10 long-running requests are still running, they’re denied.

(3) When batch jobs are submitted, they’re held in the flex queue before the system queues them for processing.

(4) Batch jobs that haven’t started yet remain in the queue until they’re started. If more than one job is running, this limit doesn’t cause any batch job to fail and execute methods of batch Apex jobs still run in parallel.

(5) This limit applies to tests running asynchronously. This group of tests includes tests started through the Salesforce user interface including the Developer Console or by inserting ApexTestQueueItem objects using SOAP API.

(6) For example, if 50 cursors are open and a client application still logged in as the same user attempts to open a new one, the oldest of the 50 cursors is released. Cursor limits for different Force.com features are tracked separately. For example, you can have 50 Apex query cursors, 15 cursors for the Batch Apex start method, 5 cursors each for the Batch Apex execute and finish methods, and 5 Visualforce cursors open at the same time.


Additional material from Using Batch Apex in the Apex Developer's Guide:

Each execution of a batch Apex job is considered a discrete transaction. For example, a batch Apex job that contains 1,000 records and is executed without the optional scope parameter from Database.executeBatch is considered five transactions of 200 records each. The Apex governor limits are reset for each transaction. If the first transaction succeeds but the second fails, the database updates made in the first transaction are not rolled back.


With the Apex flex queue, you can submit up to 100 batch jobs.

The outcome of Database.executeBatch is as follows.

  • The batch job is placed in the Apex flex queue, and its status is set to Holding.
  • If the Apex flex queue has the maximum number of 100 jobs, Database.executeBatch throws a LimitException and doesn’t add the job to the queue.

Note: If your org doesn’t have Apex flex queue enabled, Database.executeBatch adds the batch job to the batch job queue with the Queued status. If the concurrent limit of queued or active batch job has been reached, a LimitException is thrown, and the job isn’t queued.


Some things to note about System.scheduleBatch:

  • When you call System.scheduleBatch, Salesforce schedules the job for execution at the specified time. Actual execution occurs at or after that time, depending on service availability.
  • The scheduler runs as system—all classes are executed, whether or not the user has permission to execute the class.
  • When the job’s schedule is triggered, the system queues the batch job for processing. If Apex flex queue is enabled in your org, the batch job is added at the end of the flex queue. For more information, see Holding Batch Jobs in the Apex Flex Queue.
  • All scheduled Apex limits apply for batch jobs scheduled using System.scheduleBatch. After the batch job is queued (with a status of Holding or Queued), all batch job limits apply and the job no longer counts toward scheduled Apex limits.
  • After calling this method and before the batch job starts, you can use the returned scheduled job ID to abort the scheduled job using the System.abortJob method.

Batch Apex Best Practices

  • Use extreme care if you are planning to invoke a batch job from a trigger. You must be able to guarantee that the trigger will not add more batch jobs than the limit. In particular, consider API bulk updates, import wizards, mass record changes through the user interface, and all cases where more than one record can be updated at a time.
  • When you call Database.executeBatch, Salesforce only places the job in the queue. Actual execution can be delayed based on service availability.
  • When testing your batch Apex, you can test only one execution of the execute method. Use the scope parameter of the executeBatch method to limit the number of records passed into the execute method to ensure that you aren’t running into governor limits.
  • The executeBatch method starts an asynchronous process. When you test batch Apex, make certain that the asynchronously processed batch job is finished before testing against the results. Use the Test methods startTest and stopTest around the executeBatch method to ensure that it finishes before continuing your test.
  • Use Database.Stateful with the class definition if you want to share instance member variables or data across job transactions. Otherwise, all member variables are reset to their initial state at the start of each transaction. Methods declared as future aren’t allowed in classes that implement the Database.Batchable interface.
  • Methods declared as future can’t be called from a batch Apex class. When a batch Apex job is run, email notifications are sent to the user who submitted the batch job. If the code is included in a managed package and the subscribing org is running the batch job, notifications are sent to the recipient listed in the Apex Exception Notification Recipient field. Each method execution uses the standard governor limits anonymous block, Visualforce controller, or WSDL method.
  • Each batch Apex invocation creates an AsyncApexJob record. To construct a SOQL query to retrieve the job’s status, number of errors, progress, and submitter, use the AsyncApexJob record’s ID. For more information about the AsyncApexJob object, see AsyncApexJob in the Object Reference for Salesforce and Force.com.
  • For each 10,000 AsyncApexJob records, Apex creates an AsyncApexJob record of type BatchApexWorker for internal use. When querying for all AsyncApexJob records, we recommend that you filter out records of type BatchApexWorker using the JobType field. Otherwise, the query returns one more record for every 10,000 AsyncApexJob records. For more information about the AsyncApexJob object, see AsyncApexJob in the Object Reference for Salesforce and Force.com.
  • All methods in the class must be defined as global or public. For a sharing recalculation, we recommend that the execute method delete and then re-create all Apex managed sharing for the records in the batch. This process ensures that sharing is accurate and complete.
  • Batch jobs queued before a Salesforce service maintenance downtime remain in the queue. After service downtime ends and when system resources become available, the queued batch jobs are executed. If a batch job was running when downtime occurred, the batch execution is rolled back and restarted after the service comes back up.
  • Minimize the number of batches, if possible. Salesforce uses a queue-based framework to handle asynchronous processes from such sources as future methods and batch Apex. This queue is used to balance request workload across organizations. If more than 2,000 unprocessed requests from a single organization are in the queue, any additional requests from the same organization will be delayed while the queue handles requests from other organizations.
  • Ensure that batch jobs execute as fast as possible. To ensure fast execution of batch jobs, minimize Web service callout times and tune queries used in your batch Apex code. The longer the batch job executes, the more likely other queued jobs are delayed when many jobs are in the queue.

  • Thanks for that answer. Do you have any more details on how this works with Batch jobs? In the queue graph, is a batch job one 'block' in the queue, or is every single execute() a block in the queue? I would say the first, but that would imply that if Sam is using only Batch, it would never hit Extended Delay. Even more so because the documentation mentions that "Extended delay is not applicable to batch Apex." which is probably due to the same reason: it is impossible to queue 2000 Batch jobs anyway :-) – Willem Mulder Apr 26 '17 at 8:51
  • 1
    Every batch job would be a block, not an execute of a batch job. Don't forget that batch jobs can be queued programmatically from a command line or from a script along with being the consequence of many different kinds of data load/ETL jobs. I could think of other reasons too in addition to how they can be 'triggered' from @future methods. – crmprogdev Apr 26 '17 at 23:12
  • Yes but there is a hard limit of 200 Batch jobs being in the queue, I believe. So that would mean that Batch jobs alone would not be able to trigger Extended Delay. Of course, the delay could be relevant in combination with other async mechanisms, like queueing 2000 @future jobs icw a few batch jobs. But even then the documentation mentions that "Extended delay is not applicable to batch Apex." so that's probably not the problem that Sam is encountering? – Willem Mulder Apr 28 '17 at 8:41
  • 1
    The limit you speak of is for jobs with a status of "hold" and is actually 100. See my edited answer above. – crmprogdev Apr 28 '17 at 13:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.