What causes REQUEST_RUNNING_TOO_LONG test errors?

Question

For most of our clients we've got a basic ant script that does a test deploy every night to make sure all the tests pass.

Occassionally, we'll get a REQUEST_RUNNING_TOO_LONG error. What's strange is that the time stamps it print don't seem like something that would generate an error. Take the output below for example, the deployment had only been running for 11 minutes (they typically take about an hour), and the test it had to stop had only been processing for 1s. What gives? Any idea what's causing this error?

BUILD FAILED
/home/ec2-user/ant/auto_repos/example/build.xml:39: Failed to process the request successfully. Cause(UNKNOWN_EXCEPTION): Running test: Example, time spent on that test: 1236ms,  total time spent running tests: 646005ms. Error: REQUEST_RUNNING_TOO_LONG: Your request was running for too long, and has been stopped.

Total time: 11 minutes 8 seconds

Additionally, with winter '16 these is happening pretty frequently, probably around 15% of test runs. With the automated test run it's not such a big deal, but waiting an hour doing a deployment to get this error is pretty frustrating (happened three times in a row last night).

Does anyone have info on why this is happening? Is this just a fact of life of the platform?

Answer 1

Each Apex transaction has a Maximum execution time of 10 minutes which is distinct from the Maximum CPU time per transaction on the Salesforce servers of 10,000 milliseconds for synchronous code and 60,000 milliseconds for asynchronous code. The 10 minute time limit exists to help prevent a thread from "hogging" server resources because the of the below limitation on what's counted towards CPU usage limits:

Operations that don’t consume application server CPU time aren’t counted toward CPU time. For example, the portion of execution time spent in the database for DML, SOQL, and SOSL isn’t counted, nor is waiting time for Apex callouts.

Note that the Maximum SOQL query run time before Salesforce cancels the transaction is 120 seconds.

One other possibility that could be impacting your tests is related to batch classes. Although only a single batch of 200 records are tested in a batch class, the production limits would still seem to apply. There are limits on the number of attempts and the time it takes to run the query for the batch class as follows: 15 attempts to process the query are allowed with a two minute limit on the time to process the query. There's a 5 min limit to process 100 records and a 10 min limit to process a batch before it's returned into the queue for later reprocessing. If it's returned more than 10 times, the batch is marked as failed. Max query retrieval size is 1Gb.

I wouldn't normally expect to see any of the above happening in a test class, but if a server is running especially slow, any of the above possibilities for either a batch or the execution of long transaction would seem possible. More than anything, it sounds as though the limit on the Maximum execution time of 10 minutes per transaction is what's causing your problem.

See the documentation for more on Per-Transaction Apex Limits.