What qualifies as running a test synchronously?

Question

This Limits Documentation mentions:

Total number of test classes that can be queued per a 24-hour period (5) - The greater of 500 or 10 multiplied by the number of test classes in the organization

with the caveat that

This limit applies to tests running asynchronously. This includes tests started through the Salesforce user interface including the Developer Console or by inserting ApexTestQueueItem objects using SOAP API.

1. What qualifies as running a test synchronously?
2. How can I ensure that my team members working concurrently in a sandbox do not exceed the AsyncApexTests Limit while building, refining and executing unit tests? (This quickly brings test development to a grinding halt.)
3. Does running tests via the Eclipse "run tests" menu option kick off an async or synchronous test run?

---

An update that is a bit like an answer:

If you need to run a unit test synchronously, you can do so in the developer console's run tests UI by selecting just a single class to run. Selecting two or more classes will execute them asynchronously.

This makes receiving the AsyncApexTests Limit exceeded exception a little bit more bearable and less blocking.

The Developer Console runs tests asynchronously in the background, unless your test run includes only one class and you’ve not chosen Always Run Asynchronously in the Test menu.

Source

Answer 1

Ive a few references for you Mark.

From here

Tests started using Run All Tests or Run Test no longer run synchronously. When you run tests by clicking Your Name > Setup > Develop > Apex Classes, and then either click Run All Tests from the Apex Classes page or Run Test from a class page, your test results are displayed in the Apex Test Execution page and the Developer Console This means that you won’t be blocked by the test run; you can do other work in the Salesforce user interface and you can access the Apex Test Execution page at a later time to verify if the tests have completed execution.

All tests started through the Salesforce user interface now run asynchronously. Tests that are run as part of a deployment, a package install, or a package upload still run synchronously.

From here:

You can use the runTests() call from the SOAP API to run tests synchronously:

RunTestsResult[] runTests(RunTestsRequest ri)

This call allows you to run all tests in all classes, all tests in a specific namespace, or all tests in a subset of classes in a specific namespace, as specified in the RunTestsRequest object.

Conclusion:

Based on the links above, tests can be run from the following places:

• The Salesforce user interface - asynchronously
• The Force.com IDE - synchronously
• The Force.com Developer Console - asynchronously
• The API - synchronously
• As part of a deployment, a package install, or a package upload - synchronously
• By inserting records into the ApexTestQueueItem table - asynchronously

Answer 2

Originally unit tests in Salesforce only ran synchronously. In summer '11 asynchronous unit tests were released with the primary benefit being that they don't block development; you don't need to wait for them to run, and they dont result in the org administration locked error.

There are many ways to run unit tests, and over time it seems Salesforce is trying to eliminate the synchronous method of running unit tests entirely.

What qualifies as running a test synchronously?

There are API calls that run unit tests both synchronously and asynchronously. This documentation explains how to do it both ways.

Unit tests that run as a result of the metadata api (e.g. deploys from Force.com IDE & Ant Migration Toolkit) run synchronously. This can be proven because the results do not appear in the Apex Test Execution page within Salesforce. Compare this with unit tests run via the developer console, where the Apex Test Execution screen does display these results. Furthermore, all asynchronous unit tests results appear on this screen, as it appears to be baked into the API itself.

How can I ensure that my team members working concurrently in a sandbox do not exceed the AsyncApexTests Limit while building, refining and executing unit tests? (This quickly brings test development to a grinding halt.)

The limit seems fairly high, personally I have never hit it. I would imagine that you could avoid hitting it by avoiding use of functionality that runs all unit tests, and instead focus on running only the relevent unit tests. While obviously a stupid solution, this is the type of limit you can hack -- just add 100 dummy classes marked as @isTest.

Alternatively you can programatically determine how many unit tests have ran in a day and build some type of monitor to warn the developers when they are close to hitting the limits. Query the Status (Completed, Failed) and CreatedDate fields of the ApexTestQueueItem sObject to get insight into how many unit tests have ran in a day. You could even use the CreatedBy field to build functionality that blames developers who are abusing the unit test execution functionality.

Does running tests via the Eclipse "run tests" menu option kick off an async or synchronous test run?

It does a synchronous run. This can be determined because the results do not appear in the Apex Test Execution screen and the Force.com IDE returns code coverage results line by line telling you what is covered, and what is not covered. The latter information is currently not available when unit tests are ran asynchronously; at least not via publicly facing API as far as I know.

Answer 3

When packaging you sit and wait for the tests to run, and it shows progress as they go—that could be an example of them running synchronously, and perhaps when they're run on install as that process can take a long time with no output which would indicate a blocking action.

Apart from those scenarios it seems to me that by their very nature (with respect to data visibility etc.) they'd always be run in a separate thread, and therefore, asynchronously.