What are the exact differences between running a unit test in the developer console and in setup?

Question

You can run unit tests in the developer console, and you can run them in Setup under Apex Test Execution. (And other places I don't want discuss here.)

For the second time now, we had a unit test that consistently passed when run in setup, but failed in the developer console.

What is the difference between running a unit test in the console vs. in setup?

I could not find any documentation on this. Don't know if this is relevant, but we have disabled parallel test execution ("Disable Parallel Apex Testing").

Answer 1

The exact difference would be hard to cover without internal knowledge of how Salesforce is implemented.

The general difference that we can observe is the API endpoint used to start the test run. There are several ways to run tests, including:

1. /_ui/common/apex/test/ApexTestQueueServlet used by the setup UI with action=ENQUEUE.
2. The REST Tooling API /runTestsAsynchronous/ resource
3. The REST Tooling API /runTestsSynchronous/ resource
4. The SOAP Tooling API runTests() method
5. The SOAP Tooling API runTestsAsynchronous method
6. The ApexTestQueueItem sObject that can be created via DML
7. The older Apex SOAP API runTests()
8. The Metadata API as part of a deployment
9. Internally when creating a managed package
10. When validating or installing a change set

"In theory, there is no difference between theory and practice. But, in practice, there is."
- Jan L. A. van de Snepscheut

Should all the APIs produce consistent test runs? Yes, in theory they should.

In practice you can see variations all over the place depending on the API you use and if you choose synchronous or asynchronous testing.

Also, just about all those options above can also be tied to a specific API version. Changing the API version may affect the outcome. I've seen examples recently where the Spring '17 version of the developer console was still using some v37 APIs rather than the expected v39.

A number of the APIs also access additional parameters that could affect the test execution. Such as the headers on the SOAP calls.

Various issues occur with parallel testing that can occur with asynchronous test execution. This typically revolved around older test methods or those using @SeeAllData=true. See Apex Test cases fail when run in parallel

I've had problems in the past where record ID's weren't resetting between test runs - The mystery of the nondeterministic Salesforce test case

When you open the Developer Console it creates a TraceFlag that will start log capturing. This won't occur via the Setup UI. Having the debug log active, particularly towards the finer logging levels can change the way code executes.

---

In summary, there are many differences that could subtly change the outcome of the test case. They only way to really tell what is going on is to isolate the cause in your specific case.

Answer 2

Make sure you have Always Run Asynchronously selected under the Test menu in the developer console.

If you run a test from setup it defaults to the asynchronous flow in the background. You have to manually set this in the developer console.