The exact difference would be hard to cover without internal knowledge of how Salesforce is implemented.
The general difference that we can observe is the API endpoint used to start the test run. There are several ways to run tests, including:
/_ui/common/apex/test/ApexTestQueueServlet
used by the setup UI with action=ENQUEUE
.
- The REST Tooling API
/runTestsAsynchronous/
resource
- The REST Tooling API
/runTestsSynchronous/
resource
- The SOAP Tooling API runTests() method
- The SOAP Tooling API runTestsAsynchronous method
- The ApexTestQueueItem sObject that can be created via DML
- The older Apex SOAP API runTests()
- The Metadata API as part of a deployment
- Internally when creating a managed package
- When validating or installing a change set
"In theory, there is no difference between theory and practice. But, in practice, there is."
- Jan L. A. van de Snepscheut
Should all the APIs produce consistent test runs? Yes, in theory they should.
In practice you can see variations all over the place depending on the API you use and if you choose synchronous or asynchronous testing.
Also, just about all those options above can also be tied to a specific API version. Changing the API version may affect the outcome. I've seen examples recently where the Spring '17 version of the developer console was still using some v37 APIs rather than the expected v39.
A number of the APIs also access additional parameters that could affect the test execution. Such as the headers on the SOAP calls.
Various issues occur with parallel testing that can occur with asynchronous test execution. This typically revolved around older test methods or those using @SeeAllData=true
. See Apex Test cases fail when run in parallel
I've had problems in the past where record ID's weren't resetting between test runs - The mystery of the nondeterministic Salesforce test case
When you open the Developer Console it creates a TraceFlag that will start log capturing. This won't occur via the Setup UI. Having the debug log active, particularly towards the finer logging levels can change the way code executes.
In summary, there are many differences that could subtly change the outcome of the test case. They only way to really tell what is going on is to isolate the cause in your specific case.