Strange test class behavior during validate [closed]

Question

I had issue with deploying some changes to prod. First please note that this contain many changes in some triggers/classes, therefore don't sure if it'll help to post here all the relevant code and debug it, instead I'm trying to find general idea for the source issue.

I upload change set and when trying to validate it, received test class failure. Since the failure couldn't reproduce in sandbox, I searched for setup discrepancies, dependency in deploying user/profile, but found nothing.

Finally, I created new refreshed sandbox to make sure it have the exact same setup as prod, submitted the change set to the new sandbox. During validate I received the same error as in production, so I deployed to this new sandbox without running tests. Then I tried to run the failing test class from console, and surprisingly, it end up without any error.

Anyone have idea what can cause it? Does the test behave different during validation vs running from console? Note, that I disable parallel test execution.

Update: I rolled back the changes manually in the new sandbox. Upload again the change set, and check the following validate options:

1. When validate by Running all local test - the problematic test is failing
2. when validate by running only this test it is pass (failed due to coverage but no error in the test itself)
3. When validate by running the failing test and other random test class - the problematic test is failing.

What am I missing? Still, I disable parallel test execution

---

The error is:

REQUIRED_FIELD_MISSING, Required fields are missing: [QuoteId]: [QuoteId].

Seems the code try to sync between opportunity to quote but failed, therefore later it try to create quoteLineItem without quoteId.... I tried to focus more on understanding how it all it's possible that it was failing during validation to the new sandbox, but now it's working when running from console

Answer 1

After lots of investigation I finally got workaround for the issue, and it seems like bug\issue in SF.

I have trigger that querying dynamically all fields from Quote, it uses other class to get the all fields using describe method. The trigger was failing due to exception: "System.QueryException: No such column 'billingaddress' on entity 'Quote'. " (later also shippingaddress/additionaladdress/quotetoaddress)

Those fields available in soql query from API 30+, while the trigger and class that uses it are API 18.

Currently I added code inside the quote to exclude those fields while constructing the soql.

What is still not clear, is why it happen only when running several tests? I add debug log while running validate process: -If validate only the problematic test it is not failing as the describe method doesn't return the quote *address fields. -If running several tests during validate (while the first test class is API 30+), the describe does return those fields and they causing exception in the problematic test. -If running several tests during validate (but the first test is API 18), the describe method does not return those fields, and it's working.

Can only estimate that due to validate implementation by SF, the class that use the describe run in higher API version, probably as it being used by other test class with higher version. Sounds weird, as any test should run independently, but I cannot find other explanation.

UPDATE I was able to deploy the changes to prod. After deployment I debug the test run, and strangely it's not retrieving the addresses fields from Quote. I found following documentation in SF

https://developer.salesforce.com/docs/atlas.en-us.apexcode.meta/apexcode/apex_methods_system_sobject_describe.htm

When you describe sObjects and their fields from within an Apex class, 
custom fields of new field types are returned regardless of the API 
version that the class is saved in. If a field type, such as the 
geolocation field type, is available only in a recent API version, 
components of a geolocation field are returned even if the class is saved 
in an earlier API version.

Per this document, I understand that the address fields should be retrieved, regardless the API version used in the class, however I cannot notice this behavior. It was happen only in specific cases. Weird....