Apex tests fail in Dev Console Test Suite but pass when run individually

Question

Background

• There is a test utility class

  • Class is marked as @isTest
  • There are 5 methods
  • Methods are not marked as @isTest because they have parameters
  • All methods are static
  • All methods call ConnectApi methods

• There are 14 test classes that use this utility class

• I did not write any of this code, so I know no other background and don't have experience running these tests

• UPDATE: Acting on the linked question and the comment from @DerekF, I verified that all the originating test methods which call these utility methods are using SeeAllData=true. And I clarified that this answer is correct.

  Most Connect in Apex methods require access to real organization data, and fail unless used in test methods marked @IsTest(SeeAllData=true). [1]

  I do not know whether this has any bearing, but wanted to include the information.

  [1] https://developer.salesforce.com/docs/atlas.en-us.apexcode.meta/apexcode/connectAPI_TestingApex.htm

Setup

• I want to improve the test utility class: One particular method is a behemoth that creates lots of types of test data with dependencies in succession. I want to break that method up into multiple methods that each create one type of data, then this one method (and others in the class) can call them; but also other test classes can have more granular control over certain test data creation.)
• Before I made changes, I wanted to get a "control" by running methods from all 14 test classes that use the utility class, to ensure they pass, so I can verify they still pass during and after I make changes to the utility class.
• I'm not too familiar with Dev Console Test Suites, but figured I'd try one out. So I put all 14 aforementioned test classes into one suite to make it easier to run them all whenever I wanted.

Issue

• When I ran the Test Suite, individual methods in a small handful of test classes failed.
• They all failed on the same utility method

  public static String createInvoice(String fulfillment_order_id) {
      ConnectApi.FulfillmentOrderInvoiceInputRepresentation input = new ConnectApi.FulfillmentOrderInvoiceInputRepresentation();
      ConnectAPI.FulfillmentOrderInvoiceOutputRepresentation result = ConnectApi.FulfillmentOrder.createInvoice(fulfillment_order_id, input);
      return result.invoiceId;
  }
  

• They all failed for the same reason

  ConnectApi.ConnectApiException: Error 1 of 1: ORDER_MANAGEMENT_RECORD_NOT_FOUND - No matching record found. Provide a valid Fulfillment Order ID.

• When I then ran those test classes individually (I did not get as granular as selecting individual test methods to run), without modifying anything anywhere, they passed
• There are other utility class methods for creating an active Order Summary and creating a Fulfillment Order (with an Order Summary Id as the parameter). These are being called prior to calling the failing method. But as stated earlier, if I run these tests individually, they pass.

Other Notes

• When I run the Test Suite multiple times, there is not a consistent number of methods that fail, and it is not always the same methods/classes that fail. (Obviously it is only those test methods that call this utility class method, so there is some repeat, but it is not consistently ClassA or methodB() that fails in every run.)

Questions

I guess I don't know enough of the "inner workings" of the Apex code and/or testing processes....

• Can anyone tell me why (or a best guess) these fail in the Test Suite but not individually?
• Does anyone have suggestions for modifying anything so that I can run these tests in a Test Suite?

---

UPDATE (2022.06.24)

Workarounds I've tried but which have not worked:

• Adding (isParallel=false) at the class level to every class in the test suite

• Adding (isParallel=false) at the class level to only the test utility class

• Adding (isParallel=false) at the class level to every class in the test suite & the test utility class

• Running this command from my local VS Code terminal...

  sfdx force:apex:test:run --suitenames "MyTestSuiteName" --synchronous

  But I discovered this, which is not in the Run Apex Tests documentation:

  Synchronous test runs can include test methods from only one Apex class. Omit the --synchronous flag or include tests from only one class

• Running the above command without the --synchronous parameter

• Running sfdx force:apex:test:run --classnames "ClassA,ClassB,...ClassZ"

Workaround I tried that did work:

• Created a PowerShell function that uses current date/time to make a unique file name, then has these lines:

  sfdx force:apex:test:run --synchronous --classnames "ClassA" > $FileName
  sfdx force:apex:test:run --synchronous --classnames "ClassB" >> $FileName
  ...
  sfdx force:apex:test:run --synchronous --classnames "ClassZ" >> $FileName
  

---

BONUS INFO (2022.06.26)

I (accidentally) discovered that some (at least one) of the ConnectApi methods do not require SeeAllData=true.

• ConnectApi.OrderSummaryCreation.createOrderSummary() does NOT
• ConnectApi.ReturnOrder.createReturnOrder() does

If I have time after I get past all of this, I'll try to determine the full list (at least as it relates to Sales Order Management).