We have an SFDX build set up that runs our 2,800 tests against 6 different scratch org configurations in parallel. But every build fails on what seems like a random one of the scratch orgs (i.e. it isn't the configuration variations that are at play here) with timings like these:
- the tests take 2h 19min instead of 1h 24min (ballpark for the other 5) i.e. x1.65 slower
- it's not just the tests e.g. the push takes 12min 7s instead of 7min 36s (ballpark for the other 5) i.e. x1.60 slower
The failure problem is that there are some unit tests that confirm that some complex algorithms haven't had their performance degraded by code changes by running extreme cases that are about 20% inside of the "Apex CPU time limit".
Any suggestions on how to always get given "good orgs"? Are we being throttled in some way here?
I can't think of a way to modify the performance tests to workaround this. Can you?
PS
On the workaround part...
I'm looking to ensure that an existing algorithm isn't made slower by code changes so that existing customers are not broken for existing data. My problem here is that orgs are sometimes taking x1.6 of the time. But if that slow down is fairly independent of the specific code, then I could have say a 100ms of CPU time piece of code to calibarate - that would take 160ms on the "bad org" - and use that result to determine the repeat count on the code I do care about.
Dodgy territory though.
PPS
Running the test that exceeds the 10.0s CPU limit in the "bad org" in my development scratch org, the CPU time used is 1.2s. So a lot more than x1.6 happening in CPU usage for "bad orgs".
