3

We approached Salesforce development like modern software development. We have a medium sized Apex codebase (single unmanaged package), and on every pull request, we run CI (aka the test suite, linting, etc). We also run the test suite on merges. We deploy this package to our staging and production environment, running the tests again.

We all know Salesforce is dirt slow at executing code. Our codebase has about 5,000 components and about 2,000 Apex tests. This is fairly small in terms of software development, but on Salesforce the speed is untenable. Our deploys take over an hour to run. We deploy multiple times a day, which is already better than 99% of Salesforce development life-cycles, but this is an order of magnitude too slow for modern software development, where deployments should complete in minutes, not hours.

For those of you with large codebases, how do you handle modern CI/CD to improve deployment speed?

2
  • Do you have tests executed in parallel?
    – Phil W
    Sep 21, 2022 at 21:10
  • @PhilW No, I think we disabled it because of unable_to_lock_row errors
    – Andy Ray
    Sep 21, 2022 at 21:26

1 Answer 1

2

Our primary AppExchange package contains over 7000 metadata items (deployment count) and more than 2400 apex unit tests.

Creating a scratch org takes a small number of minutes and deploying the metadata takes about 10 minutes. The killer is test execution, unless you enable parallel execution. The latter is essential for development of this scale.

You mentioned in a comment that you were seeing unable to lock row type errors. For me this typically means one of:

  1. You have unit tests that access data on the org via the SeeAllData attribute of the @IsTest annotation (documentation).
  2. You are inserting the same custom settings into the database in various tests.

IMHO, the former is a total no-no. Unit tests should never rely on data held in the org and must create the data it needs during test setup or initialization. You need to refactor these tests.

The latter is, again IMHO, a long-standing platform bug where there is leakage of data between tests.

We spent a fair amount of effort to eradicate these custom settings related, row locking issues by:

  1. Updating the whole production code base to access custom settings records via an apex class. This class ensures that the requested settings are cached (via statics) in memory once "loaded" and, indeed, that if there's no settings in the database that a settings is still initialized using the default values defined for the settings fields.
  2. Making unit tests use this same means to access custom settings. Since the settings are cached in memory, the unit test can load the settings, changing values as needed, before the production code is invoked. This works nicely because every test is in a separate transaction and, as you are probably aware, static values are scoped to the transaction. This means one test does not interfere with any other.

This means that the unit tests, that would take over 40 minutes to execute sequentially, now take something like 12 minutes to run.

This does mean our CI/CD service still takes something like 25 minutes to build a scratch org, deploy the code and run the unit tests (we run our jest tests in parallel with the unit test execution so this has no overall impact), but this isn't a blocker for us; the process is simply asynchronous from the developer perspective. They can get on while the CI/CD does its thing, and get a notification if there are any issues.

9
  • A 12 minute test run is very good. We tried using spin-up scratch orgs but getting the configuration to match our staging/prod instances was very challenging, we threw a few months at it and never fully got there, and Salesforce told us scratch org configs will always lag behind what you need, and documentation is terrible for scratch org config. I guess we should revisit. Did you look into breaking up your package into smaller ones?
    – Andy Ray
    Sep 22, 2022 at 16:35
  • Since the package is 1GP and we have quite a few customers using it (some of them very large) we cannot consider breaking it up until 1GP to 2GP migration is GA and then we need Salesforce to allow metadata to be migrated from one 2GP to another (in the same namespace and adding namespaceAccessible). Suffice to say, not holding my breath.
    – Phil W
    Sep 22, 2022 at 17:46
  • Can you not use org shape to help you with appropriate scratch org setup to reflect your production org?
    – Phil W
    Sep 22, 2022 at 17:53
  • Org Shape wasn't generally available until - what - this year? Is this something you did recently? Or is it something you don't use, and have a more lightweight scratch org definition file?
    – Andy Ray
    Sep 29, 2022 at 16:51
  • It was a relatively recent thing. We have no need for any complexity here and a scratch def file is enough for us. We are, however, waiting for scratch org snapshots so we can pre-load dependency packages. At the moment it can take an hour just to get the scratch org populated for one of our integration packages.
    – Phil W
    Sep 29, 2022 at 17:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .