How to speed up scratch org creation when it includes installed packages

Question

We are just setting out on our journey with 2GP. Our first 2GP is a new "extension package" that integrates our core product (an existing 1GP) with a third-party product (another existing 1GP). As such, a scratch org for the 2GP development requires us to have both packages installed since the integration will add fields to objects from both packages and include Apex code using objects from both packages.

I have added appropriate details in the sfdx-project.json file listing the two dependencies, something like:

{
    "namespace": "mynamespace",
    "sfdcLoginUrl": "https://login.salesforce.com",
    "sourceApiVersion": "48.0",
    "packageDirectories": [
        {
            "path": "force-app",
            "default": true,
            "package": "My Integration Package",
            "versionNumber": "0.0.1.NEXT",
            "definitionFile": "config/myint-scratch-def.json",
            "dependencies": [
                {
                    "package": "CorePackage@2.1.2.1"
                },
                {
                    "package": "ThirdParty@5.10.4.1"
                }
            ]
        }
    ],
    "packageAliases": {
        "CorePackage@2.1.2.1": "04t0N000000AAAAAAA",
        "ThirdParty@5.10.4.1": "04t4P000002BBBBBBB"
    }
}

Of course, this doesn't actually handle installation of the packages when creating the scratch org, and I have written some scripting to pull required details from this JSON file (plus installation keys from "elsewhere") to handle the necessary force:package:install commands prior to pushing our new 2GP code to the scratch org.

However, right now (with a basically empty 2GP development source tree) simply creating the scratch org and deploying the two dependencies is taking just short of an hour.

We normally work with a scratch org per developer per task, so we know the scratch org is clean and fit for purpose. We also have continuous integration which again creates a scratch org each time code is committed to git. This becomes untenable when it takes this long just to create the scratch org and get it populated.

In the short term we can drop CI for this package and have developers use long lived scratch orgs (30 days) for all the 2GP development, so the creation time only happens monthly, but this is not ideal and could expose us to issues with cruft the devs create on their orgs.

What I'm looking for is a way to speed up the creation of the scratch org, including the installed packages.

It appears there was some work done around allowing scratch orgs to be created using a TSO snapshot, presented at Dreamforce 18 (see this FishOfPrey twitter post) which would be ideal here, I believe. We could install the 1GPs into the TSO, create a snapshot and build the scratch org from that. However, the mechanism used (the "template" property in the scratch def JSON file visible in the slide photo) isn't documented and there's no other mention of this approach elsewhere in official documentation.

Does the create scratch org from TSO snapshot feature exist? Is there an alternative? How else can I improve scratch org creation/install performance?

Answer 1

There does exist a feature as shown in the Dreamforce video you reference that allows creation of a scratch org from a TSO Template Id via the "template" key.

This feature is not currently GA. I reached out to the team internally that owns this feature and confirmed that it is not typically available to customers even by request.

Answer 2

To summarise the equivalent threads I raised on various Salesforce forums:

1. The "Scratch Org Snapshots" feature, as mentioned in the documentation, is this feature.
2. This pilot is now closed (after 1.5 years or so) and there's no clear timeline on what will happen next.
3. Use of these snapshots massively improves performance if you need to install a large number of dependency packages or if you need to install large packages. It was suggested that the scratch org creation from snapshot (with whatever is installed on that snapshot) takes of the order of 5 minutes and that this is more-or-less constant.
4. This feature appears to be missing the ability to associate the snapshot with the namespace defined in the sfdx-project.xml (though I wonder if it simply retains the namespace that the scratch org [from which the snapshot was created] had - this would suit us I suspect).
5. There are problems with sites and these snapshots (they have to be manually re-created at this point).
6. The feature is not based on TSOs - this was (apparently) an assumption/misunderstanding in the FishOfPrey posting I referenced in the question.
7. The feature's PM is Rohit Mehta @ Salesforce so if you also need this feature I would suggest you add your voice to the calls for this to be "fixed" and expedited to GA.

UPDATE: Feedback from Rohit Mehta:

To quote:

"The team supporting Scratch Orgs is focused on Org Shape currently but we plan to pick up Snapshots soon after in the Spring 21 time frame".

If you are also suffering from the slow-to-create-and-initialize-a-scratch-org issue, can I suggest you post a comment on this discussion thread on the Salesforce DX Trailblazer community forum (login required) expressing your desire for them to provide snapshot management for scratch orgs.