Sandboxes often reshuffles files like page layouts making it less git-friendly

Question

When refreshing Sandboxes I would prefer if all metadata is exactly copied from the source org, but rather we often see files being randomly reshuffled without their content changing when we do a full pull via CICD. This is all fine till multiple people try to merge their change to develop branch, because then we have unnecessary merge conflicts.

The current process is that No-Coders (Admins) base their sandbox by full pushing via Pipeline to their Sandbox, do their changes and retrieve them via a full pull. The pull results in the shuffeled metadata like page layouts being retrieved.

If I do this by hand and break it down for this issue, I'd do the following:

1. Repo LayoutA has items in order: 1,2,3
2. Org after cloning has: 1,3,2
3. Repo->Org: sfdx force:source:deploy force-app/main/default/layouts
4. Org->Repo: sfdx force:source:retrieve force-app/main/default/layouts
5. The Org still has the file with order: 1,3,2

Resulting in the following reshuffled file:

We are also currently working with delta deployments resulting in the page layouts being deployed a lot.

Is there any way to mitigate/solve that problem or is it only something Salesforce can fix?

*Full Push/Pull: source:force:push/pull in a CICD context is similar to doing sfdx force:source:deploy/retrieve -p force-app. For pushing it is identical to sfdx force:source:deploy -p force-app --tracksource. For pulling it pulls all components listed in SourceMember, which for our purpose is similar to retrieving everything in force-app.

Answer 1

One approach would be to mandate that when a developer makes a change and deploys/pushes that to their scratch org, that before committing they also do a retrieve. That will ensure that the ordering and indenting of the file is "normalised" to exactly the character by character form that is returned by any org.

Answer 2

This appears to be a problem of your own creation. This answer presumes that when you say "full pull via CICD," you're actually talking about using force:source:retrieve. There is a distinct difference between pull and retrieve. The former uses Source Tracking to detect changes, while the latter naively downloads everything from multiple orgs, wreaking havoc wherever it goes.

A CICD setup needs to (a) treat the repository as the source of truth, and (b) not rely on the metadata in any specific org as being 100% correct. CICD must only use pull commands against the developers' sandbox orgs, and use one, and only one, org to deploy and retrieve from. Most CICD setups I see use Scratch Orgs for this purpose. This eliminates the random noise you get from things like experimental changes the developer left in and differences in API versions between orgs, especially during release windows.

In the DX model, your repo is the Source of Truth. No other org, not even production, can be immune from this policy in order for everything to work smoothly. To keep this process smooth, you must not rely on any specific org being the Source of Truth. You should not be retrieving or pulling from any org directly that isn't under CICD's absolute control.

The entire app life cycle should be as follows.

First, the developer creates/refreshes a Sandbox, or creates a Scratch Org. Next, the developer performs a fetch and pull from the development branch. Third, the developer creates a new branch for building. Fourth, the developer deploys the branch to the org. Fifth, the developer performs any work necessary, and commits any changes. Sixth, the developer goes back to the development branch, performs another pull, then goes back to the feature branch and merges any changes. After this, the branch should be pushed to the remote repo and a pull request created.

At this point, the developer is done. CICD can take over from here. First, it should pull from the development branch, then merge to the feature branch (this should be harmless but reduces race conditions). Then, it should either select a specific, dedicated CICD sandbox, or create a new Scratch Org. After that, it should perform a full deploy and likely Run All Tests; if the deploy fails, abort here and report errors. Then, it should perform a full retrieve and commit any differences. Finally, it can merge back to the development branch and perform any additional deployments needed.

At no point should CICD be pulling anything from any org that is not a dedicated org controlled by CICD. I can't stress this enough. The entire point of the DX model is that orgs are no longer the subjective Source of Truth, but instead, the repo is the objective Source of Truth. This eliminates an entire class of problems, such as those that you've described.