How to use "sfdx force:source:pull" with folders other than "main/default" where components are added in the scratch org?

Question

I've setup the folder structure for a new project following the pattern identified in sfdx-isv/sfdx-falcon-template. Also see this question Two SFDX project folder structure questions for some background.

When I add e.g. a new SObject in the scratch org and use:

sfdx force:source:pull

the new components are put in a fixed folder "main/default" - there is no option to nominate the correct folder. So the folder structure is compromised.

The change I am making in the scratch org is adding a new SObject and multiple fields: no sfdx commands at present to make that easy from the command line.

(When components are already present locally in folders other than "main/default" the pull does place the changes correctly.)

Is there a work-around here apart from just sticking to using "main/default" for everything or creating everything locally so the component can be setup in the right folder?

PS

Tried the manual workaround of pulling then moving the files locally but so far no joy: the move is seen as a delete that is bad news when you next push.

Answer 1

This is by no means great, but you can force your way around this by editing the local file used to track changes.

1. create a new field in the org
2. sfdx force:source:pull -> field gets pulled into force-app/main/default
3. move the field to a separate directory (at this point sfdx thinks i've deleted the field)
4. open .sfdx/orgs/<your_scratch_org_username>/sourcePathInfos.json
5. delete any entries related to the original path
6. remind yourself that benioff is free from original sin, so all bugs are features, but you aren't so try and write some decent code

Big picture, SFDX had grand dreams of letting you organize stuff into multiple folders. But there are major bugs with the implementation, and it's been multiple years at this point and none of them are fixed. If you can just stick to one folder. If you do multiple folders, prepare to pay for it in pain.

Issues I'm aware of

• refreshing a file outside of the default folder ends up with an extra copy of the file in the default package folder and no updates to the one you intended to refresh
• if you move metadata from one package directory to another scratch org commands think you're deleting stuff
• when you pull metadata you've got no good way to control where it goes

Answer 2

Unfortunately, the CLI will always pull "Remote Add" metadata (ie. metadata not previously seen in any local SFDX Package Directory) into main/default/<metadata-type> inside of the default Package Directory specified in your project's sfdx-project.json file.

From the CLI's point of view, this behavior is by design. You need to have a place where previously unseen metadata can go, and <default-package-dir>/main/default provides that.

The part that needs fixing is the fact that the CLI is treating your subsequent reorganization (move) of your SFDX source files as a series of "Local Deletes" and "Local Adds" instead of just a simple "Local Moves".

Safe Harbor: The SFDX team is aware of this bug and is working on a fix. There is no ETA for when such a fix will be delivered. I'll update this answer once an ETA is available.

In the meantime, there are two ways you could work around this.

Workaround 1: Create a "local starter", push to scratch org, then customize in Setup UI

Technically, you can create your own metadata files locally and then do an initial push to your scratch org, even for those types that are more complex in SFDX, like customObjects.

Take a look at how an existing metadata type (object, permset, profile, etc.) is stored as SFDX source, copy that, then customize it as necessary for your new metadata component.

This isn't an elegant solution, but it does have the benefit of getting you comfortable with editing metadata files. For anyone who hasn't done this, it can be intimidating at first but you end up feeling like Neo seeing the Matrix once you've done it a few times. ;-)

Workaround 2: Build in Setup UI, reorganize locally in bulk, then rebuild your scratch org

You mentioned that you're aware of the SFDX-Falcon template. One of the things I like about SFDX-Falcon is the set of shell scripts provided in the dev-tools directory. They give a head start to anyone who wants to automate Salesforce CLI actions, and the one I use the most is rebuild-scratch-org.

If you know you're going to add a couple of custom objects and fields to your project, go ahead and build them in the Setup UI, pulling as you go without worrying about where the metadata is getting stored locally.

Once you're done with all the changes, open up a GUI file explorer and drag things out of <default-package-dir>/main/default and drop them wherever you want (as long as your destination is inside of an SFDX package directory).

Execute your rebuild-scratch-orgs shell script (which you've hopefully customized so that it does everything you need for a full scratch org build), and you'll be good to go. The CLI will know where everything is supposed to be from that point forward.

Closing Point: Is organizing my source worth all the trouble?

I definitely understand the frustration at not being able to fine-tune where new "Remote Add" metadata is saved by default. I still think it's worth the effort to come up with some logical structure that goes beyond the basic force-app/main/default that we get by default.

Your solution might not look exactly like SFDX-Falcon, but it's a good idea to do something, especially for projects based on large enterprise implementations and AppExchange packages.