Error creating a scratch org: This command requires a dev hub org username set either with a flag or by default in the config

Question

When creating a scratch org, I have this issue that works fine running on my local machine but not when I include in our scripts to run in the CI/CD.

ERROR running force:org:create:  This command requires a dev hub org username set either with a flag or by default in the config.

To replicate the steps, here's my command to authenticate user works fine:

sfdx force:auth:jwt:grant --clientid 3xxx.lSxxx.7Adxxx --jwtkeyfile assets/server.key --username [email protected] --setdefaultdevhubusername --setalias HubOrg

which subsequently results to:

Successfully authorized [email protected] with org ID 00D3jxxx

then the command line that displays the above error is: sfdx force:org:create --definitionfile config/project-scratch-def.json --setdefaultusername --setalias ScratchOrg --wait 10 --durationdays 1

In troubleshooting, I tried to use sfdx force:config:list and displayed only:

 Name                  Value  Location 
 ───────────────────── ────── ──────── 
 defaultdevhubusername HubOrg Local

There were suggestions from discord channel to remove alias and use the working username, however, when I try to set the username with this command: sfdx config:set [email protected], I get an error:

ERROR running config:set:  No authorization information found for [email protected].

I tried using sfdx config:set defaultusername=HubOrg and sfdx config:set defaultusername=ScratchOrg but same issue as above.

As mentioned, I tried running this on my MacOS and works fine. Doing my head in why it won't work on our CI server.

---

Original code:

I use Makefile to run the commands: docker-compose.yml file:

  salesforce:
    image: salesforce/salesforcedx:latest-rc-slim
    working_dir: /work
    entrypoint: '/bin/bash -c'
    volumes:
      - ../:/work
    env_file:
      - .env

called by Makefile:

salesforce: .env
    docker-compose -f $(PATH_DEPLOY)/docker-compose.yml pull salesforce
    docker-compose -f $(PATH_DEPLOY)/docker-compose.yml run --rm salesforce "sfdx --version && sfdx plugins --core"
    docker-compose -f $(PATH_DEPLOY)/docker-compose.yml run --rm salesforce "openssl enc -nosalt -aes-256-cbc -d -in assets/server.key.enc -out assets/server.key -base64 -K $(DECRYPTION_KEY) -iv $(DECRYPTION_IV)"
# Authorize Dev Hub
    docker-compose -f $(PATH_DEPLOY)/docker-compose.yml run --rm salesforce "sfdx force:auth:jwt:grant --clientid $(CONSUMER_KEY) --jwtkeyfile assets/server.key --username $(USER_NAME) --setdefaultdevhubusername --setalias HubOrg"
    docker-compose -f $(PATH_DEPLOY)/docker-compose.yml run --rm salesforce "sfdx force:config:list"
# Create scratch org
    docker-compose -f $(PATH_DEPLOY)/docker-compose.yml run --rm salesforce "sfdx force:org:create --definitionfile config/project-scratch-def.json --setdefaultusername --setalias ciorg --wait 10 --durationdays 1"
#   docker-compose -f $(PATH_DEPLOY)/docker-compose.yml run --rm salesforce "sfdx force:org:create --targetdevhubusername HubOrg --setdefaultusername --definitionfile config/project-scratch-def.json --setalias ciorg --wait 10 --durationdays 1"
#   this shows error in CI: ERROR running force:org:create:  This command requires a dev hub org username set either with a flag or by default in the config.
#   docker-compose -f $(PATH_DEPLOY)/docker-compose.yml run --rm salesforce "sfdx force:org:create --targetdevhubusername HubOrg --setdefaultusername --definitionfile config/project-scratch-def.json --setalias ciorg --wait 10 --durationdays 1"
# # Deploy source to scratch org and run unit tests
#   docker-compose -f $(PATH_DEPLOY)/docker-compose.yml run --rm salesforce "sfdx force:source:deploy -p 'our-sf-core' -u ciorg -c"
# # Delete scratch org
#   docker-compose -f $(PATH_DEPLOY)/docker-compose.yml run --rm salesforce "sfdx force:org:delete --targetusername ciorg --noprompt"
.PHONY: salesforce

Note: DECRYPTION_KEY, DECRYPTION_IV, CONSUMER_KEY, and USER_NAME are included as part of CI variables.

called by pipeline.yml:

steps:
  - label: ':salesforce: Run Salesforce'
    command:
    - make salesforce

I already did the following locally (but understandably, it can't be done via CI): sfdx force:auth:web:login --setdefaultdevhubusername --setalias HubOrg --instanceurl https://oursalesforcewithdevhub.my.salesforce.com

---

Updated files based on David's recommendation:

docker-compose.yml

  salesforce:
    image: salesforce/salesforcedx:latest-rc-slim
    working_dir: /.sfdx
    entrypoint: sh
    command: -c "chmod +x ./salesforce.sh && ./salesforce.sh"
    volumes:
      - ./:/.sfdx
    env_file:
      - .env

Makefile:

salesforce: .env
    mkdir ~/.sfdx
    openssl enc -nosalt -aes-256-cbc -d -in assets/server.key.enc -out assets/server.key -base64 -K $(DECRYPTION_KEY) -iv $(DECRYPTION_IV)
    docker-compose -f $(PATH_DEPLOY)/docker-compose.yml run --rm salesforce

salesforce.sh:

#!/bin/bash

set -e

export PATH=~/sfdx/bin:$PATH

sfdx --version && sfdx plugins --core

# Authorize Dev Hub
sfdx force:auth:jwt:grant --clientid ${CONSUMER_KEY} --jwtkeyfile assets/server.key --username ${USER_NAME} --setdefaultdevhubusername --setalias HubOrg
# This is where the error ERROR running auth:jwt:grant:  Unexpected argument: assets/server.key occurs even if I change the sub-directory to .sfdx

Final configuration that worked:

docker-compose.yml file:

  salesforce:
    image: salesforce/salesforcedx:latest-rc-slim
    working_dir: /sfdx
    entrypoint: sh
    command: -c "chmod +x ./salesforce.sh && ./salesforce.sh"
    volumes:
      - ../:/sfdx
    env_file:
      - .env

running the salesforce.sh (as indicated in the command above):

#!/bin/bash

set -e

export PATH=~/sfdx/bin:$PATH

sfdx --version && sfdx plugins --core

# Authorize Dev Hub
sfdx force:auth:jwt:grant \
--clientid ${CONSUMER_KEY} \
--jwtkeyfile assets/server.key \
--username ${USER_NAME} \
--setdefaultdevhubusername \
--setalias HubOrg

# Create scratch org
sfdx force:org:create --definitionfile config/project-scratch-def.json --setdefaultusername --setalias ciorg --wait 10 --durationdays 1

# Deploy source to scratch org and run unit tests
sfdx force:source:deploy -p our-sf-core -u ciorg -l RunAllTestsInOrg -c

# Delete scratch org
sfdx force:org:delete --targetusername ciorg --noprompt

called by Makefile:

salesforce: .env
    openssl enc -nosalt -aes-256-cbc -d -in assets/server.key.enc -out assets/server.key -base64 -K $(DECRYPTION_KEY) -iv $(DECRYPTION_IV)
    docker-compose -f $(PATH_DEPLOY)/docker-compose.yml run --rm salesforce

Answer 1

Your pipeline isn't persisting the location where SFDX stores authentication details, and each command is running in a completely separate container.

salesforce: .env
    docker-compose -f $(PATH_DEPLOY)/docker-compose.yml pull salesforce
    docker-compose -f $(PATH_DEPLOY)/docker-compose.yml run --rm salesforce "sfdx --version && sfdx plugins --core"
    docker-compose -f $(PATH_DEPLOY)/docker-compose.yml run --rm salesforce "openssl enc -nosalt -aes-256-cbc -d -in assets/server.key.enc -out assets/server.key -base64 -K $(DECRYPTION_KEY) -iv $(DECRYPTION_IV)"
# Authorize Dev Hub
    docker-compose -f $(PATH_DEPLOY)/docker-compose.yml run --rm salesforce "sfdx force:auth:jwt:grant --clientid $(CONSUMER_KEY) --jwtkeyfile assets/server.key --username $(USER_NAME) --setdefaultdevhubusername --setalias HubOrg"
    docker-compose -f $(PATH_DEPLOY)/docker-compose.yml run --rm salesforce "sfdx force:config:list"
# Create scratch org
    docker-compose -f $(PATH_DEPLOY)/docker-compose.yml run --rm salesforce "sfdx force:org:create --definitionfile config/project-scratch-def.json --setdefaultusername --setalias ciorg --wait 10 --durationdays 1"

Because you're using docker-compose run --rm, every one of these commands runs in an isolated container, with no access to the filesystem of the previous container.

You use a volume mount in your docker-compose.yml:

    volumes:
      - ../:/work

but it's in the wrong place. You're not persisting any data in /work inside the container. sfdx stores its authenticated orgs in ~/.sfdx, which is being discarded each time you run a command and then destroy the container.

You could fix your volume mount, but more broadly, I'm bemused why you're running a bunch of individual commands in separate containers. Just write a shell script and run it in one container, so all of your operations share state.