SFDX DSL for Jenkins pipelines?

Question

The Salesforce Jenkinsfile Walkthrough includes code such as this:

stage('Create Scratch Org') {

   rc = sh returnStatus: true, script: "${toolbelt}/sfdx force:auth:jwt:grant --clientid ${CONNECTED_APP_CONSUMER_KEY} --username ${HUB_ORG} --jwtkeyfile ${jwt_key_file} --setdefaultdevhubusername --instanceurl ${SFDC_HOST}"
   if (rc != 0) { error 'hub org authorization failed' }

   // need to pull out assigned username
   rmsg = sh returnStdout: true, script: "${toolbelt}/sfdx force:org:create --definitionfile config/project-scratch-def.json --json --setdefaultusername"
   printf rmsg
   def jsonSlurper = new JsonSlurperClassic()
   def robj = jsonSlurper.parseText(rmsg)
   if (robj.status != "ok") { error 'org creation failed: ' + robj.message }
   SFDC_USERNAME=robj.username
   robj = null
}

that using the Domain Specific Language (DSL) approach ought to read more like this:

stage('Create Scratch Org') {

   sfdx force:auth:jwt:grant --clientid ${CONNECTED_APP_CONSUMER_KEY} --username ${HUB_ORG} --jwtkeyfile ${jwt_key_file} --setdefaultdevhubusername --instanceurl ${SFDC_HOST}

   def obj = sfdx force:org:create --definitionfile config/project-scratch-def.json --json --setdefaultusername
   SFDC_USERNAME = obj.username
}

where the sh and quotes are eliminated, and things like status code error handling and JSON parsing are taken care of once. Right now we have half a dozen Jenkinsfiles, but will probably have hundreds in the future so want to make them cleaner.

It looks like what is needed is an sfdx Jenkins pipeline shared library extension. Does anyone know of one or has one they are willing to share? (The learning curve to write one from scratch looks steepish.)

PS

Or perhaps just some global functions.

Answer 1

Keith Clarke:

Now there is, what you started: https://github.com/claimvantage/sfdx-jenkins-shared-library. I'm glad to contribute to that.

Spreading the word at https://medium.com/@jefersonchaves/streamlining-sfdx-on-jenkins-114f4098166?sk=dfca4c5517c7dfd38f19a715d8ec6642.

Answer 2

I have the sample pipeline that I used to test Jenkins on localhost, hope it can be useful for you. You will need to change environment variables that are hardcoded to the System environments on the instance where Jenkins in running. Jenkins walkthrough

I used one stage because of sfdx tool that is hanging on MacOS, more about that is here: sfdx - Pipeline is hanging on authorization step

So you have to split it into multiple stages that will use withCredentials() and remove stages you do not need, like "5. Importing data to Scratch Org" or "3. Installing CC 4.6 package"

#!groovy
pipeline {

    //execute this Pipeline, or stage, on any available agent.
    agent any

    //specifying global execution timeout of one hour, after which Jenkins will abort the Pipeline run.
    options {
        timeout(time: 1, unit: 'HOURS')
    }

    //specifying variables for all stages
    environment {
        //project vars
        PROJECT_SCRATCH_PATH = "./config/project-scratch-def.json"
        SCRATCH_ORG_NAME     = "diag2_ci_test"
        PERMISSION_SET       = "diag_2_admin"
        CC_4_6_ID            = "04t0V000001Dyaf"
        USER_NAME            = "User"

        //Jenkins env vars
        SERIAL = System.currentTimeMillis()
        BRANCH = env.BRANCH_NAME.replaceAll(/[\/\\]/, "")

        //SFDX vars
        SFDX_HOME                      = "/usr/local/bin/"
        SFDX_USE_GENERIC_UNIX_KEYCHAIN = true
        HUB_ORG                        = "[email protected]"
        SFDC_HOST                      = "https://login.salesforce.com"
        JWT_KEY_CRED_ID                = "6cb80383-e53c-476e-b647-xxxxxxxx"
        JWT_KEY_FILE                   = "server.key"
        CONNECTED_APP_CONSUMER_KEY     = "xxxxxxxxxxxxxxxx.yyyyyyyyyyyyyyyy"
    }

    stages {

        stage("Checkout source") {
            steps {
                checkout scm
            }
        }

        stage("Run build") {
            steps {
                withCredentials([file(credentialsId: JWT_KEY_CRED_ID, variable: "JWT_KEY_FILE")]) {

                    script {
                        echo "on branch name: ${BRANCH}"
                        echo "1. DEV HUB auth"
                        def authStatus = sh returnStatus: true, script: "${SFDX_HOME}/sfdx force:auth:jwt:grant -i ${CONNECTED_APP_CONSUMER_KEY} -u ${HUB_ORG} -f ${JWT_KEY_FILE} -s -r ${SFDC_HOST} --json --loglevel debug"
                        if (authStatus != 0) {
                            error "DEV HUB authorization failed"
                        } else {
                            echo "Successfully authorized to DEV HUB ${HUB_ORG}"
                        }

                        echo "2. Creating Scratch Org"
                        def orgStatus = sh returnStdout: true, script: "${SFDX_HOME}/sfdx force:org:create -s -f ${PROJECT_SCRATCH_PATH} -v ${HUB_ORG} -a ${SCRATCH_ORG_NAME} -d 1"
                        if (!orgStatus.contains("Successfully created scratch org")) {
                            error "Scratch Org creation failed"
                        } else {
                            echo orgStatus
                        }

                        echo "3. Installing CC 4.6 package"
                        def packageResult = sh returnStdout: true, script: "${SFDX_HOME}/sfdx force:package:install -p ${CC_4_6_ID} -r -w 100 -u ${SCRATCH_ORG_NAME}"
                        echo packageResult

                        echo "4. Pushing changes to Scratch Org"
                        def pushResult = sh returnStatus: true, script: "${SFDX_HOME}/sfdx force:source:push -u ${SCRATCH_ORG_NAME} -f"
                        if (pushResult != 0) {
                            error "Push failed"
                        } else {
                            echo "Metadata successfully pushed to the Org ${SCRATCH_ORG_NAME}"
                        }

                        def permissionResult = sh returnStatus: true, script: "${SFDX_HOME}/sfdx force:user:permset:assign -n ${PERMISSION_SET} -u ${SCRATCH_ORG_NAME}"
                        if (permissionResult != 0) {
                            error "Permission Set Assignment failed"
                        } else {
                            echo "Successfully assigned ${PERMISSION_SET}"
                        }

                        echo "5. Importing data to Scratch Org"
                        def moduleImportStatus = sh returnStatus: true, script: "${SFDX_HOME}/sfdx force:data:tree:import -f ./data/Module__c.json"
                        if (moduleImportStatus != 0) {
                            error "Module__c records import failed"
                        } else {
                            echo "Successfully imported Module__c records"
                        }

                        def orderImportStatus = sh returnStatus: true, script: "${SFDX_HOME}/sfdx force:data:tree:import --plan ./data/order-data/order-data-plan.json"
                        if (orderImportStatus != 0) {
                            error "ccrz__E_Order__c records import failed"
                        } else {
                            echo "Successfully imported ccrz__E_Order__c records"
                        }

                        def addUserRoleStatus = sh returnStatus: true, script: "${SFDX_HOME}/sfdx force:apex:execute -f ./scripts/addUserRole.apex"
                        if (addUserRoleStatus != 0) {
                            error "addUserRole.apex script failed"
                        } else {
                            echo "Successfully update user via addUserRole.apex script"
                        }

                        def faqImportStatus = sh returnStatus: true, script: "${SFDX_HOME}/sfdx force:data:tree:import -f ./data/FAQ__kav.json"
                        if (faqImportStatus != 0) {
                            error "FAQ__kav records import failed"
                        } else {
                            echo "Successfully imported FAQ__kav records"
                        }

                        def generateDataStatus = sh returnStatus: true, script: "${SFDX_HOME}/sfdx force:apex:execute -f ./scripts/generateData.apex"
                        if (generateDataStatus != 0) {
                            error "generateData.apex script failed"
                        } else {
                            echo "Successfully update user via generateData.apex script"
                        }

                        echo "6. Running all tests"
                        sh "mkdir -p tests/${SCRATCH_ORG_NAME}"
                        timeout(time: 10, unit: "MINUTES") {
                            def testsResult = sh returnStatus: true, script: "${SFDX_HOME}/sfdx force:apex:test:run -l RunLocalTests -d tests/${SCRATCH_ORG_NAME} -r tap -u ${SCRATCH_ORG_NAME}"
                            if (testsResult != 0) {
                                error "Apex tests run failed"
                            } else {
                                echo "Apex tests successfully run"
                            }
                        }

                        echo "7. Collecting tests result"
                        junit keepLongStdio: true, testResults: "tests/**/*-junit.xml"

                        echo "8. Clean up"
                        def deleteResult = sh returnStdout: true, script: "${SFDX_HOME}/sfdx force:org:delete -u ${SCRATCH_ORG_NAME} -p"
                        echo "Delete Org status " + deleteResult
                    }

                }
            }
        }

    }

    post {
        success {
            echo "Pipeline successfully executed!"
        }
        failure {
            echo "Pipeline execution failed!"
        }
    }
}

Answer 3

Some more Googling reveals articles such as:

Share a standard Pipeline across multiple projects with Shared Libraries

and:

Extending your Pipeline with Shared Libraries, Global Functions and External Code

where it looks like the whole pipeline can be expressed in a "shared library" Groovy file so that each Jenkinsfile just expresses the unique parameters (this example is not for Salesforce but illustrates the idea):

myDeliveryPipeline {
    branch = 'master'
    scmUrl = 'ssh://[email protected]/repos/myRepo.git'
    email = '[email protected]'
    serverPort = '8080'
    developmentServer = 'dev-myproject.mycompany.com'
    stagingServer = 'staging-myproject.mycompany.com'
    productionServer = 'production-myproject.mycompany.com'
}

and the "shared library" Groovy files are registered once in the Jenkins configuration by their location in e.g. Git and are then available for all pipelines.

So rather than the ability to more conveniently write individual SFDX commands, the building blocks would be both code snippets and data references to extract out the repeated sequences. The resulting Jenkinsfile might look something like this:

pipeline {
    stages {
        stage('Checkout') { checkout scm }
        stage('Externals') { retrieveExternals } 
        stage('Create org') { createScratchOrg }
        stage('Install Abc') { installPackage "Abc v2.9" "03a6A000000CksW" env."abc.package.password" }
        stage('Install Def') { installPackage "Def v2.5" "03a6A000000Ckd6" env."def.package.password" }
        stage('Push') { push }
        stage('Test') { runApexTests }
    }
    post {
        cleanUp
    }
}