11

Situation

We are a small team of internal developers who recently adopted a more modern DevOps model (from changes sets and one org for multiple developers) and wanted to ask the community for some guidance regarding pushing changes which have different testing timelines through their source code and environments.

Current Workflow

We have 3 long lived shared origin branches:

  1. develop (represents our integration environment)
  2. stage (represent our uat environment)
  3. prod (represents production)

Each one of our developers takes a ticket and checks out a local branch and works in their own dev sandbox. Once they complete the ticket and rebase against develop, they will push this up to origin and do a pull request against develop. Once it passes code review, the changes get merged into develop and the changes get deployed to our integration environment and subsequently prod.

Pain Point

We've seen a lot of benefit from all this but one of the things we are struggling with is that each ticket has a different testing time frame.

As an example, let's say develop has recently merged feature1, feature2, and feature3. feature1 and feature2 may be ready to be merged into stage which will ultimately get deployed into UAT and into production in the next few days while feature3 may require more rigorous testing and get held back for a week or two.

Currently it's really painful to try to decouple this and has created a bit of a bottleneck for us to manage.

Considered Paths

  1. We've considered keeping the featureBranches alive post merge into develop so in this case if feature1 and feature2 both pass integration testing, we will create two merge requests from the feature1 and feature2 branches into stage.

My assumption here is that if feature3 was merged before feature2 into develop, feature2 which will have rebased against develop will already have feature3 so if we go about this route, we may need to revisit our process of rebasing before each merge into develop.

  1. The alternative is creating a branch from develop that includes feature1, feature2, and feature3 and reverting feature3 and having stage merge with this branch.

Once feature3 passes, we will merge develop into stage.

Specific Ask

We're leaning more towards path#2 but wanted to see if the community had any additional recommendations or thoughts as we try to smoothen out our initial pass at our DevOps model.

I realize this is a highly subjective question but any guidance is highly appreciated.

4
  • 2
    Is there a Salesforce aspect to your question or are you inquiring about general best practices for developing software? atlassian.com/git/tutorials/comparing-workflows/… is a popular git workflow.
    – identigral
    Commented Feb 28, 2022 at 19:25
  • 3
    A fair point @identigral but I do agree with the poster there are some SFDC specific nuances here (or at least, nuances that crop up a lot more in SFDC development teams than the average software team). In particular, the requirement for being able to push changes "past each other" in the pipeline through the various environments is common in SFDC teams but isn't addressed by most common software development git workflows, as far as I've seen.
    – olane
    Commented Feb 28, 2022 at 19:36
  • 1
    @olane There's plenty of software other than Salesforce with multiple, concurrent changes that may impact each other. The general, non-SF specific subject of workflows, branching strategies and their pros/cons has been well discussed and documented, e.g. StackOverflow.
    – identigral
    Commented Feb 28, 2022 at 20:03
  • 1
    @identigral Thank you for the resources and links. The intention here was to get guidance as to what others have tried and could point us in a direction to find a solution which could potentially get us to a solid first state. As you point out, a lot of this has been documented extensively and we've been reading up and then trying to implement it in practice. Each iteration takes up some time until we hit an issue and then we'll try read up some more. So to answer your first question, it's probably inquiring about software best practices but unsure if there is a SFDC aspect to consider.
    – Tom G
    Commented Mar 1, 2022 at 1:36

2 Answers 2

6

My perspective is that of an ISV developing managed packages, so may be unexpected. For me the thinking you have doesn't fit with ISV development because you consider:

  1. Feature testing is something separate from the feature development and
  2. Separate branches are used to represent what should be a monotonic sequence of delivery to different orgs, regardless of the org use.

Package Development Restrictions

If you consider this from the perspective of a managed package (especially 1GP), there are some specific constraints that help you identify how to manage your Salesforce development:

  1. Versioning is a one way trip via creation of releases. You cannot downgrade. In 1GP this is even tighter: versioning is entirely sequential (patches allowed but highly restricted in content - see below).
  2. Only official release versions can be installed on production orgs.
  3. Branching for a given release is limited to patch fixes that add no new components, only providing bug fixes (generally at the level of Apex method content, new non-global methods, or updated flows etc., with no new object model definitions).

Approach

We approach this by having the following:

  1. The master branch which represents the "next GA release" we are working toward - i.e. containing everything that is ready for delivery in the next official release.
  2. Various x.y tags which mark the official GA releases against master.
  3. Various x.y.next branches which represent patch development, containing everything that is ready for delivery in the next x.y patch release.
  4. Various x.y.z tags that mark the official patch releases against the related x.y.next branch.
  5. Where multiple stories contribute to an epic that requires a number of them to be present together in the delivery (a minimum viable feature delivery), have an epic branch. This is always hung off master.
  6. Every story has its own branch, off master or the related epic branch.
  7. Bug fixes are always first developed in fix branches off master and are then "backported" (frequently via cherry picking) to the required x.y.next branch or branches for earlier version(s). We generally limit the fix to master, the most recent GA release and the version where the bug was found. A backport never includes new components, since it has to go in a patch release, so may have to be adapted to avoid the need, or may not be backportable.

A story cannot be delivered without QA testing, where delivery is represented by the (squash) merging of the story branch into its target branch (master or the epic branch). An epic may require additional specific testing that covers the broader picture for the epic before it gets (non-squash) merged to master.

Bugs found in QA testing of a story or epic are addressed in the story or epic branch and are retested before the story or epic is delivered (and merged). Some bugs might be decided to be known issues, so are raised but not fixed before delivery of the story/epic.

Before official release, regression testing is applied and bugs raised and fixed as needed (using their own bug branches off master). This covers everything in master at the time the release is to be created. At this point a tag is applied on master to capture where that release was created from, and a new x.y.next branch created from there for bug fixes to be backported as needed later.

Since this is package-based development and delivery, UAT and production orgs simply have a specific version of the package installed prior to testing or adoption.

Orgs and git: the source of truth

Development uses namespaced scratch orgs, specific to individual work items (stories/bugs), as does the story/epic testing by QA, with the package metadata pushed in an unpackaged manner. Regression testing uses a beta package release installed on a non-namespaced scratch org. QA use an enhanced sfdx data loading tool to ensure test data is available in the org(s).

Note that we do not use any long-lived orgs during this development process. This is quite deliberate, since the source of truth is git, not some random org in Salesforce. Every org must effectively be reconstructed from the data in git (via installing a given version of the package when required, of course).

Packages for non-ISVs

While I would not expect non-ISVs to adopt use of managed packages, Salesforce does provide the "unlocked package" (2GP) which is ideal for non-ISV use. The restrictions applied to unlocked packages are less severe so there is room to flex the process a bit more.

5

Quick disclaimer: I work for Gearset developing tooling in this space and I've been thinking about these workflows a lot recently. That means both that I've thought about a lot of edge cases, and that I might be biased towards things that work "least bad" in the general case for most people vs. best for your team specifically. So, pinch salt as you deem appropriate.

Anyway - this is a surprisingly difficult problem in the general case, in my experience!

My instinct personally is that your path #1 is the most conceptually clean, since it keeps your features distinct all the way down your pipeline (apart from the rebase problem which I'll come back to).

A problem you haven't discussed with path #2 is that you need to be sure you've excluded everything necessary from your branch off develop each time you create one, which is an easy place to trip up. If it ever gets to the point where several long running features are stuck in one environment you're going to end up reverting all of them out of a lot of branches. Reverting them may or may not be a clean and easy process if you are rebasing every time you merge - I'd have to try it out to be sure.

If you do merge one of these "package" branches into stage but then decide only some subset of those should go into the next environment, you then need to go and find the original pointer to the branch you want to exclude and carefully cut it out again. Could be fine, could be tricky.

Coming back to option #1, you're right that rebasing branches onto the environment before merging will cause them to include anything in those environments, so naively merging those further down the pipeline will risk pulling through unrelated changes. Below I've got some ideas for how you might take option #1 and solve that problem.

If you would consider not rebasing any more

You're going to have a problem with merge conflicts. If you're not rebasing any more, the natural way to solve a merge conflict is to merge your environment branch into your feature branch, before closing the Merge Request. Unfortunately this causes the same problem as before in the next environment - unrelated changes are now in your feature branch.

Options to solve this:

  1. Never merge the environment into the feature branch. Instead, solve merge conflicts as part of the merge commit into the environment branch. This keeps your feature branch clean but most VCS providers discourage you from doing this, and it may not be possible at all if you have branch protection on your environment branch. It also gives you no opportunity for reviewing (or running any checks against) the whole state of the merge request after conflict resolution but before merge, which may or may not matter to you.
  2. Don't do a merge request between the feature and the environment, make a copy of the feature branch beforehand which you can solve any conflicts in if necessary. This keeps the feature branch clean for later propagation, but involves some annoying overhead if creating the branches manually.
  3. If a conflict occurs, create a conflict resolution branch off the feature branch to solve conflicts in, then merge that rather than the original feature branch. Depending on your VCS and branch protection settings you may need to/want to create a replacement merge request at this point. Slightly more overhead than point 2 but it only occurs when there's a merge conflict.

A variant of the second point is that you could create a "propagation" or "promotion" branch whenever you're ready to issue a merge request for N branches into an environment, and batch several feature branches into that propagation branch, which can then have any merge conflicts solved again independently to the feature branches.

If you want to keep rebasing

I haven't tried this so I'm only 90% confident it would work, but I think you could get away with doing an extra rebase before issuing a merge request against the next environment each time.

For example:

  • merge feature1 into develop (rebasing)
  • now, rebase feature1 from develop onto stage
  • issue merge request feature1 -> stage

You'd have to be careful to do this correctly each time, and I haven't fully thought through how it interacts with merge conflicts, but off the top of my head it works.

1
  • Wow! Thank you for the thoughtful consideration of the 2 approaches and you've given me some things to play with here!
    – Tom G
    Commented Mar 1, 2022 at 1:40

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .