How to keep sandbox in sync with changes from other developers?

Question

We are a team of 9 developers, working on a large project and commits are made to master branch all the time.

When I needed to work on any new enhancement I used to pull everything from master branch and deploy all the metadata(I have package.xml file in my source folder with wildcard matches for all the metadata components whichever supported ) to my sandbox using ant command ant deployUnpackaged. It was almost like refreshing my sandbox.

The above strategy worked until the payload size for ant deployment increased more than 50MB and I started receiving this error ant deploy error - Maximum size of request reached

Workaround #1: Pull metadata from master branch and look at what is changed and deploy only those components to my sandbox but it is taking too much time as I need to look at each commit from last time to list out the list of files and select those files.

Workaround #2: Deploying chunks of data instead of keeping track of what is changed for ex: First deploy objects and then classes and then pages so on.. But the problem with this approach is the size of the pages folder alone is more than 50MB and I cannot deploy complete pages folder in one go.

I just want to know from the community if there is a better approach than what I am using currently?

Answer 1

Disclaimer: I work for Gearset.

You could use Gearset (http://gearset.com) - we built it for exactly this purpose! We do several things around tracking and migrating changes between orgs that might help in this scenario:

• Daily scheduled comparisons to show org changes over time (great for auditing changes that have happened to an org since you last "refreshed", to catch those pesky production hot-fixes, etc.)
• Comparisons with a deep understanding of the metadata (not just a straight xml diff - we break apart profiles into the constituent permissions, for example, and we do some dependency analysis to catch failures before you push a deployment)
• Some batching in our API requests to avoid hitting the API limits
• Really simple UI for picking changes that you want to deploy, from arbitrary org to arbitrary org (we use the metadata API behind the scenes - we automatically build the package.xml, destrictiveChanges.xml and the deployment package for you)
• Full history of all of your deployments, including the actual deployment package zip files themselves for download

There are other features too, but those seem like the most helpful based on what you've asked. There's a free 30-day trial, we use oauth and you don't have to install anything in your org, so you can give it a go absolutely risk free. I hope that helps, and good luck! :)

Answer 2

If you are using Bamboo, I would use Dan's answer above and modify it to support your Salesforce workflow.

What I've done is to use the Atalassian's Salesforce build plans and modified it to inject a node.js task between the "Checkout Source Code" task and the ant "deployEmptyCheckOnly" task in the "Dev All Tests" plan. When setting it up use "index.js" as the Script and "master yourFeature ./deploy/" as the Arguments. Do not specify a working sub-directory and Dan's script should work just fine.

You may also want to modify the lib/metaUtils.js file from Dan's source code listed in his response above to include all of the metadata types and folder names that you plan to track via automated deployment.

You can also use my forked version at https://github.com/jasonerivera/sf-packager which already has this file updated with all of the currently known metadata types. Use at your own risk.

You'll want to take the source files and folders from sf-packager and put them in the root directory of your repository and check them in. You'll also want to update the build/build.xml file to point to the /deploy folder everywhere you see "deployRoot" as this is where the sf-packager script will build your final deployment package. Do not use /src as the deployRoot or else you will be deploying everything that you checked out of the repo which is what the Atlassian documentation has you doing by default.

deployRoot="${basedir}/deploy/*${sfdc.featureName}*/unpackaged"

This implies that ${sfdc.featureName} is a variable that will be the same as the yourFeature parameter you used in the arguments section of the node.js task you set up as that is the directory sf-packager will automatically create for you. It would be wise to keep this dynamic by using Bamboo repository variables to replace the value in your build.xml file (${bamboo.sfdc.feature.name}), especially if you will be using Plan Branching. You would in this case need to add "-Dsfdc.featureName=${bamboo.sfdc.feature.name}" to the end of the Environment variables section of the deployment ant task in the Bamboo plan you build.

Another feature you may want to turn on is Plan Branching. This will allow you to build a deployment package with only changes between your feature branch against the Dev/master branch. As long as the Salesforce org you are going to "deployEmptyCheckOnly" to (from the feature branch) is not the shared Dev org, then you can have multiple developers pulling from Dev/master automatically down into their feature/master by using automatic branch management (Branch updater), but running unit tests to a separate unadulterated staging box that looks like Dev (or production) without everyone waiting in line to run unit tests against Dev. This concept has proven to eliminate one of the most painful bottlenecks with regard to Salesforce CI in our org.

There are too many other details to post here on how to set it all up, but the key concepts I've listed above. If you follow the Atlassian Salesforce model to the "t" first, then it is easy to inject this script in a task to make building deployment packages automated. The only other thing to watch out for is to make sure you know the 'git diff' command and how it actually works from the Bamboo server once this task is running. I spent a lot of time wondering why I didn't see any changes between master and feature.

Answer 3

I would recommend setting up a Continuous Integration tool to deploy changes to your full sandbox automatically when they are committed to the Master branch.

Options:

• Copado
• Bamboo (Atlassian)
• Jenkins (Set this up on an AWS instance)

Post any issues you encounter setting these up

Answer 4

Salesforce and CI is a good start, although it's often far from perfect, especially if you have users who make config changes outside your repo.

Regarding “Workaround #1”, if keeping track of changes is a pain, you might want to look at something like Test Pilot, which lets you track changes and compare orgs as you go. (Full disclosure: I'm a developer.) You can just compare two orgs to see the list of metadata that needs updating, so it's fairly painless to do.

Answer 5

Below is my implementation of kurunve's idea from comments:

Pre-requisite:

We need to create a text file which holds the SHA of the commit till which the changes are deployed to our sandbox.

When there are any commits to the master branch by other developers, We need to get those commits into our local machine and deploy changes from these commits to our sandbox.

Below are the steps on how I implemented this:

1. Run git pull command to bring changes from repo to local disk.

2. Run the special ant command which internally invokes a python script and the python script does the below actions:

   a) Runs the command git diff --name-only <SHA_from_file> HEAD to identify what list of files were changed from the last commit that we deployed changes to our sandbox vs the HEAD.

   b) From the above list of files it copies each file to a temp folder by copying the file from the base src folder and simultaneously creating package.xml on the fly.

3. Once the above command finishes successfully, it calls regular ant target deploy Unpackaged which deploys files from the temp folder created in step 2 to our sandbox.

4. If the deploy is successful in the above command we call another ant target which calls second python script which writes the SHA of the HEAD to the file mentioned in the pre-requisite.

Below are the ant targets that I created and I start the process by just running ant deployChangesSinceLastDeploy

  <target name="createTargetFolder">
    <exec dir="." executable="python" failonerror="true">
      <arg line="createTargetFolder.py"/>
    </exec>
  </target>

  <target name="deployChanges" depends="createTargetFolder">
    <sf:deploy ../> <!-- the regular stuff here -->
  </target>

  <target name="deployChangesSinceLastDeploy" depends="deployChanges">
    <exec dir="." executable="python" failonerror="true">
      <arg line="writeSHA.py"/>
    </exec>
  </target>

With this approach, I don't need to keep track of what files are changed by looking into each commit as that is taken care by python script creating temp folder with the list of files changed from last commit to HEAD.

If any body has any other suggestions or easy alternatives..Pleas comment. Thanks!

Answer 6

I have also ran into issues deploying the entire codebase so I created a quick tool to easily create packages from branch diffs. I currently use this to create "feature" packages and deploy through ant passing the directory via parameter to the ant build. You could potentially explore wrapping this in a git hook to automate this when new code is pulled.

https://github.com/dlively1/sf-packager

sfpackage master featureBranch ./deploy/ && ant featureDeploy -DfeatureName=featureBranch

<property name="featureName" value=""/>        
<target name="featureDeploy">
        <sf:deploy username="${sf.username}" password="${sf.password}" sessionId="${sf.sessionId}" serverurl="${sf.serverurl}"
            maxPoll="${sf.maxPoll}" deployRoot="./deploy/${featureName}/unpackaged" rollbackOnError="true" checkOnly="false" />
    </target>