17

We are a team of 9 developers, working on a large project and commits are made to master branch all the time.

When I needed to work on any new enhancement I used to pull everything from master branch and deploy all the metadata(I have package.xml file in my source folder with wildcard matches for all the metadata components whichever supported ) to my sandbox using ant command ant deployUnpackaged. It was almost like refreshing my sandbox.

The above strategy worked until the payload size for ant deployment increased more than 50MB and I started receiving this error ant deploy error - Maximum size of request reached

Workaround #1: Pull metadata from master branch and look at what is changed and deploy only those components to my sandbox but it is taking too much time as I need to look at each commit from last time to list out the list of files and select those files.

Workaround #2: Deploying chunks of data instead of keeping track of what is changed for ex: First deploy objects and then classes and then pages so on.. But the problem with this approach is the size of the pages folder alone is more than 50MB and I cannot deploy complete pages folder in one go.

I just want to know from the community if there is a better approach than what I am using currently?

  • Probably solution would be start to chunk pages by some amount per request. But I would say 'wow' 50 mb of VF pages only.About partial deployment -- you can always save in file SHA of your latest save to server, and call git show --name-only SHA to grab changed files from another commit. That would give you list of changed files – kurunve Mar 2 '16 at 16:17
  • @kurunve, Could you please add more details? I am not clear on how to use SHA. Thanks! – javanoob Mar 2 '16 at 16:23
  • 1
    SHA is just a commit hash. I made assumption that you have script that performes execution of ant deployment in a row and build package.xml dynamically every time, so my idea was save last sync time commit hash to file (via command git rev-parse HEAD) , before next sync read that from file, execute git show --name-only <readed hash>, parse output into package.xml and perform deployment. after deployment -- save new hash in case of success. Script itself would be different for *nix/windows platform, but those are main keypoints. – kurunve Mar 2 '16 at 16:32
  • @kurunve You are correct, we are not creating package.xml on the fly. Looks like git show <SHA> command only shows files that are changed in that commit but I think we need a way to know what all files changes from the given commit to the HEAD. Not sure if my question is clear but your idea sounds interesting and looks like a possible solution. – javanoob Mar 2 '16 at 16:45
  • 1
    @kurunve, Did you mean git dif? Because this command shows list of files that were changed between two commits. stackoverflow.com/questions/1552340/… Looks like I have everything needed in place and I will start making it work tonight. Thanks for your help also please post if you have any scripts that would be very helpful. once again thanks for your time! – javanoob Mar 2 '16 at 16:59
10

Disclaimer: I work for Gearset.

You could use Gearset (http://gearset.com) - we built it for exactly this purpose! We do several things around tracking and migrating changes between orgs that might help in this scenario:

  • Daily scheduled comparisons to show org changes over time (great for auditing changes that have happened to an org since you last "refreshed", to catch those pesky production hot-fixes, etc.)
  • Comparisons with a deep understanding of the metadata (not just a straight xml diff - we break apart profiles into the constituent permissions, for example, and we do some dependency analysis to catch failures before you push a deployment)
  • Some batching in our API requests to avoid hitting the API limits
  • Really simple UI for picking changes that you want to deploy, from arbitrary org to arbitrary org (we use the metadata API behind the scenes - we automatically build the package.xml, destrictiveChanges.xml and the deployment package for you)
  • Full history of all of your deployments, including the actual deployment package zip files themselves for download

There are other features too, but those seem like the most helpful based on what you've asked. There's a free 30-day trial, we use oauth and you don't have to install anything in your org, so you can give it a go absolutely risk free. I hope that helps, and good luck! :)

  • 1
    So you're the one spamming us with those ads! :P Seems like a great product idea. Kudos. – Adrian Larson Mar 5 '16 at 4:10
3

If you are using Bamboo, I would use Dan's answer above and modify it to support your Salesforce workflow.

What I've done is to use the Atalassian's Salesforce build plans and modified it to inject a node.js task between the "Checkout Source Code" task and the ant "deployEmptyCheckOnly" task in the "Dev All Tests" plan. When setting it up use "index.js" as the Script and "master yourFeature ./deploy/" as the Arguments. Do not specify a working sub-directory and Dan's script should work just fine.

You may also want to modify the lib/metaUtils.js file from Dan's source code listed in his response above to include all of the metadata types and folder names that you plan to track via automated deployment.

You can also use my forked version at https://github.com/jasonerivera/sf-packager which already has this file updated with all of the currently known metadata types. Use at your own risk.

You'll want to take the source files and folders from sf-packager and put them in the root directory of your repository and check them in. You'll also want to update the build/build.xml file to point to the /deploy folder everywhere you see "deployRoot" as this is where the sf-packager script will build your final deployment package. Do not use /src as the deployRoot or else you will be deploying everything that you checked out of the repo which is what the Atlassian documentation has you doing by default.

deployRoot="${basedir}/deploy/*${sfdc.featureName}*/unpackaged"

This implies that ${sfdc.featureName} is a variable that will be the same as the yourFeature parameter you used in the arguments section of the node.js task you set up as that is the directory sf-packager will automatically create for you. It would be wise to keep this dynamic by using Bamboo repository variables to replace the value in your build.xml file (${bamboo.sfdc.feature.name}), especially if you will be using Plan Branching. You would in this case need to add "-Dsfdc.featureName=${bamboo.sfdc.feature.name}" to the end of the Environment variables section of the deployment ant task in the Bamboo plan you build.

Another feature you may want to turn on is Plan Branching. This will allow you to build a deployment package with only changes between your feature branch against the Dev/master branch. As long as the Salesforce org you are going to "deployEmptyCheckOnly" to (from the feature branch) is not the shared Dev org, then you can have multiple developers pulling from Dev/master automatically down into their feature/master by using automatic branch management (Branch updater), but running unit tests to a separate unadulterated staging box that looks like Dev (or production) without everyone waiting in line to run unit tests against Dev. This concept has proven to eliminate one of the most painful bottlenecks with regard to Salesforce CI in our org.

There are too many other details to post here on how to set it all up, but the key concepts I've listed above. If you follow the Atlassian Salesforce model to the "t" first, then it is easy to inject this script in a task to make building deployment packages automated. The only other thing to watch out for is to make sure you know the 'git diff' command and how it actually works from the Bamboo server once this task is running. I spent a lot of time wondering why I didn't see any changes between master and feature.

  • Jason: Your solution looks awesome. I have few questions and can you pls help me out on this actual implementation on Bamboo – Salesforce Learner Aug 2 '17 at 17:34
2

I would recommend setting up a Continuous Integration tool to deploy changes to your full sandbox automatically when they are committed to the Master branch.

Options:

Post any issues you encounter setting these up

  • We already have Bamboo CI in place for the master sandbox. My question is specific to how to keep individual developer sandboxes in sync with master branch. – javanoob Mar 3 '16 at 21:19
2

Salesforce and CI is a good start, although it's often far from perfect, especially if you have users who make config changes outside your repo.

Regarding “Workaround #1”, if keeping track of changes is a pain, you might want to look at something like Test Pilot, which lets you track changes and compare orgs as you go. (Full disclosure: I'm a developer.) You can just compare two orgs to see the list of metadata that needs updating, so it's fairly painless to do.

1

Below is my implementation of kurunve's idea from comments:

Pre-requisite:

We need to create a text file which holds the SHA of the commit till which the changes are deployed to our sandbox.

When there are any commits to the master branch by other developers, We need to get those commits into our local machine and deploy changes from these commits to our sandbox.

Below are the steps on how I implemented this:

  1. Run git pull command to bring changes from repo to local disk.
  2. Run the special ant command which internally invokes a python script and the python script does the below actions:

    a) Runs the command git diff --name-only <SHA_from_file> HEAD to identify what list of files were changed from the last commit that we deployed changes to our sandbox vs the HEAD.

    b) From the above list of files it copies each file to a temp folder by copying the file from the base src folder and simultaneously creating package.xml on the fly.

  3. Once the above command finishes successfully, it calls regular ant target deploy Unpackaged which deploys files from the temp folder created in step 2 to our sandbox.

  4. If the deploy is successful in the above command we call another ant target which calls second python script which writes the SHA of the HEAD to the file mentioned in the pre-requisite.

Below are the ant targets that I created and I start the process by just running ant deployChangesSinceLastDeploy

  <target name="createTargetFolder">
    <exec dir="." executable="python" failonerror="true">
      <arg line="createTargetFolder.py"/>
    </exec>
  </target>

  <target name="deployChanges" depends="createTargetFolder">
    <sf:deploy ../> <!-- the regular stuff here -->
  </target>

  <target name="deployChangesSinceLastDeploy" depends="deployChanges">
    <exec dir="." executable="python" failonerror="true">
      <arg line="writeSHA.py"/>
    </exec>
  </target>

With this approach, I don't need to keep track of what files are changed by looking into each commit as that is taken care by python script creating temp folder with the list of files changed from last commit to HEAD.

If any body has any other suggestions or easy alternatives..Pleas comment. Thanks!

1

I have also ran into issues deploying the entire codebase so I created a quick tool to easily create packages from branch diffs. I currently use this to create "feature" packages and deploy through ant passing the directory via parameter to the ant build. You could potentially explore wrapping this in a git hook to automate this when new code is pulled.

https://github.com/dlively1/sf-packager

sfpackage master featureBranch ./deploy/ && ant featureDeploy -DfeatureName=featureBranch

<property name="featureName" value=""/>        
<target name="featureDeploy">
        <sf:deploy username="${sf.username}" password="${sf.password}" sessionId="${sf.sessionId}" serverurl="${sf.serverurl}"
            maxPoll="${sf.maxPoll}" deployRoot="./deploy/${featureName}/unpackaged" rollbackOnError="true" checkOnly="false" />
    </target>
  • I wish you answered few days ago because I almost did a version of this using python script. But your solution looks more solid. – javanoob Mar 10 '16 at 3:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.