Config management for Salesforce.com - Extracting all the metadata

Question

We are trying to maintain source code repository for our Salesforce.com project (predominantly configuration). We are using Subversion (SVN) for this. The idea is to extract all the metadata in Eclipse and commit to SVN. The plan is also to use Force.com migration tool to do deployments (all components)

In eclipse, we cannot subscribe to all the metadata components and have to do it in multiple steps (select few components, extract then select the other components). However when doing this, we see that the metadata xml are not fully extracted (for example the XML for profiles doesnt contain all references). There is also limit on number of components that can be retrieved.

What approach do you follow to get all the metadata? ANT script to retrieve in batches?

Please let me know. Appreciate any references/samples in this regard.

Cheers Siva

Answer 1

I think what you're looking for is the bulkRetrieve macro in the Force.com Migration Tool. You can pass it a package.xml file listing the metadata types you wish to retrieve and it will pull them all as a single asynchronous process into a local directory.

You can see an example of this being used to fetch all unpackaged metadata from an organization here via the Force.com Migration Tool's bulkRetrieve macro: https://github.com/SalesforceFoundation/CumulusCI/blob/legacy-1.0/build/cumulusci.xml#L148

That ant macro uses the following package.xml file which lists all metadata types: https://github.com/SalesforceFoundation/CumulusCI/blob/legacy-1.0/build/all_meta_types.xml

The repository I linked to also includes all our build scripts with plenty of examples of different deployment scenarios.

Regarding some configuration not pulling down with the profile, I'd give the bulkRetrieve macro a shot to see if it grabs what you want. We're not deploying profiles as part of our project so I can't speak to what parts of the profile can be managed through the metadata api. If you can post specific components of the profile which bulkRetrieve cannot fetch, I'd be happy to look into them as I will probably need to know in the future for my own projects.

You might also consider using Jenkins to setup automation of the builds and even better a dedicated test org to test commits to SVN. You can find documentation about how we did this using GitHub (though much of it applies to SVN as well with some modification) here: http://salesforcefoundation.github.io/CumulusCI/

Answer 2

The detailed behaviour of both the Force IDE and the Ant migration tools is defined by the content of the package.xml file. This is updated by the "Add/Remove Metadata Components" wizard of the Force IDE and can also be manually edited. The best reference information about package.xml is contained in the Metadata API Developer’s Guide. Note that very occasionally you will find a component that the API does not support and such configuration has to be tracked and applied manually.

We make manual changes to this file and and transfer from org to Eclipse project and vice versa using the Force IDE. We checkin to SVN using Eclipse. To make sure the components are consistent, we use Jenkins (a continuous integration product) to automatically deploy what is in SVN to a separate org and run all the tests using the Ant migration tools. We typically deploy to production orgs using the Ant migration tools (and also managed packages).

Profiles are very much change hotspots and are automatically filtered by package.xml. If your profiles are referencing components from multiple namespaces it is hard to get a complete profile; we had to write our own tooling to accomplish that.

With package.xml you have to opt-in to get components transferred. Sometimes this is best done with wildcards and in other cases by explicit naming. (It can sometimes be difficult to work out the correct name to use.)

Below is a cut-down example of a package.xml for a project that has many different types of components:

<?xml version="1.0" encoding="UTF-8"?>
<Package xmlns="http://soap.sforce.com/2006/04/metadata">
    <types>
        <members>*</members>
        <name>ApexClass</name>
    </types>
    <types>
        <members>*</members>
        <name>ApexComponent</name>
    </types>
    <types>
        <members>*</members>
        <name>ApexPage</name>
    </types>
    <types>
        <members>*</members>
        <name>ApexTrigger</name>
    </types>
    <types>
        <members>*</members>
        <name>CustomApplication</name>
    </types>
    <types>
        <members>Activity.Closed__c</members>
        <name>CustomField</name>
    </types>
    <types>
        <members>*</members>
        <members>Account</members>
        <name>CustomObject</name>
    </types>
    <types>
        <members>*</members>
        <name>CustomPageWebLink</name>
    </types>
    <types>
        <members>*</members>
        <name>CustomTab</name>
    </types>
    <types>
        <members>Images/Logo.png</members>
        <name>Document</name>
    </types>
    <types>
        <members>EmailTemplates</members>
        <members>EmailTemplates/InquiryPortal</members>
        <name>EmailTemplate</name>
    </types>
    <types>
        <members>*</members>
        <name>HomePageComponent</name>
    </types>
    <types>
        <members>*</members>
        <name>HomePageLayout</name>
    </types>
    <types>
        <members>Account-Account Layout</members>
        <name>Layout</name>
    </types>
    <types>
        <members>Analyst</members>
        <name>Profile</name>
    </types>
    <types>
        <members>ExtraReports</members>
        <members>ExtraReports/AnyOccupation</members>
        <name>Report</name>
    </types>
    <types>
        <members>*</members>
        <name>ReportType</name>
    </types>
    <types>
        <members>*</members>
        <name>StaticResource</name>
    </types>
    <types>
        <members>Contact.SelectTitle</members>
        <name>WebLink</name>
    </types>
    <version>28.0</version>
</Package>