Resetting my development org to an empty state

Question

I would like to clear out my development org of all artefacts on a regular basis and then do a deploy of the development package from my local source.

I realise there are retrieve and list tasks available as well as the destructive changes file that I could make use of. Unfortunately, I have found that constructing an appropriate build file to co-ordinate all this has become somewhat confusing. The radical approach might be spinning up a new org each time but that seems rather wasteful.

All suggestions would be greatly appreciated

Answer 1

Presently, use SFDX with a Dev Hub to create empty time-limited Scratch Orgs.

Historically - can do this using just the out-of-the-box Force.com Migration Tool.

As you noted, destroying metadata components is achieved with a destructiveChanges.xml definition. However this differs from package.xml in that it doesn't support wildcards (*) - so the intermediate hurdle is the generation of an explicit definition containing the names of all the metadata to destroy.

This explicit definition can be generated by listing all the organization's metadata by type to get the actual component names, then combining the listings into into a named package.

Using a few regular expessions, we can bust dependencies like webLinks and quickActions, then deploy the whole lot of destructive changes in a single metadata transaction:

<target name="destroy">
    <destroy
        username="${sf.username}"
        password="${sf.password}"
        serverurl="${sf.serverurl}"
    />
</target>

Here's a macrodef - drop this into your build.xml then run ant destroy:

<macrodef name="destroy" description="Destroys all metadata in an organization - Revision 21">

    <attribute name="username" />
    <attribute name="password" />
    <attribute name="serverurl" default="https://login.salesforce.com" />
    <attribute name="tempDir" default="temp/destroy" description="Directory to write metadata." />
    <attribute name="apiVersion" default="41.0" />

    <sequential>

        <!-- prompt user to confirm -->
        <input message="THIS TASK IRREVERSIBLY DESTROYS ALL METADATA. ARE YOU SURE?" validargs="@{serverurl}/?un=@{username}" />

        <!-- http api helper -->
        <macrodef name="soapcall">
            <text name="request" />
            <attribute name="endpoint" />
            <attribute name="tempfile" default="" />
            <attribute name="soapaction" default="&quot;&quot;" />
            <sequential>
                <local name="request" />
                <property name="request" value="@{request}" />
                <script language="javascript">with (new JavaImporter(java.net, java.io)) {
                    var line, result = '', connection = new URL('@{endpoint}').openConnection();
                    connection.setDoOutput(true);
                    connection.setRequestMethod('POST');
                    connection.setRequestProperty('Content-Type', 'text/xml');
                    connection.setRequestProperty('SOAPAction', '@{soapaction}');
                    var writer = new OutputStreamWriter(connection.getOutputStream());
                    writer.write(project.getProperty('request')); writer.flush(); //request
                    var reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
                    while ((line = reader.readLine()) != null) result += line + '\n'; reader.close(); //response
                    var echo = project.createTask('echo');
                    if ('@{tempfile}') echo.setFile(new File('@{tempfile}'));
                    echo.setMessage(result);
                    echo.perform();
                }</script>
            </sequential>
        </macrodef>

        <!-- http soap login -->
        <local name="loginResponse.tmp" />
        <tempfile property="loginResponse.tmp" prefix="loginResponse" suffix=".tmp" createfile="true" deleteonexit="true" />
        <soapcall tempfile="${loginResponse.tmp}" endpoint="@{serverurl}/services/Soap/u/@{apiVersion}" soapaction="login"><![CDATA[
            <Envelope xmlns="http://schemas.xmlsoap.org/soap/envelope/">
                <Body>
                    <login xmlns="urn:partner.soap.sforce.com">
                        <username>@{username}</username>
                        <password>@{password}</password>
                    </login>
                </Body>
            </Envelope>
        ]]></soapcall>

        <!-- parse endpoint -->
        <local name="loginUrl" />
        <loadfile property="loginUrl" srcFile="${loginResponse.tmp}">
            <filterchain><tokenfilter><filetokenizer/><replaceregex flags="gs" pattern=".*(https://[^/]+).*" replace="\1" /></tokenfilter></filterchain>
        </loadfile>

        <!-- parse session -->
        <local name="sessionId" />
        <loadfile property="sessionId" srcFile="${loginResponse.tmp}">
            <filterchain><tokenfilter><filetokenizer/><replaceregex flags="gs" pattern=".*&lt;sessionId&gt;([^&lt;]+)&lt;/sessionId&gt;.*" replace="\1" /></tokenfilter></filterchain>
        </loadfile>

        <!-- stop all running jobs and clear roles and permission set dependencies with execanon -->
        <soapcall endpoint="${loginUrl}/services/Soap/T/@{apiVersion}"><![CDATA[
            <Envelope xmlns="http://schemas.xmlsoap.org/soap/envelope/">
                <Header>
                    <SessionHeader xmlns="urn:tooling.soap.sforce.com">
                        <sessionId>${sessionId}</sessionId>
                    </SessionHeader>
                </Header>
                <Body>
                    <executeAnonymous xmlns="urn:tooling.soap.sforce.com">
                        <String>
                            delete [SELECT Id FROM PermissionSetAssignment WHERE PermissionSet.ProfileId = null];
                            for (SObject cron : [SELECT Id FROM CronTrigger]) System.abortJob(cron.Id);
                            User[] users = [SELECT Id FROM User WHERE UserRoleId != null];
                            for (User user : users) user.UserRoleId = null;
                            update users;
                        </String>
                    </executeAnonymous>
                </Body>
            </Envelope>
        ]]></soapcall>

        <!-- reset working directory -->
        <delete dir="@{tempDir}" />
        <mkdir dir="@{tempDir}" />

        <!-- determines org shape -->
        <local name="describeMetadataResult.tmp" />
        <tempfile property="describeMetadataResult.tmp" prefix="describeMetadataResult" suffix=".tmp" createfile="true" deleteonexit="true" />
        <sf:describeMetadata serverurl="${loginUrl}" sessionid="${sessionId}" resultFilePath="${describeMetadataResult.tmp}" />

        <!-- clean metadata descriptions -->
        <local name="metadataTypes.tmp" />
        <tempfile property="metadataTypes.tmp" prefix="metadataTypes" suffix=".tmp" createfile="true" deleteonexit="true" />
        <concat destFile="${metadataTypes.tmp}">
            <fileset file="${describeMetadataResult.tmp}" />
            <filterchain>
                <linecontainsregexp><regexp pattern="ChildObjects|XMLName" /></linecontainsregexp>
                <tokenfilter><replacestring from="," to="${line.separator}"/></tokenfilter>
                <tokenfilter><replacestring from="ChildObjects: " to=""/></tokenfilter>
                <tokenfilter><replacestring from="XMLName: " to=""/></tokenfilter>
                <tokenfilter><replacestring from="*" to=""/></tokenfilter>
                <tokenfilter><ignoreblank/></tokenfilter>
                <sortfilter/>
            </filterchain>
        </concat>

        <!-- lists by type with regex filter -->
        <macrodef name="listMetadataForDestroy">
            <attribute name="negate" />
            <attribute name="pattern" />
            <attribute name="metadataType" />
            <sequential>
                <echo>Preparing destructiveChangesPost.xml - @{metadataType}</echo>
                <local name="listMetadataResult.tmp" />
                <tempfile property="listMetadataResult.tmp" prefix="@{metadataType}" suffix=".tmp" createfile="true" deleteonexit="true" />
                <sf:listMetadata serverurl="${loginUrl}" sessionid="${sessionId}" metadataType="@{metadataType}" resultFilePath="${listMetadataResult.tmp}" />
                <concat destFile="@{tempDir}/destructiveChangesPost.xml" append="true">
                    <fileset file="${listMetadataResult.tmp}" />
                    <header filtering="false"><![CDATA[${line.separator}<types>${line.separator}    <name>@{metadataType}</name>${line.separator}]]></header>
                    <filterchain>
                        <linecontains><contains value="FullName/Id" /></linecontains>
                        <replaceregex pattern="FullName/Id: (.+)/.*" replace="&lt;members&gt;\1&lt;/members&gt;" />
                        <linecontainsregexp negate="@{negate}"><regexp pattern="@{pattern}" /></linecontainsregexp>
                    </filterchain>
                    <footer filtering="false"><![CDATA[</types>]]></footer>
                </concat>
            </sequential>
        </macrodef>

        <!-- open destructive changes definition -->
        <echo file="@{tempDir}/destructiveChangesPost.xml"><![CDATA[<Package>]]></echo>

        <!-- iterates over (most) metadata types -->
        <loadfile property="" srcFile="${metadataTypes.tmp}">
            <filterchain>
                <!-- AppMenu - AppSwitcher.appmenu - Error: The AppMenu called 'AppSwitcher' is standard and cannot be deleted -->
                <!-- AssignmentRules - Case.assignmentRules - Error: The AssignmentRules called 'Case' is standard and cannot be deleted -->
                <!-- AutoResponseRules - Lead.autoResponseRules - Error: The AutoResponseRules called 'Lead' is standard and cannot be deleted -->
                <!-- Certificate - SelfSignedCert.crt - Error: We can't delete this certificate because your Identity Provider is using it -->
                <!-- CleanDataService - DataCloudCompanyMatch - Error: You can't delete default data integration rule -->
                <!-- Community - Zone.community - Error: invalid parameter value -->
                <!-- ConnectedApp - Connected Apps may be used in use by other orgs so don't trash them -->
                <!-- CustomSite - MyCustom.site - Error: insufficient access rights on cross-reference id -->
                <!-- CustomObjectTranslation - MyMeta__mdt-en_US - Error: The CustomObjectTranslation called 'MyMeta__mdt-en_US' is standard and cannot be deleted -->
                <!-- EscalationRules - Case.escalationRules: - Error: The EscalationRules called 'Case' is standard and cannot be deleted -->
                <!-- Flow - TaskNotify.flowDefinition - Error: insufficient access rights on cross-reference id -->
                <!-- InstalledPackage - Error: cannot modify managed object: state=installed -->
                <!-- MatchingRules - Account.matchingRule: - Error: Matching Rules have to be deleted individually -->
                <!-- RecordType - Metric.Completion - Error: Cannot delete record type through API -->
                <!-- SharingRules - Account.sharingRules - Error: The SharingRules called 'Account' is standard and cannot be deleted -->
                <!-- TopicsForObjects - Error: Entity type 'TopicsForObjects' is not available for delete in this api version -->
                <!-- Workflow - Account.workflow - Error: Cannot delete a workflow object; Workflow Rules and Actions must be deleted individually -->
                <linecontainsregexp negate="true">
                    <regexp pattern="AppMenu|AssignmentRules|AutoResponseRules|Certificate|CleanDataService|Community|ConnectedApp|CustomSite|CustomObjectTranslation|EscalationRules|Flow|InstalledPackage|MatchingRules|RecordType|SharingRules|TopicsForObjects|Workflow" />
                </linecontainsregexp>
                <!-- Layout - Remove only: Account-Account %28Marketing%29 and WorkFeedback-Feedback Layout - Summer %2715 etc -->
                <!-- Profile - Remove only: Custom: Marketing Profile and Custom: Support Profile and Custom: Sales Profile etc -->
                <!-- ListView - Leaves behind: Activity.All, Asset.All, Campaign.All, Contract.All, Product2.All, User.All etc -->
                <!-- ApexPage - Leaves behind: SiteHome Visualforce Page which is required in orgs containing Force.com Sites -->
                <!-- CustomField - Leaves behind: BigObject Customer_Interaction__b.Score_This_Game__c etc -->
                <!-- MatchingRule - Leaves behind: Lead.Standard_Lead_Match_Rule_v1_0 and Account.Standard_Account_Match_Rule_v1_0 etc -->
                <!-- CustomObject - Remove only: Big Objects, Custom Objects, Platform Events, External Objects etc -->
                <!-- BusinessProcess - Leaves behind: Case.master etc -->
                <!-- CustomApplication - Leaves behind: standard__AppLauncher etc -->
                <scriptfilter language="javascript">
                    var negate = false, pattern = '.*', metadataType = self.getToken();
                    if ('Layout' == metadataType) (negate = false) | (pattern = '%27|%28|%29');
                    if ('Profile' == metadataType) (negate = false) | (pattern = 'Custom%3A');
                    if ('ListView' == metadataType) (negate = true) | (pattern = '\\.All&lt;/members&gt;');
                    if ('ApexPage' == metadataType) (negate = true) | (pattern = '&gt;SiteHome&lt;');
                    if ('CustomField' == metadataType) (negate = true) | (pattern = '__b\\.');
                    if ('MatchingRule' == metadataType) (negate = true) | (pattern = 'Standard_');
                    if ('CustomObject' == metadataType) (negate = false) | (pattern = '__b|__c|__e|__x|__mdt');
                    if ('BusinessProcess' == metadataType) (negate = true) | (pattern = 'master');
                    if ('CustomApplication' == metadataType) (negate = true) | (pattern = 'standard__');
                    var macro = project.createTask('listMetadataForDestroy');
                    macro.setDynamicAttribute('negate', negate);
                    macro.setDynamicAttribute('pattern', pattern);
                    macro.setDynamicAttribute('metadatatype', metadataType);
                    macro.execute(); //dynamic attributes are lowercase insistent
                </scriptfilter>
            </filterchain>
        </loadfile>

        <!-- close destructive changes definition -->
        <echo append="true" file="@{tempDir}/destructiveChangesPost.xml"><![CDATA[</Package>]]></echo>

        <!-- retrieves by type and regex replaces -->
        <macrodef name="bulkRetrieveForDestroy">
            <attribute name="metadataType" />
            <attribute name="directoryName" />
            <attribute name="pattern" />
            <attribute name="expression" />
            <sequential>
                <echo>Preparing package.xml - @{metadataType}</echo>
                <mkdir dir="@{tempDir}/@{directoryName}" />
                <sf:bulkRetrieve serverurl="${loginUrl}" sessionid="${sessionId}" retrieveTarget="@{tempDir}" metadataType="@{metadataType}" batchSize="10000" />
                <replaceregexp flags="gs">
                    <fileset dir="@{tempDir}/@{directoryName}" />
                    <regexp pattern="@{pattern}" />
                    <substitution expression="@{expression}" />
                </replaceregexp>
            </sequential>
        </macrodef>

        <!-- fix layout custom links - Error: This WebLink is referenced elsewhere in salesforce.com -->
        <bulkRetrieveForDestroy
            metadataType="Layout"
            directoryName="layouts"
            pattern="&lt;layoutItems&gt;\s+&lt;customLink&gt;[^&lt;]+&lt;/customLink&gt;\s+&lt;/layoutItems&gt;"
            expression="&lt;!--\0--&gt;"
        />

        <!-- fix layout custom buttons - Error: This WebLink is referenced elsewhere in salesforce.com - Order-Order Layout -->
        <replaceregexp flags="gs">
            <fileset dir="@{tempDir}/layouts" />
            <regexp pattern="&lt;customButtons&gt;[^&lt;]+&lt;/customButtons&gt;" />
            <substitution expression="&lt;!--\0--&gt;" />
        </replaceregexp>
        
        <!-- fix layout custom actions - Error: Cannot delete action EditDescription. The following layout is referencing this. : Task Layout. -->
        <replaceregexp flags="gs">
            <fileset dir="@{tempDir}/layouts" />
            <regexp pattern="&lt;quickActionList&gt;.+&lt;/quickActionList&gt;" />
            <substitution expression="&lt;!--\0--&gt;" />
        </replaceregexp>

        <!-- fix profile default apps - Error: Unable to delete custom app. Profiles are using this custom app as default -->
        <bulkRetrieveForDestroy
            metadataType="Profile"
            directoryName="profiles"
            pattern="&lt;userPermissions&gt;.*&lt;/userPermissions&gt;"
            expression="&lt;applicationVisibilities&gt;&lt;application&gt;standard__AppLauncher&lt;/application&gt;&lt;default&gt;true&lt;/default&gt;&lt;visible&gt;true&lt;/visible&gt;&lt;/applicationVisibilities&gt;"
        />

        <!-- fix role parents - Error: Your attempt to delete the role could not be completed because at least one role reports to that role -->
        <bulkRetrieveForDestroy
            metadataType="Role"
            directoryName="roles"
            pattern="&lt;parentRole&gt;[^&lt;]+&lt;/parentRole&gt;"
            expression="&lt;!--\0--&gt;"
        />

        <!-- fix object listviews - Error: cannot delete last filter -->
        <bulkRetrieveForDestroy
            metadataType="ListView"
            directoryName="objects"
            pattern="&lt;listViews&gt;.*&lt;/listViews&gt;"
            expression="&lt;listViews&gt;&lt;fullName&gt;All&lt;/fullName&gt;&lt;filterScope&gt;Everything&lt;/filterScope&gt;&lt;label&gt;All&lt;/label&gt;&lt;/listViews&gt;"
        />

        <!-- fix site dependencies - Error: This static resource is referenced elsewhere in salesforce.com. Remove the usage and try again -->
        <bulkRetrieveForDestroy
            metadataType="CustomSite"
            directoryName="sites"
            pattern="&lt;CustomSite xmlns=&quot;http://soap.sforce.com/2006/04/metadata&quot;&gt;.*&lt;active&gt;([^&lt;]+)&lt;/active&gt;.*&lt;allowStandardPortalPages&gt;([^&lt;]+)&lt;/allowStandardPortalPages&gt;.*&lt;clickjackProtectionLevel&gt;([^&lt;]+)&lt;/clickjackProtectionLevel&gt;.*&lt;indexPage&gt;([^&lt;]+)&lt;/indexPage&gt;.*&lt;masterLabel&gt;([^&lt;]+)&lt;/masterLabel&gt;.*&lt;requireHttps&gt;([^&lt;]+)&lt;/requireHttps&gt;.*&lt;siteType&gt;([^&lt;]+)&lt;/siteType&gt;.*&lt;subdomain&gt;([^&lt;]+)&lt;/subdomain&gt;.*&lt;/CustomSite&gt;"
            expression="&lt;CustomSite xmlns=&quot;http://soap.sforce.com/2006/04/metadata&quot;&gt;${line.separator}&lt;active&gt;\1&lt;/active&gt;${line.separator}&lt;allowStandardPortalPages&gt;\2&lt;/allowStandardPortalPages&gt;${line.separator}&lt;clickjackProtectionLevel&gt;\3&lt;/clickjackProtectionLevel&gt;${line.separator}&lt;indexPage&gt;SiteHome&lt;/indexPage&gt;${line.separator}&lt;masterLabel&gt;\5&lt;/masterLabel&gt;${line.separator}&lt;requireHttps&gt;\6&lt;/requireHttps&gt;${line.separator}&lt;siteType&gt;\7&lt;/siteType&gt;${line.separator}&lt;subdomain&gt;\8&lt;/subdomain&gt;${line.separator}&lt;urlPathPrefix&gt;\5&lt;/urlPathPrefix&gt;${line.separator}&lt;/CustomSite&gt;"
        />

        <!-- fix site index pages - Error: Required field is missing: indexPage -->
        <local name="NumberOfSites.tmp" />
        <condition property="NumberOfSites.tmp" else=""><resourcecount when="ne" count="0"><fileset dir="@{tempDir}/sites" /></resourcecount></condition>
        <resourcecount property="NumberOfSites.tmp"><fileset dir="@{tempDir}/sites" /></resourcecount>
        <script language="javascript">with (new JavaImporter(java.io)) {
            if (project.getProperty('NumberOfSites.tmp')) {
                var page = project.createTask('echo');
                page.setFile(new File('@{tempDir}/pages/SiteHome.page'));
                page.setMessage('&lt;apex:page/&gt;');
                page.perform();
                var meta = project.createTask('echo');
                meta.setFile(new File('@{tempDir}/pages/SiteHome.page-meta.xml'));
                meta.setMessage('&lt;ApexPage&gt;&lt;label&gt;SiteHome&lt;/label&gt;&lt;/ApexPage&gt;');
                meta.perform();
            }
        }</script>

        <!-- fix support setting queue dependencies - Error: cannot delete queue that is in use -->
        <mkdir dir="@{tempDir}/settings" />
        <echoxml file="@{tempDir}/settings/Case.settings" namespacePolicy="all">
            <CaseSettings>
                <defaultCaseOwner>@{username}</defaultCaseOwner>
                <defaultCaseOwnerType>User</defaultCaseOwnerType>
            </CaseSettings>
        </echoxml>

        <!-- fix big objects - Error: Custom BigObjects do not support layouts -->
        <delete><fileset dir="@{tempDir}" includes="**/*__b*" /></delete>

        <!-- MANUAL COMPONENTS -->
        <!-- CustomSite URL Rewriter is not available through Metadata API -->
        <!-- Inbound Email Services are not available through Metadata API -->
        <!-- Lead Settings behaviour is not available through Metadata API -->

        <!-- strips namespaced components -->
        <macrodef name="unspecifyForDestroy">
            <attribute name="namespacePrefix" />
            <sequential>
                <echo>Stripping namespaced components: @{namespacePrefix}</echo>
                <replaceregexp
                    flags="gm"
                    file="@{tempDir}/destructiveChangesPost.xml"
                    match="&lt;members&gt;[^&lt;]*@{namespacePrefix}__[^&lt;]+&lt;/members&gt;"
                    replace="&lt;!--\0--&gt;"
                />
                <delete><fileset dir="@{tempDir}" includes="**/@{namespacePrefix}__*" /></delete>
            </sequential>
        </macrodef>

        <!-- iterates over all namespace prefixes -->
        <echo>Listing installed packages...</echo>
        <local name="InstalledPackage.tmp" />
        <tempfile property="InstalledPackage.tmp" prefix="InstalledPackage" suffix=".tmp" createfile="true" deleteonexit="true" />
        <sf:listMetadata serverurl="${loginUrl}" sessionid="${sessionId}" metadataType="InstalledPackage" resultFilePath="${InstalledPackage.tmp}" />
        <loadfile property="" srcFile="${InstalledPackage.tmp}">
            <filterchain>
                <linecontains><contains value="FullName/Id" /></linecontains>
                <replaceregex pattern="FullName/Id: (.+)/.*" replace="\1" />
                <sortfilter />
                <uniqfilter />
                <scriptfilter language="javascript">
                    var macro = project.createTask('unspecifyForDestroy');
                    macro.setDynamicAttribute('namespaceprefix', self.getToken());
                    macro.execute(); //dynamic attributes are lowercase insistent
                </scriptfilter>
            </filterchain>
        </loadfile>

        <!-- create package definition for fixes -->
        <echoxml file="@{tempDir}/package.xml" namespacePolicy="all">
            <Package>
                <version>41.0</version>
                <types>
                    <name>ApexPage</name>
                    <members>*</members>
                </types>
                <types>
                    <name>CustomSite</name>
                    <members>*</members>
                </types>
                <types>
                    <name>Layout</name>
                    <members>*</members>
                </types>
                <types>
                    <name>ListView</name>
                    <members>*</members>
                </types>
                <types>
                    <name>Profile</name>
                    <members>*</members>
                </types>
                <types>
                    <name>Role</name>
                    <members>*</members>
                </types>
                <types>
                    <name>Settings</name>
                    <members>*</members>
                </types>
            </Package>
        </echoxml>

        <!-- destroy! -->
        <sf:deploy
            serverurl="${loginUrl}"
            sessionid="${sessionId}"
            deployRoot="@{tempDir}"
            ignoreWarnings="true"
            singlePackage="true"
            purgeOnDelete="true"
        />

    </sequential>

</macrodef>

(You will need to set sf.username and sf.password as properties or command line parameters)

It boils down to a four step process:

1. use sf:describeMetadata to describe the feature "shape" of the org,

2. list all components of all the above metadata types into a manifest,

3. unlink certain specific dependencies such as buttons on page layouts,

4. deploy using destructiveChanges.xml which destroys the contents,

Stefan Goor presented "Techniques and Tools to Improve the Salesforce Development Cycle" - with a process of cleaning orgs as part of continuous integration.

Answer 2

There is an undeploy script here that may be of some help to you.

You could also consider making use of developer sandboxes which you can delete / refresh yourself, although there are some differences between developer sandboxes and true DE orgs.