14

I would like to configure the ant migration tool so that only test classes are run that we own so no managed package classes. Now i searched on the documentation of the tool and i only find the runAllTests = true, which runs all tests including managed package code.

then i found 2 other posts that are talking about the sf:compileAndTest method in ant where you can specify <runtests namespace="Default"/> but this doesn't seem to work either. there is a post here Get Status Updates When Running Tests with ANT Migration Tool

and here How to run tests in the default namespace in a sandbox using Ant

but both don't really answer the question if the <runTests namespace="Default"/> actually works in ant or not.

does this work?

UPDATE:

I checked further the runAllTests=True and seems to run all test classes also the ones from a managed package. I tried with the sample build.xml and the latest Migration toolkit.

<target name="deployCodeCheckOnly">
  <sf:deploy username="[email protected]" password="password" serverurl="https://test.salesforce.com" maxPoll="500" deployRoot="codepkg" runAllTests="true" checkOnly="true"/>
</target>

As outcome of this run it gives me these test class failures which are from managed packages like CRMFusion's DupeBlocker and others

[sf:deploy] Test Failures:
[sf:deploy] 198.  CRMfusionDBR101.DB_UnitTests.testRebuildKeys_MultipleLeads_MultiBatch_SameHash_DupeFirstBatch -- System.DmlException: Insert failed. First exception on row 0; first error: FIELD_CUSTOM_VALIDATION_EXCEPTION, Please enter valid country 

[sf:deploy] Request Status: InProgress (487/689)  -- Running Test: DB_UnitTests.test_DB_KeyBuilder_useBatchApexKeyBuilder
[sf:deploy] Request Status: InProgress (493/689)  -- Running Test: DB_UnitTests.test_Db2Upgrade
[sf:deploy] Request Status: InProgress (500/689)  -- Running Test: DB_UnitTests.test_PostInstall
2
  • Just checking, are you running the latest ant-salesforce.jar, API 29.0? Commented Feb 10, 2014 at 15:39
  • Yes to be sure i downloaded the migration toolkit again from my developer org and tried to do a checkOnly with runalltests on my local machine with the sample build.xml file included in the download. and it is the same result Commented Feb 10, 2014 at 15:44

4 Answers 4

12

Edit: Summer '15 made this "all better". If you upgrade your ant-salesforce.jar to one that supports API version 34, you can now pass a "testLevel=RunLocalTests" attribute to sf:deploy like so:

<sf:deploy
  testLevel="RunLocalTests" 
  username="${sf.username}" 
  password="${sf.password}" 
  serverUrl="${sf.server}" 
  deployRoot="src" 
  />

From the docs:

http://releasenotes.docs.salesforce.com/en-us/summer15/release-notes/rn_deployment_run_subset_of_tests.htm

As part of this change, the runAllTests deployment option is now replaced with testLevel. You can choose which tests to run in a deployment by setting the desired test level. For a description of all test levels, see test levels for the deploy() call. In particular, to run a subset of tests in a deployment, set testLevel to the RunSpecifiedTests value and specify the tests to run in the runTests option.

And

http://releasenotes.docs.salesforce.com/en-us/summer15/release-notes/rn_api_meta_new_calls.htm#testlevels

RunLocalTests—All tests in your organization are run, except the ones that originate from installed managed packages. This test level is the default for production deployments that include Apex classes or triggers.

This was the workaround for pre-Summer '15 for posterity:

Because I killed off the better part of a day hacking around in ant to accomplish the workaround @pepefloyd suggested, I wanted to publish a working example; In my case, the test classes are prefixed with "Test_" but you can fool around with the fileset as necessary

This is tested against v30 of the Force.com ant migration tool

<target name="test">
    <sfCompileAndTestUnmanaged checkOnly="true" username="${sf.username}" password="${sf.password}" server="${sf.server}">
        <fileset dir="src/classes">
            <include name="**/Test_*.cls"/>
        </fileset>
    </sfCompileAndTestUnmanaged>
</target>

<scriptdef name="sfCompileAndTestUnmanaged" language="javascript">
    <attribute name="checkonly"/>
    <attribute name="username"/>
    <attribute name="password"/>
    <attribute name="server"/>
    <attribute name="trace"/>
    <element name="fileset" type="fileset"/>

    <![CDATA[
        var filesets = elements.get("fileset");
        var filesetsIterator = filesets.iterator();
        var projectClasses = [];
        while(filesetsIterator.hasNext()){
            var fs = filesetsIterator.next();
            var iter = fs.iterator();
            while(iter.hasNext()){
                var resource = iter.next();
                var clazz = resource.getName().replace(".cls","");
                self.log("CLASS: " + clazz);
                projectClasses.push(clazz);
            }
        }

        var task = project.createTask("antlib:com.salesforce:compileAndTest");
        task.setCheckonly(attributes.get("checkonly") == 'true');
        task.setUsername(attributes.get("username"));
        task.setPassword(attributes.get("password"));
        task.setServer(attributes.get("server"));
        task.setTrace(attributes.get("trace") == 'true');

        //I tried 'importPackage' and conventional instantiation but couldn't get the inner class to instantiate; this works though
        var testsElement = task.getClass().getClassLoader().loadClass("com.salesforce.ant.CompileAndTest$RunTestsElement").newInstance();
        task.addRunTests(testsElement);

        var classClazz = task.getClass().getClassLoader().loadClass("com.salesforce.ant.CompileAndTest$CodeNameElement");
        for(i in projectClasses){
            var clazz = classClazz.newInstance();
            clazz.addText(projectClasses[i]);
            testsElement.addClass(clazz);
        }

        task.execute();
     ]]>
</scriptdef>

Alternatively, we've actually used a version of 'deploy' that does a no-op deploy (the package.xml in the 'ant' directory is empty except for the <version> element. This gives us incremental updates as the tests are running and allows the deployment of the source files to succeed in a separate ant target (not described below) while the tests may fail (insufficient code coverage, failing assertions, etc.)

<target name="test">
    <sfDeployUnmanaged purgeOnDelete="true" ignoreWarnings="true" username="${sf.username}" password="${sf.password}" serverUrl="${sf.server}" deployRoot="ant" maxPoll="75">
        <fileset dir="src/classes">
            <include name="**/*.cls"/>
        </fileset>
    </sfDeployUnmanaged>
</target>

<scriptdef name="sfDeployUnmanaged" language="javascript">
    <attribute name="purgeondelete"/>
    <attribute name="ignorewarnings"/>
    <attribute name="username"/>
    <attribute name="password"/>
    <attribute name="serverurl"/>
    <attribute name="deployroot"/>
    <attribute name="maxpoll"/>
    <attribute name="trace"/>
    <element name="fileset" type="fileset"/>

    <![CDATA[
        var filesets = elements.get("fileset");
        var filesetsIterator = filesets.iterator();
        var projectClasses = [];
        while(filesetsIterator.hasNext()){
            var fs = filesetsIterator.next();
            var iter = fs.iterator();
            while(iter.hasNext()){
                var resource = iter.next();
                var clazz = resource.getName().replace(".cls","");
                self.log("CLASS: " + clazz);
                projectClasses.push(clazz);
            }
        }

        var task = project.createTask("antlib:com.salesforce:deploy");
        task.setPurgeOnDelete(attributes.get("purgeondelete") == 'true');
        task.setIgnoreWarnings(attributes.get("ignorewarnings") == 'true');
        task.setUsername(attributes.get("username"));
        task.setPassword(attributes.get("password"));
        task.setServerURL(attributes.get("serverurl"));
        task.setDeployRoot(attributes.get("deployroot"));
        task.setMaxPoll(attributes.get("maxpoll"));
        task.setTrace(attributes.get("trace") == 'true');

        //Blows up when build timeout is reached if we don't set this (it uses this value when formatting the exception it throws)
        task.setOwningTarget(self.owningTarget);

        var classClazz = task.getClass().getClassLoader().loadClass("com.salesforce.ant.DeployTask$CodeNameElement");
        for(i in projectClasses){
            var clazz = classClazz.newInstance();
            clazz.addText(projectClasses[i]);
            task.addRunTest(clazz);
        }

        task.execute();
     ]]>
</scriptdef>
2
+50

It is strange that the sf:compileAndTest Ant task is no longer (since i am sure i've seen it in the past) documented either in the official Migration Toolkit docs or the readme.html you get with the download. I was also surprised to learn that its not formally part of the Metadata API, commonly used by the Ant tools under the covers.

Anyway, I've tracked down what seems to be at least a documented explanation regarding the namespace="Default" support. In production environments its basically ignored as per the documentation for the underlying Apex SOAP API the Ant task uses here.

If specified, the namespace that contains the unit tests to be run. Do not use this property if you specify allTests as true. Also, if you execute compileAndTest() in a production organization, this property is ignored, and all unit tests defined for the organization are run.

In contrast the runAllTests="true" on the sf:deploy target is explicitly documented as to not run managed classes tests.

Optional. Defaults to false. If set to true, all tests are run after deploy. For deployment to a production organization, all tests, except for those that originate from installed managed packages, are automatically run regardless of this argument. If any of these tests fail when the rollbackOnError parameter is set to true, the deployment is rolled back and no changes will be made to your organization.

NOTE: The above was not always the case, i cannot recall when, but there was a time it behaved like the compileAndTest request above.

So my conclusion is to try the sf:deploy task with the runAllTests="true" attribute, you don't need to actually deploy anything, though you will need a package.xml as per the documentation.

3
  • We tried both scenario's via ant. One with the sf:compileAndTest and one with the sf:Deploy and checkOnly to true but in both scenario's it seems that it runs all the test classes including the ones of a managed package. Now we do use it to deploy stuff to a full sandbox and not to production. I read the docs and indeed it states that it runs only code that is not from a managed package but this doesn't seem to be the case. At least not from sandbox to sandbox. Commented Feb 10, 2014 at 14:02
  • Hmmmm I see, I would be tempted to open a Salesforce support case then. In the meantime if you want to update your answer with some snippets from your Ant scripts, maybe there is something to be spotted in those, also share what it is that makes you think its running all the tests. This anyway will be good details for Salesforce support anyway. Hope I can help you further... Commented Feb 10, 2014 at 14:08
  • I updated my question with some Ant output and with the deploy target. I will create a support case for this Commented Feb 10, 2014 at 15:37
2

The way we do this is by executing every test individually. We first retrieve all the classes in our namespace then through a simple script we identify all those classes that are tests and build dynamically a file that looks like this. All this is done from Jenkins.

<runTests>positive_test</runTests>
<runTests>negative_test</runTests>
<runTests>someothertest</runTests>
1
  • This is a possible workaround indeed. Will create a case first to see what Salesforce is replying Commented Feb 10, 2014 at 15:39
0

I'm happy to report that as of Summer '15 (v34.0) there is now a testLevel parameter on the sf:deploy command so if you're using the Summer '15 version of ant-salesforce.jar you can now specify:

<target name="deployCodeCheckOnly">
  <sf:deploy 
    testLevel="RunLocalTests"
    username="[email protected]" 
    password="password" 
    serverurl="https://test.salesforce.com" 
    maxPoll="500" 
    deployRoot="codepkg"
    checkOnly="true"/>
</target>

This will specifically exclude any Managed Package tests from running. There are a number of other changes to the test options for deployments in Summer '15 that you can read about here. The available testLevel options are:

  • NoTestRun—No tests are run. This test level applies only to deployments to development environments, such as sandbox, Developer Edition, or trial organizations. This test level is the default for development environments.
  • RunSpecifiedTests—Only the tests that you specify in the runTests option are run. Code coverage requirements differ from the default coverage requirements when using this test level. Each class and trigger in the deployment package must be covered by the executed tests for a minimum of 75% code coverage. This coverage is computed for each class and trigger individually and is different than the overall coverage percentage.
  • RunLocalTests—All tests in your organization are run, except the ones that originate from installed managed packages. This test level is the default for production deployments.
  • RunAllTestsInOrg—All tests are run. The tests include all tests in your organization, including tests of managed packages

See the [Force.com Migration Tool documentation][1]

1
  • Ah Great news indeed! Commented Jun 3, 2015 at 9:41

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .