6

I'm looking into routing my code through Git and am wanting to do builds using Ant. However, I do not want to run all tests on each build and only want to run tests for the files that have changed from the previous commit. Does anyone have a good way of dynamically creating the build.xml file to only include tests to be ran that are related to these changed files?

  • You might find this link useful: CI Scripts for Salesforce projects – Robs Jan 11 at 22:29
  • 2
    There is not a Test Run Level for "only run tests that are currently being deployed." You'd have to do a git diff HEAD~1 or equivalent and either use a regex (if you have a class naming convention) to find the tests, or inspect for the @isTest token in the file body. Then you can pass that information with the Run Selected Tests test run level. (I don't really use Ant so I don't have implementation details!) – David Reed Jan 11 at 22:34
  • Thanks @DavidReed, I was thinking something along these lines. The only problem with that is if the test file has a significantly different name then it will be difficult to match up. This is only the case for a few of my test files but still enough of an annoyance. The only thing I can think of is if you change the test files you need to deploy by adding a line or something meaningless so they're included in the commit but that's easy to forget to do and a bit hacky. – Josh Jan 14 at 15:39
2

I wanted to expand on my comment a bit to sketch what an approach to do this might have to look like, but also to warn as to why I think it's not a great idea and likely to be more expensive to do well than it's worth.

There's a few layers to looking at a Git commit and determining "what has changed" to make our deployment decisions.

  • Which files have changed? This one is pretty easily dealt with via git diff: we can get a list of the file names changed in a commit or across a commit range, so we know how to perform a delta deployment.
  • Which files have changed that are test classes? This one is a little more complex. If the org is rigorous about naming conventions, you might be able to match a regex against the file names, like _TEST.cls$. Otherwise, you'd need to peek into the body of the file and see if you find the token @isTest. To be really rigorous about it, you'd want to do some parsing to ensure that the token isn't in a comment, etc.
  • What are the test classes for the non-test classes that have changed, whether or not the tests themselves have been changed? That's quite a bit harder, and there isn't a genuine, rigorous solution. Again, if the org is religious about naming conventions, you might be able to iterate over the changed files and say that your test for X.cls is located in X_TEST.cls. But, hmm, what are you going to do when the name of the class is long enough that it has to be abbreviated to fit the _TEST token on the end? Or what if you have critical legacy code that cannot be touched, but doesn't fit the pattern? Suddenly you may have a lot of mappings to maintain, and you're looking for a generalized solution to say "Test Class X tests Class Y", which isn't encoded anywhere in the metadata... and that leads you to the bigger question:
  • Which test classes' results are potentially impacted by changes to the files in this commit?

And the answer to that last question is, basically, "all of them".

Now that might be more or less true in different code bases, depending on how rigorously modular and decoupled your code units are, to what extent data factories are used, whether you're using unlocked or managed packages to express dependencies between larger-scale code units, and so on.

But thinking about the general case, if we try to solve this problem, where I think we'd end up is doing some complex and tricky work with the Dependency API (or a parser) to try to identify the network of references between the classes we're working with and the various test classes, so we can identify which test classes invoke code in the classes we've changed in our latest commit.

So we'd do all that engineering to finally get to say "Class X tests Class Y"... and it still wouldn't be enough. Because we also use CI and automated testing to identify situations where, for example, AccountTriggerHandler's changes break the test data setup for OpportunityTriggerHandler's unit tests, but neither explicitly invokes the other or is invoked in each other's tests - so there's no references to walk between the two.

And to do CI properly, I think we have to solve that general-case problem, because the purpose of automated testing is to identify problems that break your assumptions about how your codebase works. That includes both simple assertion violations, but also the codebase-level structures that you assume but cannot necessarily enforce in an automated fashion (like using test data factories correctly, and preserving the isolation between code units, and ensuring that new invariants are respected across all of your existing classes - if someone screws up those facets and causes a test failure or a cascade of test failures, you want to know about it at check-in time, not later!)

So, for a TL;DR: deploy deltas if you need to, to speed up deployments or shrink the size of your Metadata API payloads, but - at least at the CI/CD automation level - always run all the tests.

  • Thanks for the writeup David, it's incredibly helpful. My big problem and why I didn't want to run all tests is we have some tests in our Org that are no longer needed and will fail. You got me thinking about the benefits of CI and really why I want to do this, so I'm going to change my approach and for now create a whitelist of tests to be run, and use what you've written here so I can dynamically add to this whitelist whenever a new test is committed. – Josh Jan 14 at 21:06
  • @Josh, my pleasure -- as for your legacy tests -- any reason you're not simply deleting them if they're understood to be invalid and do not provide insight through their failure? – David Reed Jan 14 at 21:19
  • I will eventually but that's a higher level of effort as I'm new to the code base I'm working on so for now whitelisting will be easier until I can figure out for sure whether or not a certain test case is needed. – Josh Jan 14 at 21:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.