I'm looking into routing my code through Git and am wanting to do builds using Ant. However, I do not want to run all tests on each build and only want to run tests for the files that have changed from the previous commit. Does anyone have a good way of dynamically creating the build.xml file to only include tests to be ran that are related to these changed files?
I wanted to expand on my comment a bit to sketch what an approach to do this might have to look like, but also to warn as to why I think it's not a great idea and likely to be more expensive to do well than it's worth.
There's a few layers to looking at a Git commit and determining "what has changed" to make our deployment decisions.
- Which files have changed? This one is pretty easily dealt with via
git diff: we can get a list of the file names changed in a commit or across a commit range, so we know how to perform a delta deployment.
- Which files have changed that are test classes? This one is a little more complex. If the org is rigorous about naming conventions, you might be able to match a regex against the file names, like
_TEST.cls$. Otherwise, you'd need to peek into the body of the file and see if you find the token
@isTest. To be really rigorous about it, you'd want to do some parsing to ensure that the token isn't in a comment, etc.
- What are the test classes for the non-test classes that have changed, whether or not the tests themselves have been changed? That's quite a bit harder, and there isn't a genuine, rigorous solution. Again, if the org is religious about naming conventions, you might be able to iterate over the changed files and say that your test for
X.clsis located in
X_TEST.cls. But, hmm, what are you going to do when the name of the class is long enough that it has to be abbreviated to fit the
_TESTtoken on the end? Or what if you have critical legacy code that cannot be touched, but doesn't fit the pattern? Suddenly you may have a lot of mappings to maintain, and you're looking for a generalized solution to say "Test Class X tests Class Y", which isn't encoded anywhere in the metadata... and that leads you to the bigger question:
- Which test classes' results are potentially impacted by changes to the files in this commit?
And the answer to that last question is, basically, "all of them".
Now that might be more or less true in different code bases, depending on how rigorously modular and decoupled your code units are, to what extent data factories are used, whether you're using unlocked or managed packages to express dependencies between larger-scale code units, and so on.
But thinking about the general case, if we try to solve this problem, where I think we'd end up is doing some complex and tricky work with the Dependency API (or a parser) to try to identify the network of references between the classes we're working with and the various test classes, so we can identify which test classes invoke code in the classes we've changed in our latest commit.
So we'd do all that engineering to finally get to say "Class X tests Class Y"... and it still wouldn't be enough. Because we also use CI and automated testing to identify situations where, for example,
AccountTriggerHandler's changes break the test data setup for
OpportunityTriggerHandler's unit tests, but neither explicitly invokes the other or is invoked in each other's tests - so there's no references to walk between the two.
And to do CI properly, I think we have to solve that general-case problem, because the purpose of automated testing is to identify problems that break your assumptions about how your codebase works. That includes both simple assertion violations, but also the codebase-level structures that you assume but cannot necessarily enforce in an automated fashion (like using test data factories correctly, and preserving the isolation between code units, and ensuring that new invariants are respected across all of your existing classes - if someone screws up those facets and causes a test failure or a cascade of test failures, you want to know about it at check-in time, not later!)
So, for a TL;DR: deploy deltas if you need to, to speed up deployments or shrink the size of your Metadata API payloads, but - at least at the CI/CD automation level - always run all the tests.