Salesforce Stack Exchange is a question and answer site for Salesforce administrators, implementation experts, developers and anybody in-between. It only takes a minute to sign up.
Sign up to join this community
I have a lot of strong opinions about this issue, but had a hard time finding official documents today to back up my position. A lot of times we say it is bad practice to use seeAllData=true, but we don't really explain why. Maybe we can generate a more definitive and thorough answer here. So I want to ask the broader community:
Why is it important to create our own data when we write unit tests?
I think the answer is quite simply one of consistency. It's also one of minimizing variations between environments we test in. This makes our test more consistent and repeatable. These are all things that are helpful should we need to diagnose or troubleshoot a problem during deployment.
When we generate our own data, that eliminates a variable between what happens in any sandbox vs a production environment. We KNOW we've used the same test data. We know what values we can expect to see at any point in our unit test.
We can create not only the database records, but also our User records. This allows us to control the context for running our unit tests with RunAs().
Similarly, we can create users who become the owners of the records in our test data. Doing this creates even more consistency and reduces additional variances between environments (development org to development org or vs production). Again, this allows us to eliminate another variable should we need to troubleshoot a deployment from one environment to another.
We don't always know that the kind of data we need to run our tests will exist in an org if we don't create it. This is especially true when testing in a sandbox, a fresh org, or one where records of the type our code uses have never been created. How an our test class possibly grab test data from the org if there are no records records that meet the criteria to test all of the conditions our code uses? Clearly we can't do that. It's incumbent on us to ensure the data we need is available which means we need to create valid data to test with.
Our code must also be bulk safe. In my view, testing using only one record per method doesn't constitute an adequate unit test. All it does is allow code to "pass the minimum requirements" needed to deploy it. The latter doesn't ensure our code will function reliably. That's what system.asserts, positive and negative test cases, bulk test cases along with other test methods are for. I strongly believe that our unit tests need to include those methods as well.
I want to add that what I'm speaking of doesn't prohibit a developer from creating randomness in their data. It still allows for that. The randomness is created in a controlled and consistent manner so the behavior of the code can be predicted and results still asserted. Validity of the data will be known provided the environment doesn't "do something" to it that's unexpected.
It's this consistency during creation that allows the diagnostics and troubleshooting of what's "different" between two environments when there's an issue with deployment or functionality of the code is not as expected.
This goes beyond the question that's been asked, but I'm a big believer in Test Driven Development. These help me test functionality as I build my code. When I'm finished, my test class is also completed and it's not an afterthought or chore. I like to create tests using lists of records where I can change a variable in my class that initially sets the list size to a length of 1. When I'm ready to bulk test, I'm then able to set it to a larger number, including raising it to 200 (or even more if appropriate), to ensure my code is bulk safe.
During deployment, I can lower that value to create 1 record/list to test with so time isn't wasted. But the method is there in my unit test and I KNOW that my data has been bulk tested and can easily be retested at any time in production. If I want to, the value of that list size can easily be controlled through a custom setting via a test environment utility/helper class. This allows me to retest in bulk at any point in time I want to. I'll add that I generally create a test utility class that creates all of my objects. This is great way to reuse my code.
So, my answer to the question is we do it so the developer of the code can achieve the maximum amount of control over the consistency of test conditions as he or she possibly can; regardless of the environment it runs in, each and every time the test method is executed. This is a discipline I was taught in engineering school for designing and creating tests. It was also expected when I was an engineer in the automotive industry (see this article by Charles Deming famous for "The Deming Way"). It only seems logical to me that it would apply to software as well since it applies to any other kind of testing, including tests of humans using software and electronics running it as well.
static final Integer RECORD_COUNT. I actually set it to 101 for all sandbox tests and 5 for production.
Custom Setting to control test size without using seeAllData=true though. Perhaps an advantage of Custom Metadata Types?
seeAllData=true. Feel free to edit this with bullets however you'd like. This is a collaborative forum and all improvements to posts are encouraged.
May 26, 2016 at 17:00
In two words: "bitter experience"
I will admit that if you don't have any asserts in your code (another topic) you run a lesser risk but, the reasons all have to do with repeatability/predictability and the advantages therein.
Every deployment of the same test class should run the same environment every time. Today, tomorrow, and 5 years from now. Here are some examples that have happened to me over the many years:
So, for argument's sake, let's assume you are a good developer and strongly believe in system asserts to "prove" the correctness of your implementation with positive/negative/bulk tests. By the time you have done all this the last thing you want is a deployment error -- On the next deployment or on some deployment far in the future. You are under time pressure when you get to deployments, you may have limited deployment windows, and finding errors in deployments is more difficult than in sandbox.
And lastly, professional pride. You are leaving a legacy of software for some other developer/admin and why not treat them to repeatable/reliable code just as you would want to be treated.
Another point to add to cropredy's excellent list is that tests have significant value as a specification of what the code being tested does. But that only works if you can see the inputs (often the database rows) and expected outputs (the asserted values) in the test. Inserting exactly the data that is relevant to what is being tested in the test makes those inputs visible.
Somewhat related is that it is actually easier to write assertions where you have a code reference to the input. Though if you take advantage of @TestSetup to speed up the tests by setting the data up once, you lose the simple references to the data (as you have to query to get them in each test method).
@TestSetup just as well as within it, though. Can you clarify what you're getting at there?
static final constants. IMO it's cleaner anyway.
@TestSetup. I prefer using a common utility class to create records within each method at run time. I write them in a way that allows me to create related records by passing parameters within my utility classes so I can return lists with varying sizes, param names, etc. It still allows me to make consistent & predictable assertions.
Jun 10, 2016 at 13:12
The Hammer Execution team explicitly asks us to use data silos in our tests (emphasis mine):
Our Commitment To You
This complex juggling act happens three times a year. We make a commitment to you that your customizations will continue to run as you expect, no matter how much we do to change the system. In all my years of being in enterprise software, I have never heard of such a commitment. I’ve never heard of such a smooth upgrade path: you go to bed on Friday, you wake up Saturday, and your system has been upgraded. I still have flashbacks to lenghty, painful upgrade projects for other companies I have worked for – those were never easy times. To save you from these flashbacks, we do the complex ballet described here (which still makes it sound easier than it is) three times a year.
You can help us fulfill this commitment by using data silo. Tests that utilize the new data silo paradigm (seeAllData=false) create their own data, and don’t actually need access to the production data stores. These tests are much faster for us to run, and much more reliable. We don’t need to worry about reading your debug statements, because they’re outputting fake data. We don’t need to worry about having a copy of your production data, because data silo tests do not rely on that data to succeed. Org-independent tests make The Hammer smile.
I consider that compelling in its own right! I'm also happy to learn that following best practice in this case makes us good citizens of the platform.
