3

I am trying to understand the benefit of Apex Hammer Test Results feature for a Salesforce Developer or Salesforce Administrator.

I have read the official documentation about the Apex Hammer but I struggle to find any practical benefit or value of it.

The only data which is shown is aggregate data which doesn't bring any value enter image description here enter image description here

It is not clear how Salesforce determines which organization instances to use and in which release, is this just an arbitrary and completely random process?

I mean why the tests on the first client organization has been run on Jul 28, 2017 and on the second organization on Nov 10, 2015? How do they match releases and organizations?

What is the value of seeing that 58.53% is green on the chart? Does that mean that this is percent of tests run successfully on the previous or next release? Does that mean that this is percent of test run successfully in old release but failed in new release? Does that mean that Salesforce has fixed those failures in the next release?

I can't find any answers to these questions or generally I may lack understanding the benefit or the value of this feature

6

Have a read of Here Comes The Hammer, you may find it illuminating.

Q:
It is not clear how Salesforce determines which organization instances tests to run and in which release, is this just an arbitrary and completely random process?

A:

The Hammer means taking every single Apex test that you or anyone else has created and running it twice.

They run every test.


Q:
What is the value of seeing that 58.53% is green on the chart? Does that mean that this is percent of tests run successfully on the previous or next release? Does that mean that this is percent of test run successfully in old release but failed in new release? Does that mean that Salesforce has fixed those failures in the next release?

A:

You can help us fulfill this commitment by using data silo. Tests that utilize the new data silo paradigm (seeAllData=false) create their own data, and don’t actually need access to the production data stores. These tests are much faster for us to run, and much more reliable. We don’t need to worry about reading your debug statements, because they’re outputting fake data. We don’t need to worry about having a copy of your production data, because data silo tests do not rely on that data to succeed. Org-independent tests make The Hammer smile.

This number tells you how many of your tests no longer rely on SeeAllData. You should strive for very nearly 100%.


So what's the benefit?

This complex juggling act happens three times a year. We make a commitment to you that your customizations will continue to run as you expect, no matter how much we do to change the system. In all my years of being in enterprise software, I have never heard of such a commitment. I’ve never heard of such a smooth upgrade path: you go to bed on Friday, you wake up Saturday, and your system has been upgraded. I still have flashbacks to lenghty (sic), painful upgrade projects for other companies I have worked for – those were never easy times. To save you from these flashbacks, we do the complex ballet described here (which still makes it sound easier than it is) three times a year.

...

For the 60 million tests we ran this past release, we uncovered 25 potential regressions in time to fix them prior to release. On the surface, that sounds like a lot of work for a tiny result. Each of those 25 issues we solved were going to impact someone in some way, potentially in some critical way, and they often span multiple customers. The result is quite large from the point of view of each customer who never had to know that they were saved from a big headache.

  • Where did you find that that number tells how many tests no longer rely on SeeAllData. - I don't follow. Is this statement present in "Here comes the hammer" article? I can't find it there. – Patlatus Jan 24 '18 at 11:55
  • So the benefit is that this tool show percent of test which do not rely on SeeAllData = true? – Patlatus Jan 24 '18 at 11:56
  • I will rephrase the part about random process, I mean different question – Patlatus Jan 24 '18 at 11:58
  • They use every instance and release. Data Silo means opposite of SeeAllData. – Adrian Larson Jan 24 '18 at 13:35
  • If they use every instance and release, why then the tests on the first client organization has been run on Jul 28, 2017 and on the second organization on Nov 10, 2015? Why the Hammer process didn't run on Dec 8, 2017 for both of these organizations? If this is perfect process why do regressions still happen and on some releases some tests start to fail? – Patlatus Jan 24 '18 at 16:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.