Recently I put my code through the free checkmarx code scanner provided by salesforce. It gave me a stored XSS issue. But the piece of code which was identified has been in my codebase since a long time and has gone through multiple rounds of scanning but never got this error earlier.

What I want to know is this a know behavior of the scanner or is this a one of instance where the scanner was not able to find an issue earlier but has found it eventually.

I am concerned about the reliability of the scanner and want to know if anyone else has been in this situation ever.

1 Answer 1


You should view the security scanner as non-deterministic because

1) The Checkmarx queries are occasionally updated to try to reduce false positives and detect more true positives

2) The engine versions are updated occasionally which will alter the result sets of queries

3) Differences in timeouts/loads. If, for instance, there is a 60 minute timeout per query, then it may well be the case that when a machine is under heavy load a scan that took a bit less than 60 minutes before now takes a bit more than 60 minutes, and so the query may timeout in one case but not another, return results in one run but not in another.

4) Differences in the code -- just because the relevant code path of the vulnerability is not changed doesn't mean that the rest of the code in your app is the same. If anything changes, that may change the time it takes to get a control flow and dataflow graph for the package as a whole which may affect overall timeouts as well as overall results.

You should not view the scanner as detecting 100% of all issues with 0% false positives. Static analysis is, like all other forms flaw detection, both an art and a science. It should be used in conjunction with blackbox testing, manual testing, and manual code reviews and even then you are not guaranteed to catch 100% of the flaws in your code.

  • #2 is quite likely. I know they upgraded the engine version sometime this year. Sep 24, 2014 at 2:40

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .