Who Measures Risk Better

At Root Evidence, my team and I spend time thinking about a huge contradiction. Vulnerability management vendors are rewarded for finding more problems in their customers’ environments. At the same time, cyber insurance carriers are rewarded for reporting fewer problems to policyholders because it is a competitive advantage.

‍

Both claim to be measuring risk. Only one can be right in any coherent sense.

‍

Vendors sell comprehensiveness and want to win in bakeoffs that show that they find “more”, so they compete to surface every possible flaw. Insurers sell risk transference, so they compete to minimize friction in the underwriting process, which means not bugging customers about every minor flaw. But beneath that lies a deeper epistemic failure. When two institutions observe the same system with opposite incentives, we have to agree that one of these groups is not measuring risk. I’ll give you one guess who’s more accurate in terms of loss reduction, which is one very reasonable way to measure risk.

‍

When we really dug into the data, we learned very quickly that the count of vulnerabilities is not a proxy for danger. It is a proxy for effort, and it’s a hidden cost of using those vulnerability scanners. A scanner that works harder and produces more results will always appear smarter, even if it really is just passing all of the cost of determining which vulnerability matters onto the customer. An insurer that repeatedly pressures customers about every new “critical” vulnerability risks losing credibility when those issues fail to translate into real financial loss. Therefore, insurance carriers really do need to keep the amount of noise to a minimum. This creates pressure to right-size the communication by focusing only on losses.

‍

We decided early that this could not be solved by finding more issues or by suppressing them. It had to be solved by changing what we meant by evidence of risk, and the easiest answer is to look at loss, rather than breach, or even worse, an arbitrary score. Instead of asking how many weaknesses exist, we think that the smarter path is to ask which ones change outcomes. Which means telling customers that a very small amount of findings can mean significant danger, and less work to accomplish the removal of that risk.

‍

I am not optimistic that markets will fix this on their own. Attackers do not care how full of busy work our dashboards are, except to say that they obscure or delay the real risk from being addressed. If we align our measurements with their reality, we can finally make progress.

‍