The base rate fallacy, also known as base rate neglect, is a type of invalid reasoning where the actual likelihood of an event is not sufficiently taken into account when interpreting data.
One of the most counterintuitive situations involving the base rate fallacy is in the interpretation of positive results where statistically it is unlikely for the thing being tested to actually be positive.
Imagine a disease where 1 in 1000 people has the disease. That is, 0.1% of the population.
Now consider if researchers were studying a new test for the disease. They find the test has a 1% false positive rate and a 0% false negative rate.
Here's what that 1% false positive rate means in practice: If one million people are tested, the test would identify 10,000 people as having the disease. However, since we know only 0.1% of the population actually has the disease, we know that the test should have only identified 1,000 people as having the disease, not 10,000 people. In other words, for every true positive result (correctly identifying someone as having the disease), the test had 9 false positives (people who didn't have the disease but the test said they did).
In this contrived example, there isn't enough information to know whether the test could ever be useful in practice. For example, you'd have to consider how harmful it is to incorrectly identify someone as having the disease. Instead, the lesson is that the base rate of prevalence must be considered.
Why is the base rate fallacy relevant to security products such as Web Application Firewalls? Because in situations where malicious requests are often a small percentage of overall requests, even what seems like a low false positive rate can result in the majority of blocked requests being legitimate requests.
The best way to choose a WAF with a low false positive rate is to use an industry-standard WAF where the rules have been heavily tested in a wide variety of situations. HeatShield uses ModSecurity with the OWASP Core Rule Set, which is the world's most tested and trusted WAF.
You can learn more about the base rate fallacy on Wikipedia's article.