A/B Testing vs. Gambler's Intuition: Significance and the Gambler's Fallacy
On August 18, 1913, at the Casino de Monte-Carlo, the ball on one roulette wheel landed on black 26 times in a row. As the streak grew, players reportedly piled their money on red, convinced that it was "due". They lost heavily. The wheel had no memory, and each spin still offered red at just under even odds.
The gambler's fallacy
The belief that a random process "corrects itself" in the short run is called the gambler's fallacy, or sometimes the Monte Carlo fallacy after that evening. Twenty-six blacks in a row on a single-zero wheel has a probability of roughly 1 in 137 million for any given run of 26 spins; extraordinary, yet with enough wheels spinning for enough years, streaks like it are bound to happen somewhere. What does not happen is the wheel compensating afterwards.
Founders commit the same error with metrics. "Sign-ups have been low three days running; tomorrow should bounce back." "The last two campaigns flopped, so this one is bound to work." Without a reason that changes the underlying probability, a bad streak predicts nothing about the next result.
The hot hand: the opposite mistake
The mirror image of the gambler's fallacy is the belief in streaks that continue: the "hot hand". In a famous 1985 study, the psychologists Thomas Gilovich, Robert Vallone and Amos Tversky analysed basketball shooting and concluded that players' streaks were no more common than chance would predict. For three decades this was textbook evidence of a cognitive illusion. Then in 2018 the economists Joshua Miller and Adam Sanjurjo showed that the original analysis contained a subtle statistical bias, and that correcting it revived evidence for a modest hot hand. The debate is a good reminder that even careful researchers can be fooled by the maths of streaks.
In a product context, the hot-hand belief sounds like: "Our new homepage has beaten the old one for four days; it's a winner." Maybe. Or maybe you are watching noise. That is what significance testing is for.
What significance actually means
A result is called statistically significant at the 5% level when, if there were really no difference between A and B, you would see a difference this large less than 5% of the time. It does not mean there is a 95% chance B is better, and it says nothing about whether the difference is large enough to matter. Those are separate questions, covered in our guide to A/B testing statistics.
Many tests, many false winners
Run one test of a change that does nothing, and you have a 5% chance of a false positive. Run 20 such tests and the chance that at least one shows a "significant" winner is about 64%. Teams that test lots of small tweaks and ship every winner will ship plenty of changes that do nothing, and sometimes harm. Slot machines, by contrast, are certified for long-run return precisely because the short run is so noisy; casinos rely on the law of large numbers, and so should you.
| Number of tests with no real effect | Chance of at least one false winner at 5% |
|---|---|
| 1 | 5% |
| 5 | about 23% |
| 10 | about 40% |
| 20 | about 64% |
Regression to the mean
Extreme results tend to be followed by less extreme ones. The ad that performed best last week will probably do a little worse this week, not because it "wore out" but because part of its success was luck. This also explains why changes made in response to a terrible week often appear to work: things would have improved anyway.
Where intuition still wins
None of this means gut feeling is useless. Experienced poker players develop a feel for situations that is grounded in thousands of hands, and experienced founders develop a feel for customers that is grounded in hundreds of conversations. Intuition is excellent at generating hypotheses: which headline to test, which segment to try, which feature to cut. It is poor at evaluating noisy results. Let instinct choose the bets and let the data settle them. For the most common ways that instinct misreads results, see validation mistakes and false positives.
