Validation Mistakes: False Positives, Vanity Metrics and Survivorship Bias

Validation exists to protect you from your own optimism. Yet the same optimism finds ways to creep back in, through who you ask, what you count and how you read results. Here are the traps we see most often, and how to step around them.

1. Asking people who love you

Friends, family and former colleagues want you to succeed. Their encouragement is valuable for morale and worthless as evidence. Recruit strangers in your target segment, and treat warm-network feedback as practice runs for your interview script.

2. Counting vanity metrics

Page views, social likes, app downloads and total sign-ups all go up and to the right with a little effort. None of them tells you whether people get value. Prefer metrics tied to behaviour that costs the customer something: activation (did they complete the core action?), retention (did they come back?), and payment. A smoke test with 2,000 visitors and 15 sign-ups tells you more than a viral post with 50,000 likes.

3. Survivorship bias

During the Second World War, the statistician Abraham Wald, working with the Statistical Research Group at Columbia University, studied where returning bombers had been hit. The intuitive response was to add armour where the bullet holes clustered. Wald's insight was that the holes showed where a plane could be hit and still make it home; the planes hit elsewhere had not come back to be counted. The diagram above, a modern illustration of that idea, has become shorthand for the bias.

Startup advice is full of survivors. We read about the founder who ignored every sceptic and won, not the many who ignored sceptics and lost. Gambling culture has the same problem: television interviews lottery winners, never the millions of ticket holders who funded the jackpot, and casino lore remembers the big win, not the long slow drain of the house edge. When you study successful companies for lessons, ask what the failures did that was the same. We compare that lottery-ticket thinking with genuine pre-validated launches in crowdfunding versus the lottery mindset.

4. Seeing streaks that are not there

Three good days in a row on a landing page feels like a trend. It usually is not. Small samples fluctuate wildly, and our brains are built to see patterns in noise. Roulette players watching a run of reds fall into the gambler's fallacy, believing black is "due"; founders watching a run of good days fall into its cousin, believing the trend will continue. Both are reading noise as signal. The cure is to decide your sample size in advance and to use proper tests, as explained in A/B testing statistics and A/B testing versus gambler's intuition.

5. Moving the goalposts

If the pass mark is set after the results arrive, every result passes. Write down what success looks like before each experiment, including the number that would make you stop.

6. Judging the decision by the outcome

Poker players call this "resulting": concluding that a decision was good because it worked out, or bad because it did not. A well-run test that kills an idea is a success. A sloppy launch that gets lucky is still a sloppy launch, and the luck will not repeat. Review your validation process separately from your results, a habit we explore in startups as bets.

7. Testing the wrong thing

It is easy to test what is easy instead of what is risky. A beautiful landing page tests copywriting; it may not test whether the target customer can be reached at a sustainable cost. Go back to your list of assumptions and make sure each experiment attacks the riskiest one.