A/B Testing Statistics for Founders: Sample Size, Power and Peeking
An A/B test is simple in principle: show version A to some visitors and version B to others, and keep the one that performs better. In practice, most early-stage tests are too small to tell the versions apart, and many "winners" are noise. A little statistics goes a long way.
The four numbers that define a test
- Baseline rate: your current conversion rate, for example 5% of visitors sign up.
- Minimum detectable effect (MDE): the smallest improvement worth detecting, for example from 5% to 6%.
- Significance level (alpha): how often you accept a false positive, conventionally 5%.
- Power: the chance of detecting the effect if it is real, conventionally 80%.
A rule of thumb for sample size
For a conversion rate test with 5% significance and 80% power, a widely used approximation (sometimes called Lehr's rule) gives the visitors needed per variant:
n ≈ 16 × p × (1 − p) ÷ d²
where p is the baseline rate and d the absolute difference you want to detect. With a 5% baseline and a one-point lift (d = 0.01): 16 × 0.05 × 0.95 ÷ 0.0001 ≈ 7,600 visitors per variant, about 15,000 in total. Halve the effect you want to detect and the sample size roughly quadruples.
| Baseline | Target | Approx. visitors per variant |
|---|---|---|
| 5% | 7.5% | about 1,200 |
| 5% | 6% | about 7,600 |
| 20% | 22% | about 6,400 |
The practical message for early-stage teams: with little traffic, test bold changes (a different offer, price or audience), not button colours.
Feel the noise: coins and dice
Flip a fair coin 100 times and you will often see anything from about 40 to 60 heads. Even a 60 to 40 split, which feels like a clear bias, turns up in one direction or the other roughly once in every 18 runs of 100 fair flips. Now imagine version B "wins" 60 to 50 sign-ups. Your gut says B is better; the statistics say you cannot yet tell. Any regular at a craps table has watched dice run hot and cold for an hour without the dice changing at all.
The peeking problem
The most common error is checking results every day and stopping as soon as the difference looks significant. Each look is another chance for noise to cross the line, so a test planned at 5% false positives can end up far higher if you peek and stop at will. Either fix the sample size in advance and look once, or use methods built for continuous monitoring (sequential tests or Bayesian approaches with sensible priors), which many testing tools now offer.
Gamblers know a version of this trap as the martingale: double your bet after each loss and stop as soon as you win. It feels like a system that cannot lose, because you always stop on a win. In reality, it just hides a small chance of a catastrophic loss. "Test until significant, then stop" works the same way: it guarantees a winner, and quietly guarantees that some of your winners are fake. We explore the psychology behind this in A/B testing versus gambler's intuition.
Other traps
- Too many variants or metrics. Test 20 things and, at 5% significance, you should expect about one false winner by chance alone.
- Novelty effects. A change can win for a week simply because it is new.
- Uneven traffic mixes. If one variant gets more mobile or more paid traffic, you are comparing audiences, not designs.
- Wrong metric. Optimising clicks can hurt sign-ups; optimising sign-ups can hurt revenue.
Who tests the most?
Large web companies run thousands of experiments a year, and so do parts of the gambling industry, from online slot studios tuning bonus features to social casino apps testing offers. The casino floor has always been an experiment in its own right. Our case study on casino product design and testing shows what founders can borrow from that culture, and what to avoid.
