Survey Design for Idea Validation: Questions, Bias and Sample Size
Surveys are seductive because they are cheap and produce numbers. Unfortunately, a badly designed survey produces confident numbers about nothing. Used carefully, after interviews have shown you what to ask, a survey can size a problem and compare segments in a way a dozen conversations cannot.
When a survey is the right tool
Use surveys to quantify what interviews have already uncovered: how common a problem is, which segment feels it most, how people currently solve it, how much they spend. Do not use a survey to discover problems from scratch or to ask whether people like your idea. Stated intentions are notoriously poor predictors of what people will do.
Writing questions that do not lead
- Avoid leading wording. "How frustrating is invoicing?" assumes frustration. "How would you describe your invoicing process?" does not.
- One idea per question. "Is our tool fast and easy to use?" is two questions.
- Ask about behaviour. "In the last month, how many times did you...?" beats "How often do you usually...?"
- Balance the scales. Give as many negative options as positive ones, and include "none of these" and "don't know".
- Randomise option order where your tool allows it, to reduce the pull of the first item.
Watch for acquiescence bias (people's tendency to agree with statements) and social desirability bias (people describing themselves as more organised, frugal or healthy than they are). Short surveys get better answers: if it takes more than five or six minutes, completion and quality both fall.
How many responses do you need?
The margin of error on a proportion depends mostly on the number of responses, not on the size of the population. A useful rule of thumb at 95% confidence, when the answer is near 50%, is roughly one divided by the square root of the sample size:
| Responses | Approximate margin of error |
|---|---|
| 100 | about ±10 points |
| 400 | about ±5 points |
| 1,000 | about ±3 points |
For early validation, 100 to 200 well-targeted responses are often enough to see big differences between segments. But a large sample does not fix a biased one, and history has a famous lesson on that point.
The 1936 poll that lost to the bookmakers
In 1936 the Literary Digest mailed millions of ballots and received around 2.4 million back. Its poll predicted that Alf Landon would defeat Franklin D. Roosevelt. Roosevelt won in one of the biggest landslides in American history. The magazine's sample, drawn heavily from lists such as telephone and car owners, and shaped by who chose to reply, skewed towards wealthier voters in the middle of the Depression. George Gallup, with a far smaller but better-designed sample, called the result correctly.
Less well remembered is that for decades before modern polling, Americans bet heavily on presidential elections, with odds quoted on Wall Street and reported in newspapers. Economists Paul Rhode and Koleman Strumpf, who studied those markets, found that in the era before scientific polls the betting odds were a remarkably good guide to the outcome. The reason is incentive: a bettor loses money for being wrong, while a survey respondent loses nothing. That idea, putting a price on predictions, sits behind today's prediction markets, which we look at in prediction markets as idea validators.
A simple validation survey template
- Screening: two or three questions to confirm the respondent is in your segment.
- Current behaviour: how they handle the problem today, how often, with which tools.
- Cost: time and money spent, in ranges.
- Ranking: which of five problems matters most to them (forced ranking reveals priorities better than ratings).
- Optional open text: "What is the most annoying part of...?"
- Opt-in: an email field for follow-up interviews or early access.
That last question is the most valuable one. The share of respondents who volunteer an email address is a small but genuine commitment, and it gives you a list of people to recruit for follow-up interviews or a smoke test.
