Can AI Score Your Startup Idea? Scorecards, Ratings and Their Limits
In 2023, as GPT-4 arrived, a wave of small web apps promised to "validate" a startup idea in seconds. You typed a description; the app returned an analysis of the problem, the market and the business model, often with a single score out of 100 and a list of suggestions. This domain briefly hosted one such tool before it went offline. We have no connection with it, but the question it raised is a good one: can a model, or any scorecard, tell you whether an idea is worth pursuing?
What AI idea scorers are good at
- Structure. A language model will reliably walk an idea through the standard frameworks: problem, customer, value proposition, competition, revenue model, risks.
- Devil's advocacy. Asked to argue against your idea, a model will produce objections you may not have considered, at any hour and without social awkwardness.
- Desk research starting points. It can suggest categories of competitors, adjacent markets and questions to ask in interviews.
- Speed. Ten rough ideas can be screened in an afternoon, which helps you decide where to spend real testing effort.
What they cannot do
A model has not spoken to your customers. It knows what has been written about markets, not what your segment will pay next month. It can be confidently wrong about competitors, market sizes and regulations, and its numbers should be checked against primary sources. Most important, a score produced this way is not calibrated: there is no evidence that an idea rated 80 is more likely to succeed than one rated 60, because the scorer has never been tested against outcomes.
Lessons from the racetrack: ratings must predict
Horse racing has a long tradition of compressing messy information into a single number. In the 1970s the writer Andrew Beyer popularised speed figures, ratings that adjust a horse's finishing time for the track and conditions, and from the early 1990s "Beyer Speed Figures" were printed in the Daily Racing Form. Handicappers loved them because they were tested every day against results. A rating that did not help bettors pick winners at better-than-market odds would have been abandoned.
That is the standard any idea score should meet. Forecasters measure calibration with tools such as the Brier score: if you say "70%" on many events, about 70% of them should happen. A startup scorecard that has never been checked against outcomes is closer to a tipster's hunch than to a speed figure. The data businesses that sell ratings and odds to bookmakers live or die on this distinction, as we describe in sports-betting data startups.
Build your own weighted scorecard
A scorecard is still useful as a way to compare your own ideas consistently. The trick is to score evidence, not enthusiasm. Here is an example for a hypothetical B2B tool after two weeks of interviews:
Example idea scorecard (evidence-based)
Weight the rows by what matters for your model (willingness to pay and reachability usually deserve more weight than founder enthusiasm), and write down the evidence behind every number. Then, crucially, treat low scores as instructions: a 4 on competition tells you which assumption to test next.
Using an AI scorer sensibly
- Use it before interviews, to generate hypotheses and questions.
- Ask it to argue the opposite case, and to list what evidence would change its view.
- Verify every factual claim it makes about markets, competitors and rules.
- Never let a score replace a test. A single number feels decisive in the way a "hot tip" feels decisive at the track; both deserve the same scepticism. Our article on A/B testing versus gambler's intuition explains why our brains over-trust such signals.
The best use of any score is as the start of a validation plan, not the end of it. Go from score to interviews to a smoke test, and let real behaviour have the final word.
