For a quick choice between two to four options, 50 answers is a reasonable start. Use 20 to 30 for a gut check. Use 100 or more when the options are close or a wrong choice would be costly. More answers make a lead easier to trust. They don't make the question better.
What sample size changes
The number of answers changes how much a split can wobble by chance. With few answers, a small lead can happen by accident. With more, the same lead is harder to explain that way.
Sample size does not change who you asked, how you worded the question or whether the options were fair.
The same split at three sizes
50heads describes how strongly the answers favor the leader as High, Medium or Low. We compare the gap between the top two options with how much that gap could vary by chance for that number of answers. A gap about two and a half times that size or more is High. About two times is Medium. Anything smaller is Low. These are our own thresholds, set out in what confidence means.
Here is how that rule reads for some two-option splits. The numbers are illustrative, not from real studies.
| Answers | Split | Lead | How strong for this size |
|---|---|---|---|
| 20 | 13 to 7 | 6 answers | Low |
| 20 | 16 to 4 | 12 answers | High |
| 50 | 30 to 20 | 10 answers | Low |
| 50 | 33 to 17 | 16 answers | Medium |
| 50 | 35 to 15 | 20 answers | High |
| 100 | 55 to 45 | 10 answers | Low |
| 100 | 60 to 40 | 20 answers | Medium |
| 100 | 70 to 30 | 40 answers | High |
Look at the 60 to 40 split. With 50 answers it is Low. With 100 it is Medium. The split hasn't changed. Our certainty about it has.
What to do at each size
- 20–30 answers. Good for a quick gut check, or to find a clear loser. Only a large lead means much.
- 50 answers. A reasonable start for two to four options. Typically about ten minutes.
- 100–200 answers. Better when options are close, when a wrong choice would cost you, or when you want to compare groups.
On 50heads you can ask up to 5,000 on the web. The estimate shows the cost and time together before you ask. For a short question, 50 Tier 1 answers cost $13.
What more answers will not fix
- A leading question. Strong answers to a biased question are still biased. See how to write a question that does not lead people.
- The wrong group. Fifty people who are not your customers tell you about fifty people who are not your customers.
- Population claims. A quick preference test shows what this group chose. It does not tell you what share of a whole country or market would choose the same.
- Behavior. What people choose when asked is not the same as what they buy. See A/B testing vs preference testing.
If the lead is low
You have four options:
- ask more people;
- drop the weakest option, so the answers spread less thinly;
- ask 20 Tier 2 heads, people who have passed an ID check, for a short reason;
- accept the tie. Sometimes the honest answer is that people don't mind.
Pay for the answers, not the certainty
You pay only for answers that are accepted. If fewer heads answer than you asked for, the rest goes back to your balance.
A bigger sample costs more and takes longer. Match it to the cost of being wrong. If a wrong name means a rebrand, ask 100. If it means a different caption tomorrow, 30 will do.
Next step
Choose a number, then see the price and time together. See pricing and use the estimator. It charges nothing. For the method, read what is preference testing.