Confidence is how strongly the answers you got favor the leading option, for this number of answers. It is not certainty that the leader is objectively the best option. It describes what this group chose in this question.
With a small number of answers, a small lead can happen by chance. A strong lead is unlikely to be explained by chance alone for a sample this size.
The three levels
- High: the lead is large for this number of answers. Chance alone is an unlikely explanation. You can act on it.
- Medium: the leader is ahead, but with this many answers the gap could be chance. Fine for a low-stakes choice.
- Low: the leader's lead is small for this number of answers. Treat the options as close.
How we work it out
We compare the gap between the top two options with how much that gap could vary by chance for this number of answers. That is the standard error of the difference between the two shares.
A gap about two and a half times that size or more is High. About two times is Medium. Anything smaller is Low.
For 1 to 5 scale questions we use a simpler rule based only on the number of answers: 30 or more is High, 15 to 29 is Medium and fewer is Low.
Confidence firms up as answers come in.
When confidence is low
- Ask more heads. Doubling from 50 to 100 narrows the uncertainty noticeably.
- Drop the weakest option. Fewer options spread the answers less thinly.
- Ask why. A free-text follow-up to 20 Tier 2 heads often explains a split.
- Accept the tie. Sometimes the honest answer is that people don't mind.
What confidence doesn't tell you
It says nothing about whether your question was a good one. A leading question can produce a strong lead for the wrong thing. It also doesn't tell you the leader would win with a different group of people. See writing good questions.