Preference testing asks a group of people to choose between two or more options, then shows how their choices split. It tells you which option a group favoured and by how much. It doesn't tell you why, and it doesn't prove the favourite is best.
It works best before launch, when you have a handful of candidates and a choice to make.
When preference testing is useful
Use it when you are comparing two to eight options and one choice is the decision. Typical cases:
- product, brand or feature names;
- logos and packaging;
- headlines and lines of copy;
- thumbnails and ad images.
The reader of the result is usually a product, design, brand or marketing team. They want a direction fast, without building anything first.
When it is not the right method
Preference testing is a poor fit when:
- you need to know whether people will pay. Choosing is not buying;
- you need the reasons behind a choice. Add a written follow-up;
- you need to see how people use something. That is usability testing;
- you need live behaviour from real traffic. That is an A/B test. See A/B testing vs preference testing.
How to run a small preference test
- Name the decision. "We are choosing a name for the app by Friday."
- Pick the criterion. "Which would you trust with your savings?" is a different question from "Which do you like?". Choose the one that matches your decision.
- Limit the options. Two to eight. Fewer is easier to read.
- Write one neutral question. One idea, plain words, no hints. See how to write a question that does not lead people.
- Choose who answers. Everyone, or a group such as an age band or country.
- Choose how many. Fifty is a reasonable start for two to four options. See how many people you need.
- Read the result with its limits in mind.
A worked example
You are choosing between two names for a savings app: Northway and Keel.
The question: "Which name would you trust with your savings?"
The options: Northway, Keel, and Neither.
You ask fifty Tier 1 heads. Heads are the people who answer questions on 50heads. Tier 1 means each has a verified phone. That costs £10.00 and typically takes about ten minutes.
Say the result is Northway 31, Keel 14, Neither 5. The result says Northway led by 34 percentage points among these 50 answers, then says how strong that lead is for a sample this size.
The numbers in this example are made up to show how to read a result. They are not from a real study.
How to describe the result
Say what happened. Don't say what it proves.
| Say this | Not this |
|---|---|
| Northway led by 34 points among these 50 answers. | Northway is the better name. |
| The lead is strong for a sample this size. | The lead is real. |
| People chose Northway when asked which they would trust. | People trust Northway. |
Confidence on 50heads means how strongly the answers favour the leader. It is not certainty that the leader is objectively best. See what confidence means.
What the result does not mean
- It shows what this group chose, in this question, on this day.
- A different group, a different question or a different set of options could give a different split.
- A leading question can produce a strong result for the wrong thing.
- Preference is not quality. People may favour what is familiar over what works best.
What to do next
If the lead is clear, move on with that option. If it is close, you have three choices:
- ask more people;
- remove the weakest option and ask again;
- ask 20 Tier 2 heads, people who have passed an ID check, for a short written reason.
Next step
Write your options as one question and see the price first. See pricing and try the estimator. It charges nothing. To see how a question becomes a result, read how it works.