A/B testing watches what real visitors do on your live site or product. Preference testing asks people which option they choose before you launch. Use preference testing to cut a long list down. Use A/B testing to confirm what the finalists do with real traffic.
They answer different questions, so they work well in sequence.
The difference in one line each
- Preference testing asks a group to choose between options before launch. You see what people say they would pick.
- A/B testing shows different versions to different live visitors. You see what they actually do, such as click or buy.
Side by side
| Preference testing | A/B testing | |
|---|---|---|
| What it measures | What people choose when asked | What visitors do |
| When | Before launch | After launch, on live traffic |
| Traffic needed | None. You ask a group of people | Enough visitors to see a difference |
| Speed | Minutes to hours | Days to weeks, depending on traffic |
| Risk | None to your live site | Some visitors see the weaker version |
| Best for | Narrowing many options to a few | Confirming the final choice |
| Limit | People may not act as they say | Needs a finished version of each option |
When to use preference testing
- You have more options than you can put live.
- You don't have the traffic for an A/B test.
- You haven't built anything yet.
- You want a direction by tomorrow.
When to use A/B testing
- The page or product is live and has steady traffic.
- The decision depends on behavior, such as sign-ups or purchases.
- You can afford to show some visitors a version that may do worse.
Use both, in order
- Start with eight options. Names, headlines or images.
- Run a preference test. Ask one question. Keep the top two or three.
- Build the finalists.
- Run an A/B test. See which one visitors respond to.
The first step is cheap and fast, which keeps the second step short.
What neither method tells you
- A preference result is not a forecast of sales.
- An A/B result tells you what happened with that traffic, in that period.
- Neither explains why. Ask people for a short reason if you need it.
A small example
You have six headline ideas for a landing page. You ask fifty Tier 1 heads one question: "Which headline is clearest?" That costs €12 and typically takes about ten minutes. Tier 1 means each person has a verified phone.
Say two headlines lead and the rest trail. You build those two and run an A/B test with live visitors. Preference testing removed four options before you spent any traffic on them. The same approach works for headlines, names and images.
The numbers here are illustrative, not from a real study.
Next step
If you are choosing between options before launch, write them as one question. See pricing and try the estimator, which charges nothing. Our guide, what is preference testing, covers the steps in more detail.