A poll here means one short question with a few options. A survey means several questions answered by the same person.

When one question is enough

One question works when three things are true:

1. You have a real decision. You are choosing a name, a headline, a thumbnail or a line of copy.
2. The options are on the table. You already have two to eight candidates.
3. One answer will change what you do. If the result wouldn't change your choice, don't ask.

Examples that suit one question:

• Which of these three names would you trust with your savings?
• Which thumbnail would you tap?
• Which of these two lines is clearer?
• Would you order from this menu? Yes, mostly or no.

When you need a survey

Use a longer survey when the answer is a picture, not a choice. For example:

• You need to know why. One question tells you what people picked. It rarely tells you the reason.
• You need to compare groups. Do new customers answer differently from old ones? That needs more answers and more questions per person.
• You need to measure something over time. Tracking satisfaction every quarter needs the same questions each time.
• The answer depends on an earlier answer. If question two only makes sense after question one, you need a flow.
• You need an estimate for a whole population. A small, quick poll shows what this group chose. It doesn't tell you the share of everyone.

50heads does not run multi-question surveys. For those, use a survey tool or a research panel.

A short decision list

Go through these in order. Stop at the first "yes".

1. Do you need to estimate a share of a whole population? Use a survey.
2. Do you need several questions answered by the same person? Use a survey.
3. Do you need to track the same thing over time? Use a survey.
4. Do you have two to eight options and one decision to make? Ask one question.
5. Not sure what the options are yet? Talk to a few customers first. Then come back to step 4.

Why a single question can give a clearer answer

A short question is easy to answer well. Heads, the people who answer questions on 50heads, answer on a phone in about five seconds. There is little to misread and nothing to abandon halfway.

A single question also costs less and comes back sooner. On 50heads, fifty Tier 1 answers cost $13 and typically take about ten minutes. That time is typical, not guaranteed. You can ask again tomorrow with a changed question, which is often more useful than one long study.

What the result does and does not mean

A poll shows what this group chose in this question. It does not show that the winner is best, or that a different group would choose the same.

50heads gives each result one plain sentence: the leader, the margin in percentage points and how strong the lead is for the number of answers. For example, "Menu B wins by 36 points. Confidence high." Confidence means how strongly the answers favor the leader. It is not certainty. See what confidence means.

A leading question can also give a strong result for the wrong thing. Our guide to writing questions that get clear answers covers how to avoid that.

A worked example

You are choosing between two names for a savings app: Northway and Keel.

1. Write the question: "Which name would you trust with your savings?"
2. Add the two options. Add "Neither" if it is a real possibility.
3. Ask fifty Tier 1 heads, which means people with a verified phone.
4. Read the result. Say Northway got 33 answers and Keel got 17. The result line says which led, by how many points, and how strong the lead is.

You now have a direction. You do not yet know why Keel lost, or whether trust is what matters. If that matters, ask the follow-up to 20 Tier 2 heads with free text, which means people who have also passed an ID check and can write a short reason.

The numbers in this example are made up to show how to read a result. They are not from a real study.

Next step

If you have a choice to make, write it as one question and see the price first. See pricing and try the estimator. It charges nothing. To see the steps from question to result, read how it works.

We make 50heads, so we have a view. We have tried to be fair about it. Where PickFu is the better fit, we say so.

Start with the decision you need to make

Fast-feedback tools suit quick choices between a small number of options. Names, headlines, images, packaging, a line of copy.

They do not replace a long questionnaire, a usability study or live sales data. If your decision needs those, you need a different tool. Our guide to one question or a survey covers where the line is.

The eight things to compare

Each row is a question to put to any provider, including us.

Question types

PickFu offers more formats than 50heads. Its published list includes click tests with heatmaps, five-second tests, video, screen recordings and marketplace listing mockups.

50heads offers single and multiple choice, A or B images, pairs, ranking, a 1 to 5 scale, yes, mostly or no, and free text. An image or a 20-second sound clip can sit above the question.

If you need a click test, video or a listing mockup, PickFu is the better fit today.

Who answers

PickFu says it has 15 million or more people across 15 countries and over 100 targeting traits. That is a larger panel than ours and deeper targeting.

50heads is newer. Heads are the people who answer questions on 50heads. We publish pool size bands per country and tier, not a headline panel number. You can target by country, age band, gender and about 120 tags that heads choose to declare. You can combine up to 4 traits.

How answers are checked

Every provider will say its answers come from real people. Ask how they check.

On 50heads, every answer passes through an Apple or Google integrity check on the phone. We reject answers from devices or app builds that fail it. Tier 1 heads have a verified phone number. Tier 2 heads have passed an ID and live-face check, with random re-checks. We also use known-answer questions and timing checks.

These checks reduce fraud. They do not remove it. The full detail is in how we check each answer comes from a real person.

PickFu describes partner identity checks, machine scoring of written answers and an NDA signed by everyone on its panel. We can't see inside those checks, so compare what each provider publishes.

Speed

PickFu's help center says a 50-person poll to a random audience might finish in 15 minutes, and quotes 30 minutes or more as typical. Targeting slows it: one or two traits take an hour or more, and three or four can take a few hours to a day. It adds that most polls finish within a few hours, and many in under an hour. We checked this on 7 October 2026.

On 50heads, 50 answers typically take about ten minutes. That is a typical time, not a guarantee. Rush aims for under an hour at any size, and costs 1.5 times as much.

Price

PickFu's pricing page says "from $1 per response", with sample sizes from 15 to 500. Targeting traits cost extra. We checked this on 7 October 2026.

On 50heads a Tier 1 answer is $0.25, Tier 2 is $0.55 and Tier 3 is $1.00. Each targeting trait adds $0.05 an answer, and each image adds $0.10. Fifty Tier 1 answers to a short question cost $13.

Don't compare headline prices alone. Work out the total for your own question, with the same sample size, targeting and images, on each tool.

Your own audience

Some questions are best put to your own customers rather than a stranger panel. PickFu offers a link you can send to your own audience.

50heads does not. Everyone who answers does so in the 50heads app, after verification. If you need to ask your own customers, PickFu is the better fit today.

API and AI assistants

Both products have an API and an MCP server. MCP is a way for a compatible AI assistant to use outside tools.

On 50heads, every ask needs an idempotency key, so a retry never asks the same question twice. You can set a daily spend cap on each connection. Results can wait on our side instead of being polled. See 50heads for AI assistants.

PickFu also offers a Zapier integration, which can send results to Slack. 50heads has no Zapier or Slack integration as of 7 October 2026. It has a REST API, signed webhooks, an MCP server and a command-line tool.

Sharing results

On 50heads, you can switch on a public page for any result. It shows the question, the bars and how strongly the answers favor the leader. It shows nothing about individual answers or about you. Public pages are hidden from search engines.

PickFu offers PDF and PNG reports and filtered links. Look at a sample report from each tool before you decide.

When to choose which

Choose PickFu when you need:

• the written reason behind every answer;
• a format we don't have, such as a click test, video or listing mockup;
• targeting beyond country, age band, gender and our tags;
• a link to send to your own customers;
• a team workspace today.

Choose 50heads when you want:

• a quick read on one thing at a price that lets you ask again tomorrow;
• an exact sample size, such as 37, within our limits;
• verified phones on every answer;
• to ask from a script or an AI assistant with a spend cap.

Many teams will use more than one tool. That is fine.

Check the facts yourself

Competitor features and prices change. The PickFu figures here were checked on 7 October 2026 against its public site. Please confirm on PickFu's pricing page before you decide.

For the full side-by-side, see 50heads and PickFu. It lists every row we compared and the date we checked it.

Next step

See what your question would cost on 50heads. See pricing and try the estimator. It charges nothing.

Amazon's own site says Mechanical Turk closed on 30 September 2026. AWS's service availability notice lists it as reaching end of support on 29 September. Either way, new HITs can no longer be posted. A HIT is MTurk's word for one task.

For anything about your balance, results or data on MTurk, follow Amazon's notices. We can't tell you what Amazon will do.

Which jobs move to 50heads

50heads does one narrow thing. You ask one short question with a few options. Real people on verified phones answer it, typically in about ten minutes. You see how the answers split.

Your MTurk job moves across if it was really one of these:

1. Pick one of several. Names, headlines, thumbnails.
2. A or B with images. Two pictures, one tap.
3. A 1–5 rating. A choice usually tells you more.
4. A short reason in writing. Up to 200 characters, from Tier 2.

Tiers show how strongly each person answering has been verified. Tier 1 is a verified phone. Tier 2 adds an ID and live-face check.

Which jobs do not move

These don't fit 50heads, and we would rather say so now:

• transcription;
• labeling or tagging many items;
• multi-page surveys;
• anything needing hundreds of steps from the same person;
• academic studies with their own consent flow.

For those, look at a general crowdsourcing service or a research panel. If the only thing you ever asked was "which of these?", read on.

What maps to what

What it costs

While MTurk ran, you set the reward and Amazon added its commission: 20%, or 40% on HITs with ten or more assignments. Those figures come from Amazon's published pricing page, which was still online when we checked it on 7 October 2026. At those rates, fifty assignments at $0.10 each cost $7.00.

On 50heads a Tier 1 answer is $0.25. Fifty of them cost $13, about ten minutes, with no separate fee. The price covers verification, the head's pay and our margin. Heads get 60% of it.

You see the total and the time before you post. If fewer heads answer than you asked for, the rest goes back to your balance.

What you stop doing

Screening for bots. For every answer, the 50heads app asks Apple or Google for an integrity check. We reject answers from devices or app builds that fail it. Tier 1 heads have one verified phone number each. Tier 2 heads have passed an ID and live-face check. We also run known-answer questions and timing checks.

These checks reduce fraud. They don't remove it. If your research design needs its own validity checks, keep them. Adding them makes the question longer, which costs more.

Approving and rejecting. We remove answers that fail our checks. If we remove an answer after it counted, you get the credit back and the result is recalculated, as set out in our refund policy.

Working out the reward. The price depends on the tier, how long the question is to read and the kind of answer. The estimate is free and never spends anything. The time shown is an estimate, not a guarantee.

How to rewrite a HIT as a question

Keep questions short. Heads answer in about five seconds, on a phone, between other things. If your HIT had a paragraph of instructions, the question isn't ready yet.

Our guide to writing questions that get clear answers has the rules we use ourselves. In short:

• ask one thing;
• give two to eight options;
• describe, don't sell.

Moving over in an afternoon

1. Sign in and add credits. Packs start at $26.
2. Take your most common HIT and rewrite it as one question with two to eight options. Check the price and time in the estimator.
3. Ask it to fifty Tier 1 heads and compare the result with your last MTurk batch. The result says how strongly the answers favor the leading option for a sample of that size.
4. If your code used to call MTurk, those calls no longer work. Swap them for the 50heads API, or connect an AI assistant with the MCP server. The quick start has curl, TypeScript and Python.

curl https://api.50heads.com/v1/billing/estimate \
  -H "Content-Type: application/json" \
  -d '{"draft":{"type":"single_choice","text":"Which name would you trust with your savings?","options":[{"label":"Northway"},{"label":"Keel"}],"n":50,"tier":1}}'

How the two compared

If you want the side-by-side table, see 50heads and Amazon Mechanical Turk. To compare 50heads with other fast-feedback tools, read PickFu alternatives.

Next step

Work out what your first question would cost. See pricing and try the estimator. It takes a minute and charges nothing.

Heads are the people who answer questions on 50heads. Tiers tell you how strongly the person answering has been verified: Tier 1 is a verified phone, Tier 2 adds an ID and live-face check, and Tier 3 (Established) is by invitation after a record of careful answers. No check is perfect, so we use several and keep adding to them.

The problem we are solving

Paying strangers for opinions attracts three kinds of cheating. Scripts that answer at machine speed. Farms of phones run by one person. And people who are real but not paying attention. Each needs a different check, and a check is only worth having if it is costly to get round.

Check one: the phone is a real phone

When a head answers, the 50heads app asks Apple or Google for an integrity check. We reject answers from devices or app builds that fail it. The check is designed to tell us whether:

• the app is running on a genuine, physical device rather than an emulator;
• the device has not been rooted, jailbroken or tampered with;
• the app sending the answer is the one we published, unmodified.

If an answer fails, it does not count towards your result and the head is not paid. This makes it much harder to answer 50heads questions from a script on a server. It does not make it impossible, which is why the other checks below exist.

Each answer in your results carries an integrity-check reference. It shows the answer passed the check on a device, without saying whose.

Check two: one real phone number per head

To be paid, a head verifies a phone number with a code by text or WhatsApp. Virtual numbers and VoIP numbers are refused. A number can belong to one account only.

Numbers are cheap in some places, so we don't stop there. Every month, a Tier 1 head who earned in that month confirms the same number again. It takes about ten seconds for someone who still has their phone, and it makes it more costly for a farm that rotates SIM cards.

Check three: a real identity, at Tier 2

Heads who want the higher-paying Tier 2 questions pass an ID and live-face check, once. It is run by a specialist provider, not by us. The provider checks the document and checks that the face in front of the camera matches it and is live. It also keeps a face match, not the photos, to help stop one person holding two accounts.

We receive a yes or a no. The ID and face photos are deleted by the provider after its decision, within 30 days. We never see them. Tier 2 heads also get a three-second face check at random afterwards, which makes it harder to verify an account once and then hand it on.

Check four: attention

A verified person can still click without reading. So some questions have answers we already know, mixed in at random. Heads who get several of them wrong are reviewed by a person, who can move them back to warm-up questions for two weeks. Answers that arrive faster than the question could have been read are not paid, and answers that arrive in lockstep with other accounts are held back and reviewed.

What happens when we get it wrong

Automated checks make mistakes, and the people on the other end of them deserve a way out. A person decides any restriction and tells the head why in writing. Any head can appeal at any time.

A person who was not part of the decision decides appeals about an unpaid answer within 48 hours and other appeals within five business days. Either way, you get a written reply within two weeks at the latest. We publish how many appeals we receive and uphold each quarter in our transparency note.

If we remove an answer after it has counted towards your result, you get the credit back and the result is recalculated.

What we keep

We keep as little as the checks need, for as long as they need it:

The full detail is in our privacy notice and on how we verify.

Why publish it

Because "trust us" is not a verification method. If you can read how the checks work, you can decide whether they are good enough for your question, and you can hold us to them. Answers are anonymous to the requester. 50heads holds the phone and verification data described above.

The short version

• Tiers are verification levels. Verify more to qualify for higher-paying questions.
• A Tier 1 answer pays $0.15. A Tier 2 answer pays $0.30, about 2.1 times as much. Tier 3 (Established, by invitation) pays $0.60.
• Heads get 60% of every answer's price. The rest pays for verification, the platform and the company.
• You see what a question pays before you answer it.
• New earnings are Pending for up to 7 days (3 at Tier 3). Then they are Available, and you can withdraw from $7, or let us pay you automatically at the end of the month.
• Warm-up answers before you verify your phone are earned into Pending too, and become Available once you verify.

Where the money goes

A requester pays $0.25 for a Tier 1 answer. Of that, $0.15 goes to the head who gave it. The remainder covers the device and phone checks on every answer, the ID provider for Tier 2, payment fees, the servers, support, and the people who build and run 50heads.

We picked a fixed share because it is easy to check: if a requester pays more for a question (a longer one, or one with images, or a rush), the head gets more too, in proportion.

Pay is the same everywhere

We pay the same rate in every country we pay, before currency conversion. We looked hard at adjusting pay by country and decided against it. The work is the same, and a five-second answer from Manila is worth as much to a requester as one from Manchester. The pay rates by country page shows the rates in local currency, the payout methods in each country and the smallest withdrawal.

Getting paid

Heads can withdraw to PayPal everywhere we pay, to a UK bank account in the UK, to Venmo or a bank account in the US, and to a bank transfer in most other countries. Money arrives in your own currency: PayPal pays in pounds, dollars, euros or Canadian dollars, and a bank in its own currency.

The first withdrawal each month is free, except a bank transfer that needs a currency exchange. After that, PayPal and Venmo cost 2% up to $25, a UK bank $0.25 and a bank transfer $0.65. Withdrawals go out at 09:00 and 17:00 UK time on business days: PayPal and Venmo land the same or the next UK business day, a UK bank the same business day, and a bank transfer in 1–3 business days. The Withdraw screen shows exactly what will arrive, after any fees and currency exchange, before you confirm.

Why the holding period? Because some checks can only run after the fact: known-answer questions, timing patterns across many answers, and requester disputes. Holding new earnings for up to a week reduces the need to recover money after payout. Pending means earned and still inside the holding period; Available means cleared and ready to withdraw.

The honest line

Answering questions on 50heads is not a full-time income, and we won't pretend it is. It is five-second questions in spare minutes: on the bus, in a queue, while the kettle boils. How much you earn depends mostly on how many questions there are in your language at the time, and we tell you when there's nothing waiting rather than filling the gap with unpaid busywork.

What we can promise is that every accepted answer pays what it said it would, that the rates are public, and that you can always see what is Pending and what is Available.

What we publish, and when

Our fair pay statement sets out the floor we won't go below and how we'll tell heads before any rate changes. Our transparency note reports appeals received and upheld each quarter.

If you think a rate is wrong, or a payout didn't arrive, tell us. A person will reply.

The minimum useful result

Leave out anything that doesn't change what the reader does.

A one-page template

Copy this and fill it in:

Decision: [what we are choosing]
Question asked: [exact wording]
Options: [A / B / C]
Who answered: [audience] · [number] answers · [date]
Result: [Option] led by [number] points among these [number] answers.
Strength: [High / Medium / Low] for this number of answers.
Why: [two or three short reasons, if asked]
Limits: This shows what this group chose. It is not a forecast of sales.
Next: [what we will do]
Full result: [link]

The "Limits" line keeps the result honest. Don't remove it.

How to describe the result

Ways to share it on 50heads

• Public page. A page with the question, the bars and the summary. It never shows who answered. It is hidden from search engines. Anyone with the link can open it without signing in. Switch it off and the link stops working.
• Private link. An unlisted link to the full results, written answers included, for people without an account. It never shows who answered, and search engines don't index it.
• PDF report. A one-page report to attach or print.
• Image. The chart as a square, landscape or story image.
• CSV. Every answer: the option or text, the tier, the time and the verification reference. It never includes who answered.

Choose the one that fits the reader. A public page suits something you will post. A private link suits a colleague who needs the written reasons.

If you publish it

Say the result came from 50heads and how many heads answered. Our terms for requesters set this out. You may use results in your own work, decks and publications.

What to leave out

• Individual answers, unless the reader needs the written reasons.
• Claims that go beyond the result. "Customers want B" is not what the result says.
• A sample of one question as proof of a strategy. One result is one input.

Next step

Open a finished result and make a public page or a PDF report. To get a result to share, write one question and see the price first. See pricing and try the estimator. It charges nothing. For more on reading results, see reading your results.

Heads are the people who answer questions on 50heads. You choose who sees your question when you ask it.

What you can target

• Country. One or more. Countries are free.
• Age, gender and tags. Heads say these about themselves, optionally, in the app. There are about 120 tags in 12 groups, such as "parent of young children" or "plays console games".
• ID-checked age. At Tier 2 you can ask for ages taken from an ID check, rather than what heads declared.

Age, gender and each tag group count as one trait. You can combine up to 4. Every trait must match. Inside one trait, any value matches, so "dog owner or cat owner" is one trait.

What it costs

Each trait adds $0.05 an answer. Fifty Tier 1 answers cost $13 with no targeting. With two traits, such as women aged 25 to 34, they cost $17, which is $0.35 an answer.

The composer shows the price, how many heads match and how long it should take, next to the targeting, as you choose.

How to choose

Ask three things:

1. Would this group answer differently? If parents of young children would pick a different pack than everyone else, target them. If not, don't.
2. Is one trait enough? Try one before you add a second.
3. Can the question be answered by anyone? Many first-impression questions don't need targeting.

Over-targeting

Every condition shrinks the pool. A small pool takes longer, and a question may close before it fills. If fewer heads answer than you asked for, you pay only for the answers you get, and the result shows how many answered.

Targets smaller than 200 heads are refused. That means no group is small enough to identify anyone.

An example

You are testing a children's snack. You could ask everyone, or only parents of young children.

• Ask everyone first: "Which pack looks tastier?"
• If you think parents would choose differently, add one trait: the parent tag.
• Compare the two results.

If both groups pick the same pack, you didn't need the extra trait. If they differ, you have learned something. For how to test the packs themselves, see how to test packaging. For how many people to ask in each group, see how many people you need.

What targeting does not do

• It is not verified. Traits are what heads say about themselves. We don't check them independently, except an age band set from an ID check.
• It does not make a small sample representative. Fifty answers from a targeted group show what those people chose.
• It does not split results freely. Results can be broken down by country, tier, age band and gender, but only for groups of 5 or more answers.
• Nothing about a head's age or gender is shown with an answer.

Saving an audience

If you ask the same group often, set who answers in the composer and save it as an audience at the foot of that card. Then use it in a new question.

Next step

Choose a country first, then see what an extra trait costs. See pricing and use the estimator. It charges nothing. For the rules, read targeting by country, age, gender and tag.

The skill is knowing when to ask. People should be a deliberate step, not another automatic tool call.

When an agent should ask

Ask people when the agent is facing a choice that depends on human taste:

• picking between draft headlines, names or images;
• checking whether a message reads as clear;
• choosing the most appealing of a few options before it publishes.

Don't ask when:

• the answer is a fact the agent can look up;
• the question needs private information;
• the action is hard to undo and a person on your team should decide;
• a recent result already answers it.

50heads measures what people prefer. It is not a way to check facts.

The flow

Three calls, in order:

1. estimate: price, time and validation. Free, and it never spends.
2. ask: posts the question with an idempotency key. It returns a task.
3. get_results: reads the result, partial while live. Older hosts use wait_for_results.

> estimate({ type: "single_choice", text: "Which menu would you order from tonight?",
             options: [{ label: "Menu A" }, { label: "Menu B" }], n: 50, tier: 1 })

> ask({ ...same question, idempotency_key: "4f9c…" })
  task q_8k2f · working · 0 of 50 answered

> get_results("q_8k2f")
  complete · winner "Menu B" · margin 36 points · confidence high

The key is a fresh UUID for each question, so a retry never asks twice.

Guardrails

• Spend cap. When you connect an assistant, 50heads asks you to review what it can do and set a daily cap. An ask past the cap is refused with spend_cap_exceeded, and nothing is spent.
• Estimate first. Tell the agent to show you the price and time before it asks. Short, single questions work best.
• Reuse before re-asking. The agent can list recent questions and read an existing result.
• Narrow scopes. An agent can estimate, ask, read results, cancel a live question, list templates and check the balance. It can't buy credits, see payout details or answer questions.
• Revoke any time. In Developers, under Connections, you can lower the cap, remove scopes or disconnect. Revoking takes effect within a second.

Connecting it

Add the hosted server at https://mcp.50heads.com/mcp:

• Claude Code: claude mcp add --transport http 50heads https://mcp.50heads.com/mcp
• Claude, Cursor and ChatGPT: add it as a connector or in the settings, with the same URL.

The first time, your browser opens a 50heads page where you sign in and set the cap. The MCP docs have the config for each host, and connect Claude, Cursor or ChatGPT covers the steps.

Writing the rule for your agent

Give the agent a short instruction, such as:

Before publishing a headline, ask 50 people which of the two drafts is clearer. Show me the price and time first. Don't ask the same question twice. If the lead is weak, tell me and don't choose for me.

The 50heads Agent Skill teaches an assistant how to word questions and when to ask.

What the result does and does not mean

It shows what this group chose in this question. A leading question gives a strong lead for the wrong thing. Tell the agent to describe options plainly. See how to write a question that does not lead people.

A weak lead should return control to a person, not trigger an arbitrary choice.

Next step

Connect an assistant and ask it to run an estimate. It charges nothing. For more, see 50heads for AI assistants and how to add a human feedback API to a product workflow.

This post uses the 50heads API. The developer docs and API reference are the source of truth for every field.

When a human check is worth making

A person's choice is the right input when:

• the decision is a preference, such as which headline, name or image;
• a wrong choice costs more than the question;
• an automated check can't settle it.

It is the wrong input for facts you can look up, or when you need the answer in under a second. A typical result takes about ten minutes. That is typical, not guaranteed.

The four steps

1. Estimate. Price, time and validation. It never spends.
2. Ask. Post the question with an idempotency key.
3. Wait. The API holds your request until answers arrive.
4. Act. Read the leader, the margin and how strong the lead is.

Example

Set a key, then run the three calls. Keys are for servers. Requests from a browser are refused, so never put a key in a web page.

# 1. Estimate: free, no key needed
curl https://api.50heads.com/v1/billing/estimate \
  -H "Content-Type: application/json" \
  -d '{"draft":{"type":"single_choice","text":"Which menu would you order from tonight?",
       "options":[{"label":"Menu A"},{"label":"Menu B"}],"n":50,"tier":1}}'

# 2. Ask: one UUID per question
curl https://api.50heads.com/v2/questions \
  -H "Authorization: Bearer $FIFTYHEADS_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{"draft":{"type":"single_choice","text":"Which menu would you order from tonight?",
       "options":[{"label":"Menu A"},{"label":"Menu B"}],"n":50,"tier":1,"language":"en"}}'

# 3. Wait: up to 25 seconds a call, repeat while "done" is false
curl "https://api.50heads.com/v2/questions/q_8k2f/wait?seconds=25" \
  -H "Authorization: Bearer $FIFTYHEADS_API_KEY"

A fifty-answer Tier 1 question costs $13. The quick start has the same flow in TypeScript and Python.

Idempotency keys

POST /v2/questions needs an Idempotency-Key header, a UUID you make for each question.

• If a request times out, send it again with the same key. You get the original question back, never a second ask.
• Keys are kept for 24 hours.
• The same key with a different body returns idempotency_conflict.

Make the key before the first attempt and store it with your record of the decision. If you create a new key on each retry, you defeat the point.

Waiting for the result

Don't poll in a tight loop. GET /v2/questions/{id}/wait?seconds=25 holds the request until the question is answered or 25 seconds pass. Call it again while done is false.

Or register a webhook and we will call your URL. Check the signature before you trust the payload, and use the event id to ignore repeats, as deliveries can arrive twice or out of order.

Set your own overall deadline. If the answer doesn't arrive in time, decide what your product does anyway.

Failure paths

Branch on the error code, never on the message.

Also plan for an underfilled question. If fewer heads answer than you asked for, you pay only for the answers you get, and the result shows how many answered. Check the count before you act.

Keep spend bounded

Every key has scopes and a daily spend cap, set when you make it. See API keys and spend caps. Give each key only the scopes it needs. Use list_questions in the MCP server, or the question list in the API, to reuse a recent answer before asking again.

Acting on the result

The result gives the leader, the margin in percentage points and how strong the lead is for the number of answers. Decide your rule in advance:

• Strong lead: proceed with the leader.
• Weak lead: treat the options as close, and choose on another basis or ask more.

The result shows what this group chose. It is not proof that the winner is best.

Next step

Make a key and run the estimate. Open the docs for the quick start, or see pricing. The estimate charges nothing.

For how 20, 50 and 100 compare, see how many people you need for a preference test.

Three shapes of result

Most results from 50 answers look like one of these. The numbers below are made up to show the shape. They are not from real studies.

The strength labels follow the rule in what confidence means. We compare the gap between the top two options with how much that gap could vary by chance for this number of answers.

Narrow lead

A 27 to 23 result says A was ahead among these 50 answers. It says very little more. A gap this small can happen by chance with 50 people.

Treat the options as close. Choose on other grounds, such as cost or fit, or ask more people.

Strong lead

A 38 to 12 result is a large gap for 50 answers. Chance alone is an unlikely explanation. You can act on it, if the question was fair and the group was right.

A strong lead is still not proof that A is better. It shows what this group chose.

Three-way split

When answers spread across three options, no single option is clearly ahead. That is information too. It may mean the options are similar, or that people don't mind.

Consider dropping the weakest option and asking again. Fewer options spread the answers less thinly.

What 50 answers cannot tell you

• What a population thinks. A quick preference test shows what this group chose. It doesn't estimate the share of a country or market.
• Why. Ask 20 Tier 2 heads, people who have also passed an ID check, for a short written reason.
• Differences between small groups. Results can be broken down by country, tier, age band and gender, but only for groups of 5 or more answers. With 50 answers, slicing leaves groups too small to read.
• What people will do. Choosing when asked is not buying.
• Whether the question was fair. A leading question gives a strong lead for the wrong thing.

Rating questions work differently

A 1 to 5 scale question is read by its average, not by a lead. The result shows the average and how far it could be off, give or take. The more the answers cluster, and the more answers there are, the tighter that range and the stronger the confidence. With fewer than 10 answers it is always Low.

The average says how much people like one thing. It doesn't say which of two things they prefer. For a choice, ask a choice.

How to describe the result

If you publish a result, say it came from 50heads and how many heads answered. Our terms for requesters set this out.

What to do next

• Strong lead: act on it, and consider a follow-up asking why.
• Narrow lead: ask more people, or accept the tie.
• Split: drop the weakest option and ask again.

On 50heads you can add more heads to the same question and the result merges them. The estimate shows the cost and time first.

Next step

Start with a question and see the price. See pricing and try the estimator. It charges nothing. Fifty Tier 1 answers to a short question cost $13 and typically take about ten minutes.

Fast feedback is a way to narrow choices. It is not a replacement for talking to customers.

The six decisions

Each question goes to its own group of heads, the people who answer questions on 50heads. Don't treat results from different questions as coming from the same people.

1. Name

Ask about one quality: trust, fit or memorability. Keep each name within 40 characters.

Don't use it when the name must be legally available or work in other languages. Check those separately. See how to test a product name.

2. Message

Compare two to four lines for clarity. Ask "Which line is clearer?" before "Which makes you want to know more?"

Don't use it when the message depends on the page around it. People will see it with an image and a button.

3. Concept

Put two to four concepts in a line or picture each. Ask which they would be most likely to use.

Don't use it to find out whether people will pay. Choosing is free. See how to test a product idea.

4. Visual

Use an A or B images question, which takes exactly two pictures. Ask about one quality, such as which they would notice first.

Don't use it to judge print quality or how a design feels. Heads see images on a phone.

5. Priority

Use a Rank them question, where people drag options into order. You see the average rank of each option. Ranking is priced at 1.3 times the base price.

Don't use it with more than eight options. Ranking takes more effort than choosing, so keep the list short.

6. First impression

Show the product and ask what people think it is for, with a few possible answers. If most pick the wrong one, your page isn't clear.

Don't use it as the only test of usability. It shows what people think at a glance, not whether they can use it.

How to keep it organized

In Projects, you can group questions by what you are trying to find out, so each round sits next to the last. Templates are ready-made question formats at a fixed price, which save you setting up the same shape again.

What fast feedback does not tell you

• It shows what this group chose, not what all your customers think.
• It doesn't explain why. Ask 20 Tier 2 heads, who have also passed an ID check, for a short reason.
• It doesn't replace interviews, usage data or sales.

Next step

Pick the decision in front of you this week and write it as one question. See pricing and try the estimator. It charges nothing. Fifty Tier 1 answers to a short question cost $13 and typically take about ten minutes.

We can describe our own pay model exactly. We can't prove that it improves your results, and we don't claim to.

What the research says

Studies of online participants point in different directions:

• Earlier work on Amazon Mechanical Turk found that pay affected the quantity of work more than its quality. Those studies generally looked at rates below minimum wage.
• A 2015 study by Litman, Robinson and Rosenzweig, published in Behavior Research Methods, compared US and India-based workers. It found that data quality among India-based workers was affected by compensation, at rates set around the minimum wage there. It also found that quality was lower for India-based than for US workers even when pay was set well.

Treat these as pointers to the research, not a full review of it. Pay is one factor among many. Instructions, task length, screening and checks matter too.

What we can say

Fair pay doesn't guarantee good answers. It does remove one reason to rush: a person paid so little that carelessness is the rational choice.

We also think it is simply right. People who answer questions deserve clear pay that they can check.

How 50heads splits the money

A requester pays $0.25 for a Tier 1 answer. Heads, the people who answer questions on 50heads, get 60% of every answer's price. For a Tier 1 answer, that is $0.15.

The rest covers:

• the device and phone checks on every answer;
• the ID provider for Tier 2;
• payment fees;
• servers and support;
• the people who build and run 50heads.

These are for a short pick-one question. Longer questions, images and rush answers cost more and pay more in proportion.

What else we commit to

Our fair pay statement sets these out:

• the same rate in every country, before currency conversion;
• pay shown before the head answers;
• a floor: a Tier 1 answer never pays less than $0.15;
• at least 30 days' notice in the app before any change that lowers pay;
• a person decides appeals, and we publish the numbers.

See the rates in local currency on pay rates by country.

What it is not

Answering questions is not a full-time income. Questions come in bursts, and how much a head can earn in an hour depends on how many are waiting. We don't promise an hourly rate.

What this means for a requester

• You don't pay extra for fair pay. It is built into the per-answer price.
• Your result depends mostly on your question, who you asked and how many answered.
• Pay is one reason to trust a panel more, alongside identity checks and published appeals. See what makes an online research panel high quality.

Next step

See what your question would cost and what the heads who answer it earn. See pricing and use the estimator. It charges nothing.

Use this list on 50heads or on any other provider.

The seven questions

1. How do you check who is answering?

Ask what "verified" means. A verified phone number is not the same as a checked ID, and neither is the same as an established track record.

On 50heads, Tier 1 is a verified phone number. Tier 2 adds an ID and live-face check. Tier 3 is by invitation after a long record of careful answers.

2. How do you stop one person answering many times?

Ask about phone numbers, identity matching and device checks. Ask how often they re-check.

On 50heads, a phone number can belong to one account, and Tier 1 heads confirm it again monthly if they earned in that month.

3. How do you catch careless answers?

Ask about known-answer questions, timing checks and what happens to repeat offenders.

On 50heads, hidden known-answer questions and timing checks run on every head. Careless patterns lead to review by a person.

4. What do you check about the device?

Ask whether the provider checks that an answer came from a genuine device running its own app.

On 50heads, every answer passes an Apple or Google integrity check, and we reject answers that fail it.

5. How are people paid?

Pay affects who takes part and how carefully. Ask what the share is, whether rates are published and whether you can see them.

On 50heads, heads get 60% of every answer's price, and the rates are public. See what heads earn and does paying people fairly improve research quality.

6. What do you publish?

Ask whether the provider reports its own mistakes, such as appeals received and upheld.

On 50heads, we publish appeals received and upheld each quarter in our transparency note. The first report covers October to December 2026 and is due in January 2027, so there are no figures yet.

7. What can't you do?

Ask what the panel is not suited for. Every panel has limits.

Known limits on 50heads

We would rather you heard them from us:

• It is one short question at a time, not a multi-page survey.
• The panel is new. We publish pool size bands per country and tier rather than a total.
• Targeting traits are what heads say about themselves. We don't check them, except an age band taken from an ID check.
• Verification reduces fraud. It does not remove it.
• A preference result shows what this group chose. It is not a population estimate.

Scoring a provider

Give each question a mark:

• Specific: names the check and how often it runs.
• Vague: says "real people" or "high quality" with no detail.
• Missing: doesn't answer.

Most of the value is in the difference between the first two.

Next step

If a short question to verified phones fits your decision, see pricing and try the estimator. It charges nothing. To see the checks in detail, read how we check each answer comes from a real person.

This post separates the threats and says what 50heads does about each. No check is perfect, and we don't claim any answer is guaranteed to be human.

Four threats, not one

Putting them in one word hides which problem a check is meant to solve.

Fake or duplicate accounts

If one person can open fifty accounts, one person can write fifty answers.

On 50heads, Tier 1 heads verify one phone number each. Virtual and VoIP numbers are refused, and a number can belong to one account only. Every month, a Tier 1 head who earned that month confirms the same number again. Tier 2 heads also pass an ID and live-face check, with random face re-checks afterwards. Tiers tell you how strongly the person answering has been verified.

This makes running many accounts harder and more costly. It does not make it impossible.

Automated environments

A script on a server, an emulator or a modified app can answer questions at machine speed.

For every answer, the 50heads app asks Apple or Google for an integrity check. We reject answers from devices or app builds that fail it. The check is designed to tell us whether the device is a genuine physical one, whether it has been tampered with and whether the app is the one we published.

This makes scripted answering much harder. It is not a promise that no automation can get through. That is why the other checks exist.

Inattentive answering

A real person can still click without reading.

Some questions have answers we already know, mixed in at random. Heads who get several wrong are reviewed by a person. Answers that arrive faster than the question could have been read are not paid. Answers that arrive in lockstep with other accounts are held back and reviewed.

AI-assisted written answers

A person can paste a model's text into a free-text box.

We don't claim to detect this. Free-text answers on 50heads come only from Tier 2 heads, who have passed an ID check, and are limited to 200 characters. If your decision depends on free text, read it as the view of one verified person, not as proof that no tool helped.

What a requester can see

Each answer in your results shows its tier and an integrity-check reference. The reference shows the answer passed the device and app check. It never shows who answered. If an answer looks wrong, you can flag it for review, and an upheld flag is refunded.

Questions to ask any provider

• How do you stop one person running several accounts?
• What do you check about the device or app?
• What happens to an answer that fails a check, and do I pay for it?
• How do you handle people who answer carelessly?
• How do you treat AI-written free text?
• What do you publish about mistakes and appeals?
• What do you say you cannot do?

A good answer is specific. A weak answer repeats "real people" or "no bots" without saying how.

Where to read more

The detail of each check, and what we keep, is in how we check each answer comes from a real person and on how we verify. Appeals and decisions are in our transparency note.

Next step

If you want answers from verified phones, see pricing and try the estimator. It charges nothing.

The test is simple. Ask what your source of truth is.

Three kinds of source of truth

If you can check the answer without asking anyone, you don't need a panel. If the answer is a choice that real people make, you do.

Where a model is useful

A model can help you before you ask anyone:

• draft ten name ideas, then cut them to four;
• spot unclear wording in a question;
• summarize the written reasons people gave afterwards;
• write the code that calls an API.

These are jobs where the output can be reviewed by you.

Where a model is not your customer

A model can imitate how a person might respond. That is not the same as measuring people. Three limits matter:

• It has not chosen anything. A model's answer is generated, not decided by a person with a budget, a phone and a mood.
• It can sound sure and be wrong. A confident explanation of why people prefer option B is not evidence that they do.
• It cannot be a new group. If your customers are parents in Leeds, a model has not met them.

For taste, trust and first impressions, the thing you want to measure is a human reaction.

A short decision list

1. Can you check the answer against a document, a test or a rule? Use a model or a test.
2. Is the answer what a group of people would choose? Ask people.
3. Do you need both? Use the model to prepare the question, then ask people.

Using both together

Many teams combine them:

1. A model drafts six headline options.
2. You trim them to three.
3. People choose between the three, on their phones.
4. A model summarises the written reasons from a short follow-up.

The human step is the only one that measures what people choose. Everything around it is preparation or reading.

Letting an AI assistant ask people

An AI assistant can ask real people on your behalf, so a person's choice becomes part of its workflow. On 50heads, a compatible assistant can connect through MCP, a way for an AI assistant to use outside tools. It estimates the price first, asks, waits for the result and reads it. You set a daily spend cap when you connect it.

See how an AI agent can ask real people before it acts and 50heads for AI assistants.

What a human result still does not mean

• It shows what this group chose in this question. A different group could choose differently.
• It is not a forecast of behavior.
• A leading question gives a strong result for the wrong thing. See how to write a question that does not lead people.

Next step

If your question is "which would people choose?", write it as one question and see the price. See pricing and try the estimator. It charges nothing.

Printing the wrong design is expensive. A quick test is cheap.

Test one quality at a time

Packaging does several jobs. Each one is a separate question:

A pack that wins on stand-out can lose on clarity. One overloaded question hides that.

A test sequence

1. Start with stand-out. If nobody notices the pack, the other qualities matter less.
2. Then ask about clarity. Can people tell what it is?
3. Then the quality your brand depends on, such as trust or premium feel.
4. Keep the winner. Compare it against the next design.

Each question goes to its own group of heads, the people who answer questions on 50heads. Don't treat the results as coming from the same people.

How to run each test

1. Prepare two images. Use the same angle, background and size. A better photograph will beat a better design.
2. Write one neutral question. See how to write a question that does not lead people.
3. Use an A or B images question. People tap one of two pictures. Images add $0.10 an answer each, and the estimator shows the total before you ask.
4. Add "Neither" if both could fail. It shows in gray in the results.
5. Choose who answers. Target people who buy in the category. Each trait adds $0.05 an answer. Heads say these traits about themselves in the app, and we don't check them independently, except an age band set from an ID check.
6. Ask fifty to start. Fifty Tier 1 answers to a short question cost $13 before images and typically take about ten minutes. Tier 1 means each person has a verified phone.

Test at phone size

Heads see your images on a phone. That is useful: a pack that reads well at that size will probably read well in a small online listing. It is also a limit. A phone screen doesn't show the feel of the material, the size on a shelf or the finish.

Use it to test what the design communicates. Don't use it to judge print quality.

A worked example

You have two designs for a fictional oat snack: one with a photograph of oats, one with a bold flat color and a large name.

You ask fifty Tier 1 heads: "Which pack would you notice first on a shelf of similar products?" Say the result is the flat color 35 and the photograph 15. The result line says the flat color led by 40 percentage points among these 50 answers, and the lead is strong for a sample this size.

Then you ask a second group: "Which pack makes it clearer what is inside?" Say the photograph wins. You now know the bold pack stands out and the photograph explains. That may point to a design with both.

The numbers in this example are made up to show how to read a result. They are not from a real study.

What a packaging test does not tell you

• Whether people will buy it. Choosing a pack when asked is not buying.
• How it looks in a real shop. Lighting, neighbouring products and position all matter.
• How it feels. A screen shows no material or finish.
• Whether a different group would choose the same. The result shows what this group chose.
• Anything about real brands. Use your own designs, or fictional ones, never someone else's pack without permission.

Next step

Prepare two images and see the price first. See pricing and try the estimator. It charges nothing. For the method behind this, read what is preference testing.

This removes obvious losers cheaply. It does not predict clicks or sales.

Why test before you spend

Every impression of a weak ad costs money. A live test also needs enough traffic to separate the winners, which takes time and budget.

A quick preference test with a small group costs a fraction of that. Use it as a first filter. See A/B testing vs preference testing for how the two fit together.

What to ask about

Pick one quality per question:

Ask each as its own question. An ad can grab attention and still confuse. Each question goes to its own group of heads, the people who answer questions on 50heads, so the results come from different people.

How to run the test

1. Prepare two versions. Change one thing between them, such as the image or the first line, so you know what drove the result. Use the same size and framing.
2. Write one neutral question. See how to write a question that does not lead people.
3. Use an A or B images question. People tap one of two pictures. Images add $0.10 an answer each, and the estimator shows the total before you ask. For text-only hooks, options can be up to 40 characters.
4. Show it as it will appear. Heads answer on a phone, so a mobile ad should be tested at phone size. Small text that is clear on your desktop may not be on theirs.
5. Choose who answers. Target the audience you will buy. Each trait adds $0.05 an answer.
6. Ask fifty to start. Fifty Tier 1 answers to a short question cost $13 before images and typically take about ten minutes. Tier 1 means each person has a verified phone.

A worked example

You have two ad images for a savings app: a close-up of a coin jar and a phone screen showing a balance.

You ask fifty Tier 1 heads: "Which image would you stop to look at?" Say the coin jar gets 38 and the phone screen 12. The result line says the coin jar led by 52 percentage points among these 50 answers, and that the lead is strong for a sample this size.

You then ask a second group: "Which ad makes it clearer what is being offered?" The answer could differ. You may find the coin jar attracts attention and the phone screen explains the product. That points to combining them.

The numbers in this example are made up to show how to read a result. They are not from a real study.

What a quick test does not tell you

• Whether people will click or buy. Stating a choice is not behavior.
• How the ad performs in the feed. Real ads sit among other content, at different times of day.
• How the platform will deliver it. Ad platforms choose who sees what.
• Whether a different group would choose the same. The result shows what this group chose.
• Anything about brands that are not yours. Test your own work, or fictional examples.

Next step

Prepare your two versions and see the price first. See pricing and try the estimator. It charges nothing. For the method behind this, read what is preference testing.

This is a first filter. It removes weak headlines cheaply. It does not replace a live test.

Why test before paying for traffic

A live A/B test needs visitors, and visitors cost money. Every visitor who sees a weak headline is wasted spend.

A quick preference test with a small group lets you drop the obvious losers first. See A/B testing vs preference testing for how the two methods fit together.

Two questions, not one

The first is about comprehension. The second is about interest. A clever headline can interest people and still confuse them. A plain one can be clear and dull.

Ask each as its own question. Each goes to its own group of heads, so don't treat the two results as answers from the same people.

How to run the test

1. Write two to four headlines. Change one thing between them, such as the benefit or the wording, so you know what drove the result.
2. Write one neutral question per criterion. For example, "Which headline is clearer?" See how to write a question that does not lead people.
3. Mind the length. Options on 50heads are up to 40 characters. For a longer headline, put it in a picture of your page's top section and use an A or B images question. Images add $0.10 an answer each.
4. Don't add a slogan or brand name to one option only. Keep the presentation equal.
5. Choose who answers. Target the people the page is for. Each trait adds $0.05 an answer.
6. Ask fifty to start. Fifty Tier 1 answers to a short question cost $13 before any images and typically take about ten minutes. Tier 1 means each person has a verified phone.

A worked example

You have two headlines for a savings app:

• A: "Save without thinking about it"
• B: "Automatic savings from your spare change"

Ask one group: "Which headline is clearer?" Ask another: "Which headline makes you want to know more?"

Say B wins clarity, 36 to 14, and A wins interest, 29 to 21. The first result says B led by 44 percentage points among these 50 answers, which is a strong lead for a sample this size. The second is a narrower lead, so treat it as close.

You might then write a third headline that keeps B's clarity and A's pull, and run both questions again.

The numbers in this example are made up to show how to read a result. They are not from a real study.

What a headline test does not tell you

• Whether people will click or buy. Stating a preference is not behavior.
• How it works next to your page. Visitors see a headline with an image, a price and a button.
• Whether the traffic you buy will respond the same way. The group may differ from your visitors.
• Why a headline won. Ask 20 Tier 2 heads, people who have also passed an ID check, for a short written reason.

Next step

Write your headlines and see the price first. See pricing and try the estimator. It charges nothing. For the method behind this, read what is preference testing.

Your team knows the brief and has seen every draft. People outside don't, which is why they are worth asking.

Why not ask your team

Your team has three biases that outside people don't share:

• they know what the logo is meant to say;
• they have watched the drafts change and are attached to some;
• they judge it against the brief, not against the other logos a customer sees.

An outside group sees the logo cold, the way a customer will.

What to ask about

Pick one quality. Each is a different question:

"Which do you prefer?" is the weakest. It is fine as a first look. It is a poor basis for a decision.

How to run the test

1. Prepare two logos as images. Use the same size, background and framing for both. A difference in presentation will decide the result for you.
2. Write one neutral question for one quality. See how to write a question that does not lead people.
3. Use an A or B images question. Heads, the people who answer questions on 50heads, tap one of two pictures. Images add $0.10 an answer each, and the estimator shows the total before you ask.
4. Add "Neither" if both could fail. It shows in gray in the results.
5. Choose who answers. If the logo is for a particular group, target that group. Each trait adds $0.05 an answer.
6. Ask fifty to start. Fifty Tier 1 answers to a short question cost $13 before images and typically take about ten minutes. Tier 1 means each person has a verified phone.

If you have more than two logos

A or B takes exactly two images. For three or four designs, run the pairs and keep the winners. Compare two, drop the clear loser, then bring in the next design against the winner.

Don't test six at once and read the order as a ranking. Each extra option spreads the answers thinner.

A worked example

You have two logos for a savings app, one a rounded wordmark and one a compass symbol. You ask fifty Tier 1 heads: "Which logo looks more trustworthy for a savings app?"

Say the result is the wordmark 34, the compass 16. The result line says the wordmark led by 36 percentage points among these 50 answers and how strong that lead is for a sample this size.

That tells you which one people picked when asked about trust. It does not tell you the wordmark is the better design.

The numbers in this example are made up to show how to read a result. They are not from a real study.

What a logo test does not tell you

• Preference is not design quality. People may favor the familiar over the distinctive.
• A logo is seen in context. It sits on a product, a website, a shop window. Test it there too.
• Recognition takes time. A cold first look says nothing about whether a logo becomes memorable.
• A different group could choose differently. The result shows what this group chose.
• Don't test other people's brands. Use your own designs, or fictional ones for practice.

Finding out why

Ask 20 Tier 2 heads, who have also passed an ID check, to write a short reason (up to 200 characters). It often explains a split that the numbers alone don't.

Next step

Prepare your two images and see the price first. See pricing and try the estimator. It charges nothing. For the method, read what is preference testing.

The answer changes with the question. That is the point of testing.

Start with the job the name has to do

A name does different jobs in different places. Decide which one you are testing:

• Trust. Would you hand over money or data to a company with this name?
• Fit. Does the name suit what the product does?
• Memorability. Would you remember it tomorrow?
• Clarity. Can you guess what it is from the name?
• First impression. What do you think of it, with no other context?

Test one at a time. A name can win on trust and lose on memorability.

How the question changes the result

Suppose you have three names for a savings app: Northway, Keel and Brightpot.

Each can give a different leader. "Which do you like?" is the weakest, because it doesn't match a decision. People can like a name they would never trust.

How to run the test

1. Shortlist two to eight names. Fewer is easier to read.
2. Pick the criterion from the list above.
3. Write one neutral question. For example, "Which name would you trust with your savings?" Don't add praise such as "great" or "new". See how to write a question that does not lead people.
4. Keep each name short. Options on 50heads can be up to 40 characters. Show the name alone, not a slogan.
5. Add "Neither" if it is a real possibility. It appears last and shows in gray in the results.
6. Choose who answers. If the name is for a particular group, such as parents or a single country, target them. Each targeting trait adds $0.05 an answer. Countries cost nothing.
7. Choose how many. Fifty is a reasonable start. See how many people you need.

A worked example

You ask fifty Tier 1 heads: "Which name would you trust with your savings?" Heads are the people who answer questions on 50heads, and Tier 1 means each has a verified phone. The options are Northway, Keel, Brightpot and Neither. Fifty Tier 1 answers to a short question cost $13 and typically take about ten minutes.

Say the result is Northway 24, Keel 12, Brightpot 9, Neither 5. The result line says Northway led Keel by 24 percentage points among these 50 answers, then says how strong that lead is for a sample this size.

You then run a second question on the same shortlist: "Which name would you remember?" That goes to a new group of heads, so you can compare the two results but not the same people.

The numbers in this example are made up to show how to read a result. They are not from a real study.

Finding out why

A split tells you what, not why. If one name wins and you don't know what drove it, ask 20 Tier 2 heads, people who have also passed an ID check, for a short written reason. Free text is up to 200 characters.

What a name test does not tell you

• Whether the name is available. Check trademarks and domains separately.
• Whether the name works in other languages. Ask in each language, as every question is asked in one.
• Whether it will sell. Choosing a name when asked is not buying.
• Whether it works with your logo and product. A name is seen alongside other things.
• Whether a different group would agree. The result shows what this group chose.

Next step

Write your shortlist as one question and see the price first. See pricing and try the estimator. It charges nothing. For the method behind this, read what is preference testing.

What sample size changes

The number of answers changes how much a split can wobble by chance. With few answers, a small lead can happen by accident. With more, the same lead is harder to explain that way.

Sample size does not change who you asked, how you worded the question or whether the options were fair.

The same split at three sizes

50heads describes how strongly the answers favor the leader as High, Medium or Low. We compare the gap between the top two options with how much that gap could vary by chance for that number of answers. A gap about two and a half times that size or more is High. About two times is Medium. Anything smaller is Low. These are our own thresholds, set out in what confidence means.

Here is how that rule reads for some two-option splits. The numbers are illustrative, not from real studies.

Look at the 60 to 40 split. With 50 answers it is Low. With 100 it is Medium. The split hasn't changed. Our certainty about it has.

What to do at each size

• 20–30 answers. Good for a quick gut check, or to find a clear loser. Only a large lead means much.
• 50 answers. A reasonable start for two to four options. Typically about ten minutes.
• 100–200 answers. Better when options are close, when a wrong choice would cost you, or when you want to compare groups.

On 50heads you can ask up to 5,000 on the web. The estimate shows the cost and time together before you ask. For a short question, 50 Tier 1 answers cost $13.

What more answers will not fix

• A leading question. Strong answers to a biased question are still biased. See how to write a question that does not lead people.
• The wrong group. Fifty people who are not your customers tell you about fifty people who are not your customers.
• Population claims. A quick preference test shows what this group chose. It does not tell you what share of a whole country or market would choose the same.
• Behavior. What people choose when asked is not the same as what they buy. See A/B testing vs preference testing.

If the lead is low

You have four options:

• ask more people;
• drop the weakest option, so the answers spread less thinly;
• ask 20 Tier 2 heads, people who have passed an ID check, for a short reason;
• accept the tie. Sometimes the honest answer is that people don't mind.

Pay for the answers, not the certainty

You pay only for answers that are accepted. If fewer heads answer than you asked for, the rest goes back to your balance.

A bigger sample costs more and takes longer. Match it to the cost of being wrong. If a wrong name means a rebrand, ask 100. If it means a different caption tomorrow, 30 will do.

Next step

Choose a number, then see the price and time together. See pricing and use the estimator. It charges nothing. For the method, read what is preference testing.

Why it matters

A leading question can produce a strong result for the wrong thing. The answers may be clear and still tell you nothing, because the wording did the choosing.

A strong lead means the answers favoured one option. It doesn't mean your question was fair.

Five ways a question leads

1. Praise in the wording

Before: Which of these great new names do you prefer?

After: Which of these names would you trust with your savings?

"Great new" tells people what to think. Describe the choice instead.

2. A hidden assumption

Before: How much do you enjoy our fast delivery?

After: Which delivery time would you choose? Same day · Next day · Within 3 days

The first question assumes people enjoy it and that it is fast. The second asks for a choice.

3. Two ideas in one question

Before: Which logo is more modern and trustworthy?

After: Which logo looks more trustworthy?

Two ideas give you an answer to neither. If you care about both, ask two questions.

4. Options that are not equal

Before: Which would you pick? A short, clear headline · A long headline full of jargon about our platform capabilities

After: Which headline is clearer? Headline A · Headline B

If one option is obviously weaker, the result tells you nothing. Make each option a serious candidate, written in the same style.

5. Missing options

Before: Which would you buy? Plan A · Plan B

After: Which would you buy? Plan A · Plan B · Neither

If "neither" is a real possibility, include it. Otherwise people who like neither are forced to pick one.

A checklist before you ask

• One idea. Could someone agree with half of it?
• Plain words. No jargon, no abbreviations, no internal names.
• No praise. Remove "great", "new", "leading", "powerful".
• Equal options. Same length, same style, same effort to read.
• Behavior, not opinion about others. "Which would you order?" beats "Which is more popular?"
• Short. On 50heads, up to 150 characters to read costs the base price. Longer questions cost more and tend to get less clear answers.

Check it on a phone

Heads, the people who answer questions on 50heads, answer on a phone in about five seconds. The composer shows your question exactly as they will see it. If it doesn't fit on the screen without scrolling, cut it.

What a neutral question still can't do

A fair question gives you a fair read of what this group chose. It doesn't tell you why. It doesn't show that the winner is best. And a different group might choose differently.

If you need the reasons, ask a follow-up. On 50heads, you can ask 20 Tier 2 heads, people who have passed an ID check, for a short written reason.

Next step

Take the question you were about to ask and run it through the checklist. Then see pricing and check the estimate. It charges nothing. The full rules are in writing questions that get clear answers. For the method behind it, read what is preference testing.

They answer different questions, so they work well in sequence.

The difference in one line each

• Preference testing asks a group to choose between options before launch. You see what people say they would pick.
• A/B testing shows different versions to different live visitors. You see what they actually do, such as click or buy.

Side by side

When to use preference testing

• You have more options than you can put live.
• You don't have the traffic for an A/B test.
• You haven't built anything yet.
• You want a direction by tomorrow.

When to use A/B testing

• The page or product is live and has steady traffic.
• The decision depends on behavior, such as sign-ups or purchases.
• You can afford to show some visitors a version that may do worse.

Use both, in order

1. Start with eight options. Names, headlines or images.
2. Run a preference test. Ask one question. Keep the top two or three.
3. Build the finalists.
4. Run an A/B test. See which one visitors respond to.

The first step is cheap and fast, which keeps the second step short.

What neither method tells you

• A preference result is not a forecast of sales.
• An A/B result tells you what happened with that traffic, in that period.
• Neither explains why. Ask people for a short reason if you need it.

A small example

You have six headline ideas for a landing page. You ask fifty Tier 1 heads one question: "Which headline is clearest?" That costs $13 and typically takes about ten minutes. Tier 1 means each person has a verified phone.

Say two headlines lead and the rest trail. You build those two and run an A/B test with live visitors. Preference testing removed four options before you spent any traffic on them. The same approach works for headlines, names and images.

The numbers here are illustrative, not from a real study.

Next step

If you are choosing between options before launch, write them as one question. See pricing and try the estimator, which charges nothing. Our guide, what is preference testing, covers the steps in more detail.

Be clear about which of those you are testing. They are different questions with different methods.

Four questions people mix up

A quick test covers the first row. It is fast and cheap, which makes it good for narrowing choices early. It cannot cover the rest.

Choose the decision first

Don't start with "Is this a good idea?". Nobody can answer that in five seconds. Start with the decision you need to make this week:

• Which of three concepts do we prototype first?
• Which problem should the first version solve?
• Which description makes the product clearest?

If a result wouldn't change what you do, don't ask.

How to run the test

1. Write each concept in a line or a picture. Keep every concept at the same level of detail and in the same style. A polished mock-up will beat a rough sentence, whatever the idea behind it.
2. Write one neutral question. For example, "Which of these would you be most likely to use?" Describe, don't sell. See how to write a question that does not lead people.
3. Pick the format. Options on 50heads can be up to 40 characters, so a short concept fits as text in a Pick one question. For anything longer, use a picture of each concept with an A or B images question, which takes exactly two. Images add $0.10 an answer each.
4. Add "Neither" so people who want none of them can say so. It shows in gray in the results.
5. Choose who answers. Target people like your customer, not everyone. Each trait adds $0.05 an answer, and countries cost nothing.
6. Ask fifty to start. Fifty Tier 1 answers to a short question cost $13 and typically take about ten minutes. Tier 1 means each person has a verified phone. See how many people you need.
7. Keep your questions together. In Projects, you can group questions by what you are trying to find out, so each round sits next to the last.

A worked example

You are choosing which of three ideas for a savings app to prototype first:

• Round up spare change
• Save towards a goal
• Split bills with housemates

You ask fifty Tier 1 heads, the people who answer questions on 50heads: "Which of these would you be most likely to use?" Say the result is 29 for the spare-change idea, 12 for goals, 6 for bill-splitting and 3 for Neither. The result line says the spare-change idea led by 34 percentage points among these 50 answers and says how strong that lead is for a sample this size.

That tells you which idea this group picked. It does not tell you they would pay for it. For that, your next step is a different test, such as a price question or a landing page.

The numbers in this example are made up to show how to read a result. They are not from a real study.

Finding out why

Ask 20 Tier 2 heads, people who have also passed an ID check, for a short written reason (up to 200 characters). It often shows what drove a split.

What a quick test does not tell you

• Willingness to pay. Picking an idea is free. Paying is not.
• Whether the problem is real. Speak to customers for that.
• Whether you can build it well. A concept is easier to like than a product.
• Whether a different group would pick the same. The result shows what this group chose.
• Whether it will fit the market. Only launching shows that.

Next step

Write your concepts and see the price first. See pricing and try the estimator. It charges nothing. For the method behind this, read what is preference testing.

It works best before launch, when you have a handful of candidates and a choice to make.

When preference testing is useful

Use it when you are comparing two to eight options and one choice is the decision. Typical cases:

• product, brand or feature names;
• logos and packaging;
• headlines and lines of copy;
• thumbnails and ad images.

The reader of the result is usually a product, design, brand or marketing team. They want a direction fast, without building anything first.

When it is not the right method

Preference testing is a poor fit when:

• you need to know whether people will pay. Choosing is not buying;
• you need the reasons behind a choice. Add a written follow-up;
• you need to see how people use something. That is usability testing;
• you need live behavior from real traffic. That is an A/B test. See A/B testing vs preference testing.

How to run a small preference test

1. Name the decision. "We are choosing a name for the app by Friday."
2. Pick the criterion. "Which would you trust with your savings?" is a different question from "Which do you like?". Choose the one that matches your decision.
3. Limit the options. Two to eight. Fewer is easier to read.
4. Write one neutral question. One idea, plain words, no hints. See how to write a question that does not lead people.
5. Choose who answers. Everyone, or a group such as an age band or country.
6. Choose how many. Fifty is a reasonable start for two to four options. See how many people you need.
7. Read the result with its limits in mind.

A worked example

You are choosing between two names for a savings app: Northway and Keel.

The question: "Which name would you trust with your savings?"

The options: Northway, Keel, and Neither.

You ask fifty Tier 1 heads. Heads are the people who answer questions on 50heads. Tier 1 means each has a verified phone. That costs $13 and typically takes about ten minutes.

Say the result is Northway 31, Keel 14, Neither 5. The result says Northway led by 34 percentage points among these 50 answers, then says how strong that lead is for a sample this size.

The numbers in this example are made up to show how to read a result. They are not from a real study.

How to describe the result

Say what happened. Don't say what it proves.

Confidence on 50heads means how strongly the answers favor the leader. It is not certainty that the leader is objectively best. See what confidence means.

What the result does not mean

• It shows what this group chose, in this question, on this day.
• A different group, a different question or a different set of options could give a different split.
• A leading question can produce a strong result for the wrong thing.
• Preference is not quality. People may favor what is familiar over what works best.

What to do next

If the lead is clear, move on with that option. If it is close, you have three choices:

• ask more people;
• remove the weakest option and ask again;
• ask 20 Tier 2 heads, people who have passed an ID check, for a short written reason.

Next step

Write your options as one question and see the price first. See pricing and try the estimator. It charges nothing. To see how a question becomes a result, read how it works.