Survey Playbook

A survey is worth sending only when you know the decision it will settle. Write that decision down first, ask only the questions it needs, invite the people whose answers count, report the result with its margin of error, and then tell respondents what you decided.

Most teams write the questions first. Surveys then fail for four predictable reasons: no clear decision behind them, biased or confusing questions, the wrong people answering, and results that never get acted on. The eight steps below deal with each, from the decision sentence to the email that closes the loop.

A survey measures how common something is; it is a poor tool for finding out what to ask. If you don’t yet know the words your customers use, interview first, as the Market Research Playbook sets out. If the question is why deals were won or lost, the Win-Loss Analysis Playbook is the better method.

Before writing a single question, finish this sentence: “After this survey, I will decide ____ based on ____.” If you can’t name the decision, don’t run the survey yet.

Then write down three things: the one decision the survey informs, the one primary metric that decides it (a share choosing an option, a ranking, NPS or CSAT), and two to four secondary questions that explain that number. Everything else is curiosity; cut it, along with anything your product or CRM data already tells you.

Define the audience precisely, for example “restaurants that have used inventory counts in the last 30 days”, not “customers”. Two hundred answers from the right people beat 2,000 from the wrong ones. Mixing groups with different needs, such as prospects and power users, produces an average that describes neither: split them, or plan to report them separately.

If your list is a few hundred people, invite all of them rather than a sample. Then estimate how many answers you will get from your own past response rate, and what that number buys. The margin of sampling error, the ± that comes with a percentage, shrinks quickly up to a few hundred answers and slowly after that (Figure 2). AAPOR, the American Association for Public Opinion Research, puts it at ±3 points for 1,000 answers and notes that doubling to 2,000 gains only about a point. Each segment you compare has its own, larger margin, set by the segment’s own count^[1].

Two limits matter. The margin covers only the luck of who happened to be sampled, not who chose not to answer or how a question was worded, and AAPOR says it does not apply at all to opt-in samples such as a link posted publicly. For a first, directional read, Superhuman’s founder found that product-market fit survey results start to be “directionally correct” at around 40 respondents^[2]. Below that, report counts, not percentages.

Small wording choices change answers, as the Market Research Playbook shows with one added clause. Check every question against seven rules:

1. One idea per question. Split “How fast and helpful was support?” into two.
2. Stay neutral. “How much did you love the new dashboard?” presumes love. Ask “How would you rate the new dashboard?”
3. Answers that fit the question. Offer balanced options, as many negative as positive, and name what is being rated instead of asking people to agree or disagree. Agree-or-disagree items invite agreement whatever the content; in a study comparing the two formats, item-specific options gave much better-quality answers^[3].
4. Options that don’t overlap and cover everything. No ranges like 0–5 and 5–10, and always an out such as “None of these” or “Other”.
5. Order matters. Easy and general first, specific and sensitive last. Pew Research Center’s guide shows how a specific question asked first changes the answers to a general one that follows^[4].
6. Match the type to the question. Single choice for one answer, checkboxes for several that apply, a rating for intensity, short text for a fact, long text for the why.
7. Keep it short. In a web-survey experiment, the longer the stated length, the fewer people started and finished, and answers to later questions were faster, shorter and more uniform^[5]. Keep open text to one question, after the closed ones it explains, in a survey short enough that it isn’t buried.

The primary metric decides the scale. If the decision is a choice between options, ask a single-choice question with an out. If you are tracking loyalty, satisfaction, effort or product-market fit over time, use the standard question for it, unchanged, so each round compares like with like. Figure 4 shows the four standard questions, their scales and how each is read.

Use the standard scale exactly. The Net Promoter Score is asked from 0 to 10, with promoters at 9 and 10 and detractors at 0 to 6^[7]. Survey Studio’s rating questions start at 1, so for NPS use a single-choice question with the eleven options 0 to 10; a 1 to 10 rating drops the zero and is not NPS.

These six starter surveys cover the decisions founders run most often. Copy one into Survey Studio and trim it to your decision sentence.

The invitation decides who answers, and the evidence on what works is unusually good. A Cochrane review pooled randomised trials of ways to raise response to electronic questionnaires. A brief invitation rather than a detailed one roughly tripled the odds of a response; personalising it, giving a deadline and offering to share the results each helped; and putting the word “survey” in the subject line lowered response^[8]. Say how long it takes, and make that true.

Then say what happens to the answers. Anonymous means nobody can link an answer to a person. Under the EU’s GDPR, truly anonymous data falls outside the regulation, but answers tied to an email address or a personal survey link, even under a code, are still personal data (Recital 26). If you collect personal data from people in the EU, Article 13 requires you to tell them, when you collect it, who you are, why you are collecting it and on what legal basis, who receives it, how long you keep it and what rights they have; Article 5 limits you to what the purpose needs^[9]. Promise anonymity only when the answers really can’t be traced; candid questions about churn or your brand often deserve it.

In the US, the CAN-SPAM Act covers email whose primary purpose is to advertise or promote, and it makes no exception for business-to-business email. A plain request for feedback is not that, but add an upgrade offer and the email may become commercial, which brings a working opt-out and your postal address among other rules^[10]. I keep survey emails free of anything to sell.

Match the channel to the people and the moment. Survey Studio gives you a link for email, SMS or social posts and an embed snippet for a web page.

• Email to people who know you. Customers and users answering in context: usually your best-quality reads.
• In the product, or right after an interaction. Embed the survey on a thank-you page, in the app or in the help center while the experience is fresh.
• SMS. Only straight after a transaction, and only for a single question.
• A cold or bought list. Expect a small fraction to answer, skewed toward the already opinionated.
• A public link on social media. Reach, not a sample: the answers are self-selected, so there is no margin of error to report. Treat them as anecdotes.

I publish no response-rate benchmark here. Rates vary too much by audience, relationship and channel, and your own past sends to the same list are the only estimate worth planning on.

Send one reminder, only to people who haven’t answered. In a web-survey experiment, extra reminders raised the response rate, though they did not make the respondents more varied^[11]. Incentives work too: in the Cochrane trials of electronic questionnaires, money nearly doubled the odds of a response, and non-monetary incentives and an offer of the results also helped^[8]. Offer the same thing to everyone. In my view, avoid a discount on your own product as the reward, because it draws the people who already want more of it.

Survey Studio designs and previews the questions and works out how many responses you need; collect the answers in a survey tool that does. Read them in this order:

1. Who answered. Report the response rate as AAPOR defines it: complete responses divided by everyone eligible you invited, including those who never opened the email. Its strictest version, RR1, counts only completes; RR2 adds partial responses^[12]. Then check where people dropped out, which often flags a bad question.
2. Whether they look like the list. Compare respondents with everyone invited on what you already know, such as size, plan or tenure. A low response rate doesn’t prove bias on its own: a meta-analysis of 59 studies found very little correlation between nonresponse rates and the bias they measured^[13]. What matters is whether the people who didn’t answer differ on the thing you are measuring.
3. The primary metric, with its margin. Compare it with the line you set in Step 1, not with your hopes. The margin on the gap between two answers from the same group is about twice the margin on each^[14].
4. Segments. Split by the groups you named before sending. Averages hide the story, but a split is only a finding if the gap is bigger than its margin.
5. Open text, coded into themes. Read every answer, tag recurring themes and count them, the way you would code interview notes with the Customer Interview Kit.

• Make the decision you defined in Step 1. If the data is ambiguous, that is a finding too: often the question was wrong, or the segments differ. Re-cut the data or ask a different way; don’t re-run the same survey hoping for a better answer.
• Share a one-screen summary with the team: the decision, the number with its margin, the top theme and the next step.
• Close the loop with respondents. “You said X; we’re doing Y.” Offering the results is one of the incentives that raised response in the Cochrane trials (Step 5), and keeping that promise is what makes the next invitation believable.

What changes by stage?

What mistakes should I avoid?

• No decision behind the survey. Write the decision sentence before any question (Step 1).
• Questions that measure your own hopes. Leading, double-barreled and agree-or-disagree wording all do it (Step 3).
• A changed scale. A 1 to 10 “NPS” or a reworded CSAT question can’t be compared with last round (Step 4).
• A public link read as a sample. Self-selected answers have no margin of error (Step 6).
• Splits read without margins. A gap smaller than its margin is noise, however good the story (Step 7).
• No reply to respondents. Results that change nothing teach people to ignore the next invitation (Step 8).