The Hardest Questions, Part II: Making Surveys Work

In The Hardest Questions, Part I: Why Surveys Usually Fail, I explained how easy it is for even highly-trained researchers to produce awful surveys. It’s often the questions, which can be pointless, biased, confusing, or even impossible to answer, but just as often are surveys that are so long that people stop responding, rush their responses, or start to deliberately mess up their answers to make themselves feel better about the time they’ve wasted on the survey. And far too many surveys simply ask the wrong people, hoping they will answer the same way as the people who should have been asked.

These problems are only for starters because there are many more ways surveys go wrong. I learnt a lot about these problems from my own experience of using questionnaires in research, much of which has been published in high-level scientific journals. I have also taught many researchers and evaluated the work of many more, so I have a lot of experience with how surveys work or fail.

As I explained in Part I, once you have people’s responses it’s usually impossible to tell whether a badly designed survey has produced answers that are right, wrong, or utter nonsense. That means that many surveys are worse than useless because not only will the interpretations be more confused than your respondents were, but you can’t even tell if there is a problem, let alone what it is.

It’s easy to make these simple mistakes, so it’s good to know there are also simple solutions that make surveys far more useful. All of them require effort but without them, you’re probably wasting your time, and worst of all, you’ll never know. So, if you can’t get a professional survey business to run your survey, make the effort. Even if you have hired some professionals, insist that they do these things — if they won’t agree, find someone else.

Clarify Your Goals

Shock! Horror! Someone wrote you need clear goals!

It’s hardly a surprise that clear goals are fundamental to so many activities, and surveys are no exception. However with surveys, people often have goals about how and when they’ll do the survey, when the most important goals are about the information you’ll gain. Much better if your goals answer questions like: What do you want to learn? Who do you want to learn about?

For each goal, take time to consider why it is important and especially, how you’ll use the results of your survey. Simply doing that will make all the other steps far easier.

Invite the Right People (and don’t annoy them)

Regardless of whether you’re doing market research, organisational reviews, or community development, the people you want answering your survey are the people whose decisions affect you.

They make decisions, you get the consequences.

It might be customers who decide where they spend their money or staff deciding how much they’re willing to commit to your business. It’s almost never someone who has already decided to have nothing to do with you. They’re yesterday’s problem and at best, they can only tell you something useful if they’re similar to the people who really matter, the people who have yet to decide.

So, think about those people and where you can find them. To do that, it’s important to start with your goal, the information you truly want. That provides the biggest hint, the strongest idea of who you need to hear from.

Once you’ve identified the right people, it’s vital that you play straight with them. Tell people honestly what your survey is about and why it’s important to you, but also give them some ideas about why it might be important to them. There is more about that in the next section on Meaningful Rewards, but for now it’s worth noting the value of helping people see why it’s in their best interests to participate.

At the same time, you need to avoid overselling the benefits of their participation. An overblown sales-job is guaranteed to annoy people. Another thing that annoys people is when it seems like their answers could be used to manipulate them. In this age when data privacy is increasingly an issue, it helps to let people know if their responses will be confidential and how they’ll be stored, who will see their responses and what they’ll do with them.

And be honest about what will happen to their responses, even if only because if you’re less than honest you’re risking a PR disaster that will cost you more than your survey is worth.

So, don’t annoy people but do give them a reason to tell you what they really think or feel.

Meaningful Rewards

Many businesses try to make surveys more reliable by giving people rewards, like discounts or an entry in a prize-draw. While that may get people involved, it doesn’t make them care about the accuracy of their answers. Instead, people who are focused on rewards often answer more quickly and with less care so they can get their reward faster.

A far better reward is information that people care about and can use, especially information about themselves. My most successful surveys promised people feedback about their personality, but you could offer information about health, finances, work, or community, or merely how their answers compare with other people’s responses. Most people love getting personal information they can use, especially when it is surprising or encouraging. If they care about the feedback, they will care about the quality of their responses, not just about completing the survey.

So whenever possible, you should offer to report back to people the results of your survey, either directly (e.g., by email) or indirectly (e.g., by giving them a website where they can see the results, once they’ve been posted). If it seems like they wouldn’t be interested in the results, there is a good chance they won’t be interested in the questions, which means you probably need to reconsider your strategy. Involving people by sharing the results also helps to build relationships with them that may be helpful in the future, including for the next survey.

Write the Survey

Good questions make it more likely to get good answers, so writing the survey deserves some time.

But not too much.

People often focus so much on writing the survey that they miss the things that follow, the bits that take your survey to possibly OK if you’re lucky to probably going to give you what you want.

Writing the survey is a lot like any other form of writing, which means you should never use your first draft. The following steps are essential if you want to convert the first draft of the survey into something useful.

But you’ll still need a first draft, which should always be based on your goals. For each of your goals, try to write a few questions to match. If you’re not sure about the format for survey questions, have a look at other people’s surveys and questionnaires, or check out the options available with online survey tools.

Whatever you do, stay focused on writing questions that answer your goals, what you need to learn about.

You should also write a brief introduction and guide for the survey, once more based on your goals, any reassurance you should give people, and a description of how the results will benefit them (the stuff from the previous two sections).

Trial the Survey

Once you’ve written the survey, it’s time to stop.

Stop.

Before you go further, you should always try out the survey, something that researchers call running a ‘pilot’ study.

To understand why a pilot study is essential, it helps to think about how language really works. Have you ever noticed how often people change the words they use when they recognise their listeners have misunderstood them? It happens so often and quickly it’s easy to overlook, yet adapting your words to your listener happens constantly and is essential for effective communication.

But you can’t see when someone looks puzzled by your survey, which is one of the main reasons why so many surveys have those really stupid questions. Those questions didn’t seem stupid when they were written — the stupidity only emerged when someone else read them.

This is why you need to run a trial survey, a pilot study. Sometimes, you need to do this more than once, so that you can properly adapt the survey in response to feedback from people in the trial.

When running a pilot study, you should invite people who don’t know the goals of the survey but are similar to the people you’re targeting. They should also be friendly enough to give honest feedback, even to the point of being blunt. It’s useful to get them to comment on anything about the survey, including their personal reactions, things they found confusing or ambiguous, the questions that seemed irrelevant or unimportant, as well as anything they think the survey should have asked.

If you run a good trial and respond properly to the feedback, you’ll avoid most of those horrible questions that would otherwise have people questioning your intelligence.

Don’t make yourself look stupid — trial your survey.

Cut. Cut. Cut.

It’s hard to say what’s the optimal length of a survey but a good rule of thumb is the survey should be shorter than what you want. If you think all the questions you want are included, the survey is probably far too long. If you want a more specific rule, surveys within an organisation should take no more than five minutes to complete, while customer surveys should take a minute or two, at most.

A different way to gauge the right length is by asking people in the pilot study how long they were truly interested in your survey, interested enough that they wouldn’t go back to their emails or phone. Real interest may only last for a question or two, so all credit to my web-host who recently sent me a one-question survey.

It’s also crucial to consider what questions you really want answered. It’s a bit like Marie Kondo’s approach to tidying, in which you get rid of everything that doesn’t clearly bring you joy. With surveys, the principle is that if you’re not sure you need the answer, get rid of the question.

Every additional question makes it less likely for someone to complete your questionnaire and more likely to give nonsense answers. Either way, you’d be wasting their time and yours, so cut, cut, cut.

Conclusion

Of all the tools used for business improvement, few seem easier than surveys and few are more difficult to get right and there are many ways your survey can go wrong. People may tell you the survey is awful, which is the least-worst outcome. Far worse is when people don’t tell you how awful your survey is. Worst of all is when they have no idea that it’s awful. That’s ironic given that the whole point of surveys is uncovering people’s views, especially when they’re unexpected.

The solutions presented here won’t solve all the problems with surveys, but they will greatly reduce them. Most of all, if you run a survey well you learn things you are unlikely to discover in any other way.

And of course, if I can help, just ask.