Asking Reasonable Questions

By Ron Sellers
Originally published in The NonProfit Times, January 15, 2000

The concept of marketing research is simple, really: you ask questions, and people answer them. But this simplicity often functions as a trap, because it can be much easier to ask a question than to answer it.

In research surveys, it is crucial for your organization to have a realistic view of what questions can and cannot be reasonably answered by a survey respondent. This is important to keep in mind whether you are designing the questionnaire yourself, or are helping provide the objectives for a questionnaire being designed by a research professional.

For instance, could you reasonably answer this question: in the past three years, how many times have you made a gift to a non-profit organization? (If you can answer it, you either have an amazing recall of your activities or you’re not giving to non-profits often enough!) Many people can barely remember what they did in the past week or month, never mind the past three years. Yet that is exactly the type of question which is asked all too frequently in research surveys.

An immutable law of marketing research is that just because you need certain information does not necessarily mean that people will be willing or able to provide it to you.

Abuses and mistakes in this area tend to center around a few key problems. The first one is simply expecting too much information from people. This problem usually manifests itself in questionnaires that are just too long or too complex. In most cases, it just isn’t reasonable to expect people to spend more than about twenty minutes answering your questions (unless they are getting compensated for their time).

Once respondents get through the first two or three questions of a telephone survey, about 95% of them will complete the questionnaire, as long as it runs no longer than around 15 to 18 minutes. After about 18 minutes, two things happen. First, some people become impatient, and simply terminate the interview (often by hanging up on the interviewer). Many of the remaining respondents will suffer from “respondent fatigue” – no longer listening carefully to questions or giving thoughtful answers. Simply put, the accuracy of the data you receive after this point drops sharply.

One mail survey recently received by a frequent business traveler contained eight legal-sized pages jammed with complex questions, including one section in which respondents were asked to rate nearly 40 different business hotels on around 12 different attributes (480 separate ratings). This was sent to busy travelers, and included a one-dollar incentive (which was likely more of an insult than an incentive for a questionnaire this long). Expecting people to provide this amount of information just isn’t realistic, no matter how badly you want the data.

Another common problem is using unrealistic time frames. This is particularly true when querying people about relatively mundane activities. If you asked people how many times they have purchased a car in the past two years, most could answer the question pretty accurately. If you asked them how many times they’ve washed their car in the past two years, it’s another story.

Keep in mind that the greater the time span for which you’re asking people to remember their activities, the lower the likelihood you’ll get accurate information. Also bear in mind that the time frame needs to match the activity.

For instance, religious organizations often ask questions about frequency of church attendance. For many people who attend church, it’s a weekly activity. Asking how many times they’ve attended church in the last week is akin to asking a yes or no question about whether they attended. Another problem is that people who attend church two or three times per month will tend to claim they attended last week, even if they didn’t – simply because they usually attend church every week. They’ll answer a question like that without even really thinking about it (they usually attend, so in their minds they probably attended last week).

On the other hand, asking them how many times they’ve attended in the last six months will get broad estimates and guesses that probably won’t reflect reality, because the time frame is just too long.

One final comment on time frames: people tend to be rather poor at estimating the amount of time they spend on activities. Surveys of television viewing consistently show people think the TV is on far less than it actually is. This problem will be exacerbated if respondents are asked to provide such estimates over longer periods of time, such as the number of hours they watch per month.

Another time-related problem in research often results from poor planning by the organization. Research has to be timely in order to gather accurate perceptions. It’s not realistic to send someone a newsletter, then contact them three months later to ask their opinions about it. If you’re planning to gather information about an event you sponsored, ask people for their opinions as soon after the event as possible. The more time elapses, the more respondents will either forget important details, or allow the passing of time to color their perceptions of the event.

For example, donors might leave your July event feeling very positive about your organization. If in August you suffer through some bad publicity, people will subconsciously allow that publicity to impact their feelings about the event they attended. When you interview them in September, what you’ll get is feedback on how they feel about you right then, rather than how they really felt about your July event. This same principle applies to lapsed donor studies, product purchases, advertising or direct marketing campaigns, etc. Make sure to plan for any research you need well ahead of time in order to avoid this problem.

Another difficulty that has lead many companies and organizations into trouble is that people are notoriously poor at predicting their future activities. Some organizations figure they can simply ask people how likely they are to write a contribution check in the next six months, and they’ll receive honest and accurate answers. In asking a question like that, what you actually measure is willingness to give, or interest in giving. Don’t depend on questions like that to be accurate predictors of future action.

It’s also a mistake to figure that all questions can just be asked directly. For instance, most consumers will insist that advertising does not motivate them to buy things. The reality, of course, is quite different. Unfortunately, too many researchers ask direct questions like this when respondents simply can’t give direct answers – often because they really don’t know the true answers.

Some clients have insisted that their focus groups include a bevy of questions like this: “If the print on this book cover were in yellow rather than in green, would that make you more likely to buy it?” Few customers mentally tie the colors used in a book’s cover art to an actual purchase decision. Even though an actual connection between them might exist (such as making the cover more noticeable, or more appealing), it’s not a connection which can be discovered with a sledgehammer, but with tweezers and a microscope. You can often arrive at the information you need, but in more subtle, roundabout ways, rather than through direct questioning.

Finally, it’s important to understand that you live and breathe your work every day, while to the typical consumer it’s just a tiny part of their lives. What you see as meaningful, the average person very well may not. This means that many of the things you care about deeply – your organization’s direct mail pieces, the quarterly newsletter, the annual report, etc. – consumers might not even remember. It’s not uncommon to have donors not remember giving to your organization, or not recall receiving the newsletter.

This is important to remember in survey design. For instance, consider a very small telephone study of, say, 200 donors, with the purpose being to measure attitudes about your donor magazine. If every donor who was sent the magazine is qualified to undertake the survey, and just half remember the publication, what you end up with is a survey with usable responses from just 100 people. But if the survey is designed to exclude those who don’t recall the publication from the sample, you’ll have usable responses from 200 people, plus you’ll know the overall recall for the publication.

To help alleviate some of the problems discussed here, remember two critical things. First, a good questionnaire must anticipate every possible answer – including the many people who might not remember or have an opinion about what you feel is an important topic. Second, when questions are written, try to answer them as a respondent might. If you can’t honestly recall how many times you’ve done something in the last six months, the chances are pretty good that your respondents won’t, either. If you tire while answering a litany of questions that all sound the same, respondents probably will, too. Just like you have to think like donors in order to market to them, you also have to think like donors in order to research them.