Respondent Abuse – A Plague to the Profession
By Ron Sellers
Originally published in Alert! (Marketing Research Association), March 2005.
I received a research questionnaire today in the mail. It’s twelve pages long. Twelve legal size pages.
It has 91 questions. However, that’s not really accurate. Question 3, for instance, actually has 17 statements with which I need to agree or disagree. Taken all together, there are 771 individual items for me to read and answer.
For this, the research firm enclosed a dollar. That’s about eight questions for every penny of incentive money.
The cover letter assured me that “Although this survey seems lengthy,” I may not need to complete each section. This is true – if my household has no modern technology, investments, or vehicle, and I never use the internet, I get to skip about four pages. If my household owns none of these things but I use the internet even occasionally, I get to answer about 11 of the 12 pages. I counted it up – personally I would have gotten to skip all of 22 out of the 771 questions.
It would be easy to shake my head, throw the mailing in the recycle bin, and ignore it, except for three things:
- this was sent out by a nationally known research firm
- some company or set of companies undoubtedly is paying a wad of cash for the findings,
and will be making crucial decisions based on the resulting data - I get questionnaires like this all the time
This is an example of the growing problem of respondent abuse. It’s not just a matter of annoying a few people in order to gather important information, but a systematic pattern of taking advantage of potential study respondents. Respondent abuse has a number of critical repercussions for individual projects, as well as for the industry as a whole.
For one thing, every time a respondent is annoyed or angered during the research process, that makes it less likely he or she will participate in future research projects. This means response rates trend downward for all research efforts.
It also means our industry reputation declines. Why do so many people hate spam e-mails? Because they are such an annoyance. Why has telemarketing been hit with significant restrictions such as the Do Not Call legislation? Because it was such an annoyance. If research makes itself an annoyance rather than an interesting experience of sharing opinions, it won’t be long before our industry will face restrictions.
Another problem is that the quality of the research gathered during a project that abuses respondents is highly dubious, because of both the quality of the respondents and the quality of the responses.
Overly long and unnecessarily complex questionnaires lead to much lower cooperation rates, which can bias the sample. If the only people who take half an hour to complete your lengthy questionnaire are people who have absolutely nothing better to do with those 30 minutes, you’re probably not reaching a lot of businesspeople, busy parents, active adults, and others who actually have a life. You simply don’t get many high quality respondents.
K.C. Scott, Senior Project Director of the 1,500-line field center Western Wats Center in Provo, Utah, says on phone surveys, “If it’s longer than 15 minutes, you’re going to see a drastic decrease in your cooperation rate if you don’t have an incentive.”
Even the respondents who will stick with it and answer all of those questions are likely to take a lot less time and effort toward the end of the questionnaire than at the beginning, as they begin to tire and the annoyance factor rises. Chad Kuepker is Director of Information Unlimited in Dallas, a research field center with 100 phone lines. He notes that toward the end of long phone surveys, “People start putting a lot less thought into each answer. It’s more of an ‘I’m ready to go’ kind of response. If you go over 18 minutes, I think it’s ridiculous.”
Both problems – poor quality respondents and responses – lead to research that is non-representative and misleading. Decisions based on bad research can obviously have a disastrous impact on the end users. As Kuepker warns, “The respondent is either annoyed and will not complete the survey, or they just give any answer to get through the questions. Using this data would be dangerous.”
Long questionnaires are far from the only forms of respondent abuse. Contacting the same people again and again, unless they have agreed to it (such as in a panel relationship), is also abusive of their time. Some tracking studies require recontacting the same respondents in multiple waves of the study. After a while, this often becomes an annoyance. “I hate recontacts,” says Kuepker. “We don’t like to take those types of projects, because it gives us as researchers a bad name.”
Even in panel relationships, asking respondents to answer the same questions over and over can become a real grind for them. I’ve received the same questionnaire every single month for two years from one panel company. Frankly, I’m pretty tired of answering it, even if it’s about half a page. Another panel sends me the same battery of advertising image questions every three months – I answered the first three, and now I’m so tired of the same questions that I throw the questionnaire away unopened.
Even within an individual project, there are plenty of ways to abuse respondents. One is by asking questions that are difficult to answer because of vague wording, unclear meaning, or response options that don’t include every possibility.
Phone respondents often ask the interviewer for clarification; the interviewer, of course, is forbidden to interpret the question. After a few answers of “It’s however you interpret the question,” or “I don’t know, I’m just reading what’s written here,” respondents can stop caring about the answers they provide, as the client obviously didn’t care much about the questions they are asking. Online and mail respondents will be forced to mark whatever response they think may be closest to their actual answer, or leave the question blank.
Another way to abuse respondents is to probe them to death. On unaided recall questions, there are people who can name 30 different brands of cars, cola, or computers. How will it really help you to probe until you can capture every one of them? Once someone has given you four or five solid reasons they don’t like your product, do you really need to continue probing for more?
Yet another way is to ask people things they really can’t answer. “The big thing is being asked to rate a product they have no experience with. Respondents really don’t like to do that. They get to where they’re just giving you answers rather than thinking it through,” says Kuepker. He also rues that when he raises the issue to clients, too many wave it off and tell him, “Just ask it anyway and see what they say. Maybe it will tell us something.”
Repetition is also a problem – asking people to rate many different things on the same scale, asking numerous questions that have only slight differences in wording, or asking a long series of similar questions rather than breaking up the monotony of the questionnaire. Kuepker says, “That’s where we see our respondents get the most angry, when questions are repetitious, especially when many of the attributes are very similar. It just seems like you’re asking the same things over and over again.”
Misleading respondents is one of the worst forms of abuse. An example of this is claiming they’ll be able to skip large portions of the lengthy questionnaire, such as the one described at the beginning of this article (I would have been able to skip 2.6% of the questions in the one I received).
Telling them a questionnaire will take 10 minutes when you really know it will take 15 or 20 is another example. Western Wats has declined projects because the client wanted to be less than honest up front about the length. Rather than lie, Scott explains that on longer questionnaires, “If there’s no incentive, we tend to just shut our mouths about the length and hope for the best.”
As researchers, we have to understand that we rely on the good will and participation of respondents not just for the information we need, but for our very livelihoods. We essentially make a contract with them – we get some of their time and opinions, and in return they get a pleasant (or at least painless) experience and the ability to have a say in everything from the naming of a new product to which advertising campaign will be run.
It would be nice to think that we as researchers have total control over this issue. However, often we have internal or external clients who do not see things the same way. There are a few things researchers can do to help mitigate the problems:
Try to educate clients about these issues before projects happen. Particularly for corporate research directors, the research department should be a source of education and intellectual capital within the corporation, rather than functioning just as an order-taker. Send out articles occasionally, or take key clients to lunch to review with them how the research process can be more effective.
Try to position your department or company as a research partner with the client, which allows you to offer suggestions and offer alternatives rather than just acquiescing to every client demand.
When these issues arise, bring them to the client’s attention and explain the ramifications in a way that is meaningful to the client. Your client probably isn’t interested in response rates, but may be interested in the possibility that a long questionnaire will decrease the reliability of the findings. Also, be prepared to offer alternatives and solutions, rather than just bringing up the problem.
Rely on others as experts so that you’re not the only one on the hook. Set up a conference call with the field director on a phone survey, for instance, or have the client monitor the pre-test so he can hear respondents’ objections personally. Send the client articles such as this one or ask for input from trade associations such as the MRA or the QRCA.
Ask the client to answer the questionnaire herself, or to have a colleague who’s not involved in the research process complete it. This may reveal to the client that certain questions are ambiguous or impossible to answer, or that the survey process is frustrating for the respondent.
Respondent abuse is not acceptable, and it is a problem all research clients and vendors face. Unless we take more care to treat respondents well, rather than milking them for every last drop of information no matter how the experience leaves them feeling, we face the likelihood of poor quality data and declining participation rates for individual projects, and for the industry as a whole.
