Mail vs. Phone Studies
By Ron SellersOriginally published in The NonProfit Times, March 15, 2000
Mail or phone?
That’s a question commonly faced by organizations as they design a marketing research project. Should you mail a questionnaire to people, or contact them by telephone? And how do you make that decision?
Even if you completely turn over the project to a professional research firm, the final decision should be made jointly between your organization and the consultant – which means you’re still faced with the question.
There’s no blanket right or wrong answer that fits all organizations. Each method provides its own advantages and drawbacks.
Cost and timing.
Although there are exceptions, the rule of thumb is that mail is cheaper, but phone is quicker. The interviewing for a typical telephone project might take one to three weeks, and if the questionnaire is administered using a CATI system (computer-assisted telephone interviewing, in which responses are directly entered into a computer database while the interview takes place), there is no additional time necessary for data entry. With mail, there’s the graphic design of the questionnaire, printing, mailing, a week or two for people to respond, time for the responses to get mailed back, and then a load of data entry. This can easily take six weeks or more in some projects.
At the same time, mail surveys can be much less expensive than telephone. The difference between the two tends to grow as the number of people being interviewed grows, so it’s particularly noticeable if you’re seeking responses from a large sample of individuals. It’s not advisable to make the methodology decision on budget alone, but it’s the rare non-profit that does not have to use costs as part of the criteria.
Complexity of the questionnaire.
Telephone surveys are generally administered by trained, expert interviewers from a centralized location (assuming you’re not trying to gather three interns and do the interviewing “informally” in-house). Therefore, they can employ very complex questionnaires, particularly when aided by CATI. Every “i” is dotted and every “t” is crossed – and the data is consistent.
With a mail survey, you’re depending on the ability of each respondent to follow directions with care. This usually means quite a few skipped questions, incomplete answers, and even illegible responses.
Open-ended questions (which people answer in their own words, rather than by checking a box or circling a number) are also a challenge with mail surveys. With phone surveys, trained interviewers can probe for more information when a respondent provides an answer which is incomplete or unclear. With a mail survey, what you get is…what you get.
Finally, respondents commonly skim through a mail questionnaire to get a sense of the whole thing before they start with question one. This means you can’t hide anything, or avoid biasing an initial question by revealing important information later in the questionnaire. With a phone survey, the respondent doesn’t know what the next question will be, which allows substantially greater flexibility in questionnaire design.
The quality, complexity, consistency, and depth of information is simply better with a phone interview than with a mail survey.
Response rates and response bias.
A common mistake in research is to evaluate a project’s accuracy solely by the number of respondents in the study. Also important is the response rate – what proportion of the people who you contact actually provide you answers. Phone surveys should provide response rates of at least 40%, and often up to 80%. This means 40% of the people you reach by telephone will participate in the survey. Mail surveys usually bring lower response rates, which means the study has a much higher chance of not truly representing the people it’s supposed to.
Some mail surveys – particularly those which “ride along” with other mailings, such as your newsletter or a fundraising letter – can result in response rates as low as 5 - 10%. When response is this low, it means there is a very high likelihood that you are not truly getting a representative sample; in other words, that the data is misleading, and essentially useless. There is a much greater danger of this with mail than with telephone.
Response bias is a major part of the problem. This means that one type of person is more likely to respond than another type of person. Older people and women, for instance, are consistently more likely to respond to surveys (both mail and phone) than are younger people and men. Good researchers balance for this type of response bias, or else the accuracy of the study is compromised.
In some types of research, however, it’s not possible to balance for this, particularly with a mail survey. This is why it’s rarely a good idea to use a mail survey to test response to a new product concept, for instance.
Let’s say your organization is considering introducing a new service to donors, and you want to test donor response to the concept. In reality, we’ll say 20% of your donors would be excited about the new idea, with the rest unimpressed. When you send the mail survey out, the people who are really excited about the idea are substantially more likely to complete and return the survey than are the apathetic. So half of the people who really like the idea return the questionnaire, versus just two out of ten among those who weren’t impressed.
The resulting data would now show 38% of all the respondents as really excited about the new idea, or almost double the actual proportion of positive feelings among all your donors. This is the trouble that response bias can cause. Properly conducted, a telephone survey eliminates much of this risk.
Prospecting for certain respondents.
Often, research must be done among certain types of people – just those of a certain age, or who have done certain activities, for instance. With a phone survey, the initial questions determine whether the respondent is qualified for the interview. If not, the interview is politely terminated and the amount of wasted time limited. With a mail survey, you’re still paying to print and mail the questionnaire and receive the response even if the respondent isn’t the type of person you want to include. If 80% of the respondents qualify for the project, this probably isn’t a big deal. If only 15% qualify, a mail survey ends up with a lot of waste.
When to use mail.
Other than the cost factor, telephone surveys generally provide substantial advantages over mail: greater accuracy, more data, less risk, increased flexibility, less bias, etc. So should you avoid mail surveys?
Not at all. Mail surveys are a useful tool in numerous circumstances:
- When you have an important population and you’re concerned about perceived “invasion of privacy.” Few people have strong negative reactions to getting a questionnaire in the mail. A phone call can sometimes evoke a more severe reaction. With typical donors, the possibility of upsetting two or three pales in comparison to the value of the information you’re getting. If you’re talking about major donors, however, the risk of upsetting even one or two can mean substantial loss to your organization. Mail surveys will minimize the risk of any strong negative reactions of this type.
- When you need to show people something, it’s tough to do by phone. Some research requires that people respond to something they’re seeing – a color chart, or a series of advertisements, for instance. Even some complex ideas will be hard to understand if heard over the phone, but simplified greatly when read in a mail survey.
- Some populations are terribly difficult or expensive to reach by phone. The cost of reaching businesspeople by phone at their place of work, for instance, dramatically increases the gulf between the costs of mail and phone. Sometimes this factor alone dictates that either the survey be done by mail.
- Some surveys require people to look up and provide data from records they keep. Surveying clergy about church attendance or giving figures, for example, often means they have to consult records to answer accurately. This is also frequently true when interviewing regional or local directors in large organizations. In these cases, it would be very difficult to get accurate answers in a phone survey.
So how is the decision between mail and phone to be made? One method you might want to consider is to ask “Is there any specific reason why this needs to be done by mail, or cannot be done by phone?” (considering the reasons listed above). If so, there’s the answer: it’s a mail survey. If not, the quickness and accuracy of a professional telephone interview is usually preferable to what you get through the mail, and well worth the added expenditure.
