Doing It Right or Not at All
By Ron Sellers
Originally published in The NonProfit Times,
May 1999
Every organization has them: those "quickie" surveys when someone wants information but doesn't want to spend much time or money on the research. A brief questionnaire is inserted in the donor newsletter to find out what articles are being read. People are asked to fill out a one-page questionnaire at the end of a conference. Patrons leaving the museum are handed a quick list of questions and asked to rate their experience.
Rarely are these kinds of projects handled through a marketing research company. Many times, even if an organization has an internal research department, the questionnaire is designed and analyzed by the individual or department that wants the information, rather than by the researchers. You may have done some of these yourself – or may be doing some in the future.
This kind of research is caught in the middle of a conundrum: it's apparently important enough to do the project, but not important enough to commit resources to it. Or, there may simply be no available resources to commit. Either way, if it's important enough to do, it's important enough to do right. Following are some of the most common mistakes made in these quick research projects, and thoughts on how to correct them.
Not having clear objectives.
Even if the research is
three questions on a postcard, there should be clear objectives to what you want to know.
It's not sufficient to say, "we want to know what our donors are
thinking." Thinking about what? In what way?
If you want to know how people rated their experience at your conference, for instance, you need to determine what you must know about their experience in order to judge whether it was a success. Will you consider it a success if people enjoyed themselves? If people said they learned something? If people felt it was a good value for the attendance fee? If people left with a higher opinion of your organization?
Once you've determined your objectives, ask the questions which will address those objectives. The more specific your objectives are, the more meaningful the research will be.
Confusing sample size with response rate.
Just because
you have 600 completed questionnaires doesn't mean it's a good survey.
That's a very usable sample size (i.e. the number of people who responded),
but only if the response rate was good, with little potential response bias.
If only 2% of the people you sent the questionnaire to responded, it doesn't matter how many questionnaires are returned – the response rate is simply too low for it to be considered usable in any way. The key to any research sample is that the people who respond must be representative of the people who did not respond (or were not selected to receive the survey). A very low response rate is usually a sign that only certain types of people are responding, which leads to substantial response bias and misleading data.
Even if the response rate is good, response bias in a mail survey or self-administered survey may be a real problem. In a recent Ellison Research mail survey to church leaders, for instance, the response rate was tracked according to the church size. While the response rate was 20% among small churches, it rose to 54% among the largest churches. Without correcting for this factor in the data and analysis, the sample would have been badly skewed by response bias – and no longer representative of the real world – because of the abundance of large churches participating.
This may seem like a technical issue, but it's critical even in small surveys. Imagine the potential damage if you surveyed your donors about a new program you wanted to start, and the only people who responded to the survey were the 5% who thought it was a great idea.
Not preparing for data entry.
This is one of the most
common mistakes. When data from a questionnaire is being entered into a computer
database, generally it needs to be in a numeric format. If you have ten boxes respondents
might check, the only way to enter which boxes were selected is to have a numeric code on
each box on the questionnaire itself (1 q ,
2 q , 3 q , etc.). This makes data entry a snap.
Without these codes, the data entry person has to stare at the boxes, note that the third,
fourth and ninth boxes were checked, and then enter those numbers. When using check
boxes (or asking respondents to circle options), always assign numeric codes to the
possible responses, to make your life easier on the back end.
Choosing the wrong kind of questions for a situation.
This
is a common problem when asking people to rate an activity they've just experienced.
People leaving a seminar, museum, class, etc. are on their way out, and often
won't stop to fill out a comprehensive survey. When the questionnaire is filled
with open-ended questions, in which respondents have to write their answers in their own
words, many people won't take the time to give good answers, or will skip these
altogether. Make sure to be realistic in what you're expecting people to do in
a questionnaire, and tailor the questions around those constraints.
Missing response options.
This seems like a no-brainer,
but you'd be surprised how often questionnaires have a list of categories such as
"under 2 hours, 4 to 5 hours, 5 to 10 hours, or more." There are two
problems with that list. First, if someone spends five hours on whatever activity this is,
which category do they choose? The categories aren't mutually exclusive.
Second, there are options missing entirely (two to three hours). Make sure your
response options cover every possibility without overlapping.
Unbalanced scales.
This is another very common mistake.
Consider the following rating scale for a question: "excellent, very
good, good, fair." More often than not, questionnaires employ this type of
scale. The problem is that it's highly biased. The scale includes three
positive options, a so-so option, and no negative options. You've basically
just communicated to the respondent that you're seeking a positive rating (or at
worst, an average one). Managers who insist on scales like this often say they
don't want to "suggest that we feel there's a chance our organization could
be rated poorly." What they're really afraid of is finding out that the
organization actually is rated poorly.
Assuming no expert help is available.
Just because you
don't have $20,000 to spend on your research project doesn't have to mean
you're completely on your own. First, some full-scale projects can be
professionally done for $5,000 or less, even through an experienced research firm.
These "quickie" projects may cost much less. Investigate the costs before
assuming you can't afford it.
Second, even if you can't afford to have a research firm do the whole project, you can still get expert help. Some research companies will consult on an hourly basis to review the questionnaire design for you, for example. A couple hundred dollars spent this way is a pretty small investment if you plan to make decisions based on the research (and if you won't be making decisions based on the research, why are you doing it?).
Another area in which you might seek assistance is in data entry and tabulation. Few sources other than a research company or department have a useful data entry program. Often, organizations are reduced to counting responses up by hand, or trying to use general purpose programs such as Microsoft Excel to enter and work with data. If you're using office staff for this purpose, it can take them hours and hours to do this, because the questionnaire and software weren't really designed for simple data entry.
Instead of paying for hours of staff time, it may be much less expensive in the long run to pay a research company to enter and tabulate the data. Again, check into the costs before just assuming it isn't affordable (especially considering what it may be costing you in staff time). Note: if data is to be entered professionally, the research company needs to review the questionnaire before it is disseminated, to make sure it's set up properly to allow for efficient data entry.
Not using internal resources.
Before doing the work
yourself, check out what resources are available within your organization. It's
amazing how many quickie surveys are written by HR directors, conference coordinators,
magazine or newsletter editors, and others within organizations that have qualified
marketing research departments in-house. If you have the resources on staff, by all
means use them!
Even if your organization doesn't have researchers, check with the marketing department. Someone in marketing may have research experience. Some research experience will probably be better than none. Also, before plunging ahead, check with a qualified research company, to see how they might be involved in various parts of the project on an affordable basis.
Any research project will be handled best by people who have extensive experience and expertise in this area. If that's simply not an option, at least explore what you can do to make your "quickie" research project as accurate and useful as possible. If it's not worth that amount of effort, it's not important enough to be doing in the first place.
