Sifting Out the Oddballs: The Key to a Successful Qualitative Study
By Ron Sellers
Originally published in Alert! (Marketing Research Association), May 2004.
It seems that in every quantitative study, there are a few oddball respondents – people who claim to have 300 credit cards, make $15,000 a year and drive a new Bentley, or rate every brand of ice cream as their favorite. While field centers and questionnaire writers do their best to avoid these odd answers and individuals, they inevitably pop up in a survey of 600 people – in large part because these oddballs actually exist within a representative sample of the U.S. (c’mon, you can probably name one or two you know personally).
Getting a strange respondent in a quantitative survey may not be desirable, but it isn’t the end of the world, either. But what researchers and clients need to understand is that the same is not true of qualitative research, particularly in a group setting.
Focus groups are at their most valuable when every single person represents the type of individual the client really wants to reach – the core market, rather than the fringe market. Even one oddball person in a group setting can make things uncomfortable or even downright unhelpful.
Understand that by “oddball,” in this case we’re not talking about people who think FDR is still president or who like to collect used chewing gum. We’re referring to respondents who technically qualify under the definitions in the recruitment screener, but who really don’t fit with the rest of the people in the groups.
Any target market will include a core target and a fringe target. For example, Cadillac doesn’t target people in their twenties, but there are some very young people who drive Cadillacs. Those young drivers are a fringe market for Cadillac, but not their true core target market. In focus groups, you generally want to go after the core target (unless the research is specifically being done within the fringe market to explore market expansion opportunities).
There are women who wear trendy men’s suits to make a fashion statement. But unless a suit manufacturer wants to go after this niche as a unique market segment, how odd would it be to have one woman in a group of men, even if they all wear the client’s suits? The responses of the woman probably would be of little value to the client, the woman would be taking the place of another male respondent who could provide important feedback, and the men in the room may feel uncomfortable or unwilling to open up fully in the presence of someone so different from them in obvious ways.
Yet if not anticipated in the screening process, it would be entirely possible for this to happen. Take two recent real-life examples from projects:
- An investment company wanted to talk to investors with retirement accounts. They insisted that the only qualifier was that respondents had to have an IRA or a 401(k), including accounts administered through work. They did not want to screen based on investment interest or knowledge, because they felt that today’s casual investor could be converted to tomorrow’s serious investor. What they ended up with in some of their groups was six or seven people who could talk knowledgeably about investments and were great respondents, along with three or four people who basically said they signed up for an account through their work because it seemed like a good idea, even though they knew nothing about investments and cared about them even less. Their responses varied from “I don’t know” to “I don’t care” to “I don’t understand anything about investing and I don’t even know what I’m doing in this group.” Not a comfortable situation, or a helpful one.
- The producer of a program for Christian radio stations commissioned groups to test a new program. Screening questions focused on what stations people listened to and how much they listened to Christian radio. A young lady in one group easily qualified because she listened to hours of Christian radio per week. In the groups, however, it was discovered that she was agnostic and had an extremely low opinion of Christian radio. She listened all the time primarily because she liked to make fun of it or disagree with it, the way a political liberal might listen to the Rush Limbaugh Show.
The first scenario described above was caused by a client who was trying to include every type of customer, regardless of whether that person would make a good respondent. They felt it was more important to hear from every type of person they might reach than to hear from people who actually could discuss the subject intelligently. (They regretted their decision when they found these people could not contribute at all in the groups, which were about the design of potential new investment products.)
The second scenario was caused by not anticipating ways in which people could slip through a screener and find their way into groups, even if they really do not fit the target profile.
Avoiding unqualified or distracting recruits is a two-step process involving the definition of the ideal respondent as well as the design of the recruitment screener.
In deciding on the screening qualifications, make sure to target the core market, rather than the fringe. Marketers often say things such as, “Oh, but we reach that kind of customer, too,” or “Well, but wouldn’t hearing a different perspective be valuable?” Think of it this way: Cadillac will happily sell cars to teenagers, but the company probably won’t spend its advertising resources in Seventeen magazine.
It may be helpful to think of the core market for recruiting in the same way a marketer might define the target for an advertising campaign, to make sure recruiting focuses on the people from whom the most valuable, insightful information will come.
A frequent objection to this is that we’re “biasing the sample” or “not fully representing the customer base.” But keep in mind that qualitative research is not projectable in any way. You’re searching for ideas and insights, not quantitative data. As soon as the recruiting terminates someone who does not speak English, uses an articulation screener question, or terminates employees in the client’s industry, the customer base is not fully represented. An inarticulate bank teller who participated in a group last week may be a bank customer, too – but few would argue that he should be recruited for a financial services focus group.
Whether someone should be considered part of the core market may depend on the subject matter for the groups. In the investment example mentioned previously, the problem with the non-savvy investors was that they couldn’t discuss the design or appeal of a new investment product. Had the subject matter been advertising to encourage people to invest more in their IRAs, they might have made great respondents, because it would have been helpful to find out the perceptions of both savvy investors and less-savvy investors.
Screener design is the second step. Try to anticipate every possible way an undesirable respondent could slip in. Think of a screener like the IRS thinks of the tax code, and close those loopholes.
This may require longer screeners with more questions. In reviewing screeners, clients often say things such as, “The question about marital status isn’t necessary, since 95% of our clients are married.” Fine – but what if someone on their customer list from that other 5% slips through? Will that person be a valuable respondent or a distraction?
Clients usually understand targeting their advertising to their core market, as well as closing possible loopholes in project contracts. Applying those same business standards to focus group recruiting, and filling groups with people who represent the core target for the client, will provide more valuable information while avoiding distracting respondents who can’t really participate as part of the group.
