Qualitative By the Numbers – Blog Three: Six Biases in QLMR Part Two
Recap: First three biases
- The data collected cannot be projected to a universe of similar respondents.
- The moderator is the instrument of research.
- The same Qs are not asked in every group in the series the same way Qs are asked on a survey, which could include skip patterns.
Biases Four, five, and six
4. Respondents will influence each other. – “Oh. I hadn’t thought about that aspect…but now that Jake brought it up — it makes me reconsider my earlier statement.” This “influence” factor is one reason some clients are wary of focus groups because the respondent comes to a new understanding “in the moment” rather than reporting something they may have believed for years. A good moderator will follow up with a probe such as: “What part of what Jake just said, made you reconsider?”
Often a very telling response can arise such as, “I thought that once you had gotten shingles, you couldn’t get it again…sort of like you can only get chickenpox once. But Jake said he got a 2nd case of shingles a year after his first case. That make me reconsider getting that new shingles shot.” That type of response can help clients understand what factors can drive behaviors, so while it is true that Jake influenced a respondent, the focus group process allows the uncovering of a new perception or belief that would not have emerged in a simple “I ask- you answer” line of moderator questions.
This “group influence” issue comes up frequently when client teams are pushed to include qualitative research in the project plan. It is interesting to note that some clients see it as a “big deal,” but don’t stop to think where “influencers” are across all levels of life:
- Parents influence how their kids think about others
- Teachers influence students with what is emphasized in a curriculum
- Spiritual leaders influence their congregations
- Media influences with advertising
- Packaging influences what consumers buy
- Managers influence the way subordinate staff are treated
And this list can be expanded in countless ways. So yes, respondents can influence each other in a focus group, but that dynamic, in the hands of a trained moderator can be mitigated.
5. Respondents sometimes don’t say what is true.
Often, they say what will make them look smart or savvy or win approval. The dynamics of the focus group process may drive respondents to answer differently than if they were alone in an interview or talking to a trusted friend.
The human need for acceptance and approval runs high when strangers are in a group dynamic. A savvy moderator will learn to set ground rules to mitigate this process, but it cannot ever totally be eradicated.
If a moderator suspects the behavior exists, he/she sometimes can ask a question such as, “Is there ever a time when X might be the case?” This approach requires the moderator to take either a devil or angel advocacy position to get under the human approval or look smart dynamic.
6. The data is not representational of a section of the country, or the gender, age or lifestyle of any single group of people in a study. This echoes points made in #1 above: Data cannot be projected to a universe of similar respondents.
What is useful to understand can best be explained via an example: If one wants to measure the level of chlorine in a swimming pool, the method involves taking a small sample of pool water and testing for ratio of chlorine to water in that small sample.
The test will not tell you how deep the water is or the temperature of the pool water or how many people swam in the pool yesterday, but there will be enough data to see if more chlorine is needed to be considered safe for swimming. It is a tipping point to show the way toward making a decision [e.g., add more chlorine or not?]
Summary
Question: With six biases present in qualitative research – what is the key reason that QLMR is so important?
Answer: If you know how someone thinks, or what they believe…you have tipping points to provide clues to future purchases decisions, beliefs and behaviors.
An example that shows how qualitative research played in key role in one American company:
Coca-Cola wanted to introduce a new type of Coke to be closer to the sweetness of Pepsi, so New Coke appeared in blind taste tests with Pepsi and, as the metrics showed, [survey research with hundreds of taste-testers], majority of respondents in the survey preferred New Coke. Coca-Cola promptly rolled out a new ad campaign and also let consumers know that “original” Coca-Cola would no longer be available.
A firestorm of upset and anger rolled across the world and some consumers began to hoard current Coca-Cola and protested/boycotted New Coke through letters and demonstrations.
The management team knew they needed to solve the problem quickly and one of the approaches: Conduct focus groups to find out what drove the high degree of anger/upset among customers?
A series of focus groups across America turned up three key findings:
a. Coca-Cola is not just a product, it is an icon in the history of America – it is tied to many different cultures, lifestyles and part of what America is all about – the threat of its removal struck a chord, akin to losing a loved relative.
b. Coca-Cola had done a very good job with advertising and brand image and removing Iconic Coca-Cola and replacing it with an unknown upstart didn’t honor loyal customers
c. Coca-Cola as a Brand had the same quality of reverence to Americans as the flag and the National Anthem played before sports events.
Yes, survey research proved that New Coke was preferred over Pepsi on taste, but qualitative research illuminated a deep respect for the brand, not just the product.
Insight:
Quantitative research asks questions.
Qualitative research questions the answers.
Written by: Naomi Henderson, CEO & Co Founder