User Interviews: Bias and How to Reduce It
Imagine the following scenario. You’re a UX or CX researcher, and you’re working on a website redesign. You’ve had some preliminary meetings with your client and you’ve done a little of your own research about the marketplace. Based on what you know so far, you have a few ideas that you think would be great for the new website. You’ve set up a half-dozen interviews with your client’s customers to understand their needs, and during the interview, you bring up one of your ideas to solicit their feedback.
Interviewer: “Wouldn’t it be great if you were able to [XYZ] online?”
Interviewee: “I never thought of that before. I guess so…sure!”
Interviewer: “It would be quicker and more efficient…”
Do you believe the interviewee really wants this feature?
If you answered yes, perhaps you shouldn’t have. When you’re conducting user interviews for research, every question that you ask and every topic that you introduce has the potential to skew your results. There are several types of bias that may influence the interviewee’s response to this type of questioning.
Politeness and Halo Effect
Often, people will be polite in their answers to spare your feelings, to be friendly, or to build a rapport. There’s also a chance that your interview subject doesn’t want to make waves in his organization, or maybe she or he fears that a negative response will get back to a boss or an important contact.
Alternately, if you’re conducting interviews and you’ve been introduced as an expert in the field, your interview participants may think you have special insight, and your position might cast a positive glow over all of your suggestions, whether they are good ideas or not. This is called "halo effect."
Tips for remediating politeness and the halo effect:
Downplay your level of expertise and your own investment in the project. “I’m just the researcher…”
Tell them you’re at an early stage in your research, even when you’re not.
Say that negative feedback is encouraged, and is often more helpful than positive feedback.
Query Effect
The Nielsen Norman Group cautions researchers to be aware of the query effect. They claim that when you ask someone for their opinion, that person will come up with an opinion, even if it is unimportant to them or is about something they have very little information about.
Tips for reducing query effect:
Ask followup questions that dig deeper into their need and prompt for specifics. For example, ask if they had the proposed feature for 5 days, how many times would they use it based on their current workload? How much time might a feature like this save for them personally? Try a technique like asking why.
Even better, don’t ask if they want a certain feature. Inquire about their current experience and whether or not they have any suggestions for improvement. They might suggest the feature. If possible, observe them doing whatever process it is that you’d like to replace or improve.
What You See Is All There Is (WYSIATI)
In his book, Thinking, Fast and Slow, Daniel Kahneman proposed a theory called What You See Is All There Is (WYSIATI). Kahneman suggests that your mind is hard-wired to tell a story based on the information you already know and to draw a conclusion from it, forgoing additional research. In the scenario above, the interviewer's findings from her preliminary research made her instinctively identify problems and formulate solutions. This is completely natural, but it can stop you from gathering more information and possibly finding a more appropriate solution.
When conducting need-finding interviews with users or customers, your primary goal is to deepen your knowledge and to find out what you don’t know. User interviews should not be used to gain confirmation or sign-off for your ideas.
Tips for reducing the influence of WYSIATI:
Be open to new information. Although you will absolutely come to the interview with assumptions and solutions, don’t share them with the participant. Pretend that you are a complete novice. Avoid steering the conversation too much in one direction or inadvertently selling the participant on your idea.
Ask questions like “is there anything else that might be useful to us as we research XYZ?”
When in doubt, it’s always best to keep solutions out of need-finding interviews, and instead focus on gathering as much information as possible about the problem. Save coming up with ideas and recommendations for after you’ve completed all your research, when you can properly reflect on more comprehensive findings.