Stop asking people what they think! (and start watching what they do)

I listen to developers of software and websites all day, they have great ideas and observations but I try to balance what I hear with what I see. Many product managers tell me what their customers want -- they say, “Customers tell me all the time…” or “I’ve been using this software for 10 years – I know what customers want…”
I tell my clients, “Don’t listen to what people say but watch what they do!” People tend to say what they think you want to hear, they make errors recalling what they experienced and finally people have a tough time even recognizing what they did experience!

Here are the top three things people tend to do:

• Respond to question in socially desirable ways
• Make errors in remembering
• Poorly able to “get in touch” with their thoughts and feelings

Responding to question in socially desirable ways

Social Desirability Bias occurs when people want to look as good as possible (which is just about all the time). People tend to emphasizes what they want you to hear and de-emphases what they don’t.  Even when there is something truly bad, they will place it in the best possible light.  When you talk to people in groups, like a focus group, this bias is even more pronounced.

Errors in remembering

We all have trouble remembering what we had for dinner last night, let alone last week (unless it was truly extraordinary). When we recount our experiences, we make all sorts of errors. There is time error, the more time that passes since the event occurred, the harder it is for us to remember it accurately. Leading question error, that’s when the interviewer suggests an answer by how the question is phrased, for example: “Do you think that it’s hard to add a product to the shopping cart?” In fact, Wikipedia lists 50 types of memory errors in this entry! Our memories fail us all the time.

Poorly able to “get in touch” with their thoughts and feelings

Do you know someone who can’t introspect, as Webster (http://www.merriam-webster.com/dictionary/introspection) defines it: “a reflective looking inward : an examination of one's own thoughts and feelings”? We all know someone who sees themselves one way, but is really a very different person. Here are two schools of thought on this: 

• Psychodynamic: our most important feelings and experiences are unconscious and we have many defense mechanisms that keep our experiences hidden from us. Only by learning special skills (with the help of a guide) can we reveal our true feelings.
• Cognitive: Memories are stored in our minds with errors, are retrieved with errors and communicated with errors. With so many points for error along the chain from encoding to retrieval, we can’t totally rely on accessing our thoughts and feelings. Our ability to access our memories is severally hampered and yet people strongly believe that they can gain self-understanding by introspection.
  
  One way Cognitive Psychologist and many Usability Analyst compensate for the difficulty of introspection is the Think Aloud (TA) protocol – asking people to express what they are thinking as they complete a task. If you want to learn more about TA here is a good overview from HowTo.gov.

Conclusion

Clients get their information from all sorts of sources:  customer complaints (only the squeakiest wheels are heard), customer request (again, only the loudest or most influential are heard), the hottest new thing (the cool factor), the competition (more features the better) and many other sources. All this information is biased! All this information is useful, but we should keep in mind the errors we bring to the situation.
Talk to your customers, but also watch what they do by looking at error logs, customer services reports, and other analytics. Most importantly, watch what your customers are doing with your product. For more on listening to your customers, read my article “Listening to users with Iterative Design.”