Tag Archives: surveys

Response rate: the elephant in the room

noun_14049“What’s the sample size?”, you might get asked. Or sometimes (wrongly), “What proportion of customers did you speak to?”. Or even “What’s your margin of error?”.

Important questions, to be sure, but often misleading ones unless you also address the elephant in the room: what was the response rate?

Low response rates are the dirty little secret of the vast majority of quantitative customer insight studies.

As we march boldly into the age of “realtime” high volume customer insight via IVR, SMS or mobile, the issue of low response rates is a body that’s becoming increasingly difficult to hide under the rug.

Why response rate matters

It’s too simplistic to say that response rates are directly correlated with nonresponse bias1, which is what we’re really interested in, but good practice would be to look for response rates well over 50%. Academics are often encouraged to analyse the potential for response bias when their response rates fall below 80%.

The uncomfortable truth is that we mostly don’t know what impact nonresponse bias has on our survey findings. This contrasts with the margin of error, or confidence interval, which allows us to know how precise our survey findings are.

How to assess nonresponse bias

It can be very difficult to assess how much nonresponse bias you’re dealing with. For a start, its impact varies from question to question. Darrell Huff gives the example of a survey asking “How much do you like responding to surveys?”. Nonresponse bias for that question would be huge, but it wouldn’t necessarily be such a problem for the other questions on the same survey. Nonresponse bias is a problem when likelihood of responding is correlated with the substance of the question.

There are established approaches2 to assessing nonresponse bias. A good starting point for a customer survey would be:

  • Log and report reasons for non participation (e.g. incorrect numbers, too busy, etc.)
  • Compare the make-up of the sample and the population
  • Consider following up some nonresponders using an alternative method (e.g. telephone interviews) to analyse any differences
  • Validation against external data (e.g. behavioural data such as sales or complaints)

How to reduce nonresponse bias

Increasing response rate is the first priority. You need to overcome any reluctance to take part (“active nonresponse”), but more importantly “passive nonresponse” from customers who simply can’t be bothered. We find the most effective methods are:

  • Consider interviews rather than self-completion surveys
  • Introduce the survey (and why it matters to you) in advance
  • Communicate results and actions from previous surveys
  • Send at least one reminder
  • Time the arrival of the survey to suit the customer
  • Design the survey to be easy and pleasant for the customer

Whatever your response rate is, please don’t brush the issue under the carpet. If you care about the robustness of your survey report your response rate, and do your best to assess what impact nonresponse bias is having on your results.

1. This article gives a good explanation of why.

2. This article is a good example.

Tagged , , , , ,

Empathy in Customer Experience

empathyI often talk about how important empathy is, but I realised the other day that I was using it in two different ways:

1) Empathy as a tool to inform the design of customer experiences

2) Building empathy at the front line as an essential output of insight

Let’s look at both of those in a bit more detail.

Empathy for design

To design good experiences you need to blend a deep understanding of customers with the skills, informed by psychology, to shape the way they feel. Getting that understanding requires in-depth qualitative research to get inside the heads of individual customers, helping you to see the world the way they see it.

When you understand why people behave the way they do, think the way they think, and (most importantly) feel they way they feel, you can design experiences that deliver the feelings you want to create in customers.

Design, to quote from Jon Kolko’s excellent book Well Designed is…

“…a creative process built on a platform of empathy.”

Empathy is a tool you can use to design better experiences.

Empathy at the front line

Improving the customer experience sometimes means making systematic changes to products or processes, but more often it’s a question of changing (or improving the consistency of) decision making at the front line.

Those decisions are driven by two things: your culture (or “service climate”), and the extent to which your people understand customers. If you can help your people empathise with customers, to understand why they’re acting, thinking, and feeling the way they are, then they’re much more likely to make good decisions for customers.

I’m sure we can all think of a topical example of what it looks like when front line staff are totally lacking in empathy.

The best way to build empathy is to bring customers to life with storytelling research communication. Using real customer stories, hearing their voices, seeing their faces, is much more powerful than abstract communication about mean scores and percentages.

Empathy at the front line is necessary to support good decisions.

Two kinds of empathy?

Are these two types of empathy fundamentally different? Not really. The truth is we are all experience designers. The decisions we make, whether grounded in empathy for the customer or making life easy for ourselves, collectively create the customer experience.

You can draw up a vision for the customer journey of the future, grounded in a deep understanding of customers, but if you fail to engage your colleagues at the front line it will never make a difference to customers.

To design effective experiences you need to start by gaining empathy for customers, but you also need to build empathy throughout your organisation.

Tagged , , , , , , , ,

The Graphic Gameplan

noun_75258My job is to give clients actionable insight about their customers.

“Actionable insight”—what a dreadful phrase! Can we make it a bit less management speak?

My job is to help clients understand what their customers want so that they can do a better job of giving it to them.

The trouble is that understanding is only the first step. If we stop at understanding we’re likely to do more harm than good. I like to quote Bruce Lee:

“Knowing is not enough; we must apply.

Willing is not enough; we must do.”

Bruce Lee

So how do we turn our knowledge about customers, and our willingness to improve, into action?

You need three things: top-level commitment, buy-in from throughout the business, and ideas. To get them, you’re going to need to go further than simply presenting the results of your customer insight—you need to involve your colleagues in creating an action plan.

That means some kind of workshop. Workshops are great, but they can often be feelgood days that generate loads of ideas and enthusiasm with little in the way of concrete results.

Good workshops require structure. Build exercises to explore and generate ideas, but finish with a converging exercise in order to deliver a clear way forward. ‘Gamestorming’ is a great book I turn to when I need an exercise for a workshop.


One of my favourites for helping people move from insight to action is the “Graphic Gameplan“. The beauty of this exercise is that it forces participants to break ideas for improving the customer experience into specific actions, slotting them into a strategic timeline view. It leaves you with momentum, accountability, and a clear vision of what is happening next.

If you don’t have a gameplan for improving your customer experience, maybe it’s time to organise a workshop?

Tagged , , , , , , , ,

Insight & internal comms: a match made in heaven

noun_marriage_192896Every internal communications team I know is crying out for content.

Every customer insight team I know is crying out for airtime and tools to get their messages to staff.

I think you can see where I’m going with this.

So why do we not see more use of customer (and employee) insight in internal comms? I think the main problem is that we, as insight people, have tended to be boring.

We know there’s loads of brilliant stuff in our 60 slides of bar charts, so we send the slide pack off to internal comms. Then we’re a bit hurt they don’t do anything with it.

Bar charts are boring.

Stories are interesting.

But stories are not something that simply emerge from talking to customers. What distinguishes a story is not that it is human (although that’s important), but that it has a point.

To turn insight into effective comms you need to become a storyteller. That means you have to have the courage to craft a story for internal comms to tell, or you could work with them to craft a story together.

Figure out who your audience is, what interests them, and how your insight can change that for the better.

Let customers tell their stories, and flag up the turning points that sent their narratives in different directions.

Stories are told, not found.

Tagged , , , ,