Tag Archives: correlation

Experiments to learn about action

noun_1280396I usually describe my job as helping clients to understand their customers and staff.

In particular, I help clients to understand how people think and feel (their attitudes), how those relate to their experiences, who they are (segmentation), and what they do (behaviour). Usually the ultimate reason is to answer the question…

“If we do X what will happen to Y?”

Learning about people

There are basically two tools in the researcher’s armoury: asking questions and observation. Which works best? Broadly speaking we know that observation works better for behaviour, because people aren’t very good at remembering or (in particular) predicting what they do. We ask questions because it’s the only way to try to understand what’s happening inside people’s heads. It’s not perfect, but it’s often the best tool we have. Where possible, combining both techniques can give insights that neither on its own is capable of.

In either case, however, we’re simply bystanders observing what happens to customers. That means that it’s very difficult to prove the links we identify, especially if we want to predict what will happen if we make a change of some sort.

It’s the knotty old problem of correlation versus causation. The classic example here is the early 20th century study that found a significant link between US households which owned a vacuum cleaner and those that sent their kids to college. The link is true, it held for the population at the time, but it isn’t a direct causal relationship.

The point here is that the correlation holds for prediction (if I know whether or not you have a vacuum cleaner I can make a better-than-chance guess about whether your kids are at college), but fails for intervention (buying a vacuum cleaner doesn’t make it more likely that my child will get into Harvard). That’s why observational studies are flawed if we want to draw conclusions about what actions to take.

Learning about action

To prove a case for intervention, in other words to answer the question “If we do X what will happen to Y?”, we almost always* need to use an experiment. Experiments can be very difficult to design well, so read up on the details, but the important principles are:

  • You need a control condition to serve as a baseline
  • Participants are randomly allocated to receive control or treatment
  • Participants shouldn’t know which group they’re in
  • People interacting with the participants shouldn’t know what group they’re in

It’s usually difficult, and often impossible, to meet all these conditions in practice for the kinds of customer experience change we’re looking at, but that doesn’t mean we shouldn’t try to do the best we can.

One place where the experimental approach has taken hold is in digital A/B testing. Web design (A/B testing is almost an illness at Google) and communications (email subjects etc) understand the value of making data-based decisions about which choices will deliver the best results.

Another is in the public sector, where the popularity of “Nudge” theory has seen behavioural economics tactics teamed with experiments to see which messages have the most impact on behaviour. This discussion of Kirklees Council’s GDPR mailings is an interesting example.

It’s high time we spread that enthusiasm for experiments throughout the rest of the customer experience.

Experiments are the only way for businesses to know the impact of planned changes on customer attitudes and business success.

* There are times it is possible to prove causation from correlation, but it’s tricky. Judea Pearl’s Book of Why is probably worth a read if you’re interested in this stuff.

Tagged , , , , , , ,