Tag Archives: science

Experiments to learn about action

noun_1280396I usually describe my job as helping clients to understand their customers and staff.

In particular, I help clients to understand how people think and feel (their attitudes), how those relate to their experiences, who they are (segmentation), and what they do (behaviour). Usually the ultimate reason is to answer the question…

“If we do X what will happen to Y?”

Learning about people

There are basically two tools in the researcher’s armoury: asking questions and observation. Which works best? Broadly speaking we know that observation works better for behaviour, because people aren’t very good at remembering or (in particular) predicting what they do. We ask questions because it’s the only way to try to understand what’s happening inside people’s heads. It’s not perfect, but it’s often the best tool we have. Where possible, combining both techniques can give insights that neither on its own is capable of.

In either case, however, we’re simply bystanders observing what happens to customers. That means that it’s very difficult to prove the links we identify, especially if we want to predict what will happen if we make a change of some sort.

It’s the knotty old problem of correlation versus causation. The classic example here is the early 20th century study that found a significant link between US households which owned a vacuum cleaner and those that sent their kids to college. The link is true, it held for the population at the time, but it isn’t a direct causal relationship.

The point here is that the correlation holds for prediction (if I know whether or not you have a vacuum cleaner I can make a better-than-chance guess about whether your kids are at college), but fails for intervention (buying a vacuum cleaner doesn’t make it more likely that my child will get into Harvard). That’s why observational studies are flawed if we want to draw conclusions about what actions to take.

Learning about action

To prove a case for intervention, in other words to answer the question “If we do X what will happen to Y?”, we almost always* need to use an experiment. Experiments can be very difficult to design well, so read up on the details, but the important principles are:

  • You need a control condition to serve as a baseline
  • Participants are randomly allocated to receive control or treatment
  • Participants shouldn’t know which group they’re in
  • People interacting with the participants shouldn’t know what group they’re in

It’s usually difficult, and often impossible, to meet all these conditions in practice for the kinds of customer experience change we’re looking at, but that doesn’t mean we shouldn’t try to do the best we can.

One place where the experimental approach has taken hold is in digital A/B testing. Web design (A/B testing is almost an illness at Google) and communications (email subjects etc) understand the value of making data-based decisions about which choices will deliver the best results.

Another is in the public sector, where the popularity of “Nudge” theory has seen behavioural economics tactics teamed with experiments to see which messages have the most impact on behaviour. This discussion of Kirklees Council’s GDPR mailings is an interesting example.

It’s high time we spread that enthusiasm for experiments throughout the rest of the customer experience.

Experiments are the only way for businesses to know the impact of planned changes on customer attitudes and business success.

* There are times it is possible to prove causation from correlation, but it’s tricky. Judea Pearl’s Book of Why is probably worth a read if you’re interested in this stuff.

Tagged , , , , , , ,

Stories & science; belief and knowledge

noun_1011170We talk about Storytelling a lot at TLF.

Finding ways to tell better, more compelling, more persuasive stories is essential if you want to achieve difficult, long term, goals such as culture change or improved Customer Experience.

Good stories touch people emotionally, link their day to day decisions with an outcome that means something to them, and persuade them to make change happen.

Stories are deceiving

But stories can also be dangerous. We find narratives so compelling that we rush to invent stories to explain any fact, statistic, or research finding. Nassim Nicholas Taleb calls this the “Narrative Fallacy”, and data science expert Kaiser Fung reserves a whole category of posts on his blog for what he calls “Story Time“.

Our ability to weave simple explanations for complex, often random, series of events means that the stories we tell ourselves have a feel of inevitability, in hindsight. As Daniel Kahneman suggests in the classic “Thinking, Fast and Slow“:

“…the ultimate test of an explanation is whether it would have made the event predictable in advance.”

So, if making up stories to account for the data we have is flawed, how do we make sense of the world? Science.

Science (a word derived from the Latin scire, “to know”), has developed over centuries as a systematic method for learning about the world. The scientific method is designed to minimise the impact of our cognitive biases, such as making up stories or only noticing things which confirm our beliefs.

Rigorous analysis is the only way to learn robust truths about the world. Every time you (or someone else) come up with an explanation, challenge yourself by asking how you know. If your story is robust, you should have been expecting the finding before you saw it.

Science to learn, stories to teach

Should we give up on storytelling, given that we’re so prone to be misled by it?

Absolutely not. Once we have learned a fundamental truth about the world (through science), we need to communicate that insight to other people. We need to get their attention, persuade them to believe us, and convince them to change what they do.

Too often, in society and in organisations, we see arguments won by people with a simple story over those trying to explain a much more complicated truth. If we want to learn about the world, and use that knowledge to make better decisions, we need to learn to tell better stories with a firm foundation of science at their heart.

On my Data Presentation and Infographics workshop I use this graphic to summarise what I believe the job of a information designer to be:

We need the care and objectivity of a scientist to learn important truths, the flair of a graphic designer to engage people’s attention, and the craft of a storyteller to communicate and persuade people to change.


Tagged , , ,