<< Back to Learning Resources
Approx Time

In a marketing landscape where everything is constantly changing, what is a reasonable level of risk on brand investments? How can we breed a test and learn environment where failure is accepted as a natural part of the process?

Our panel, from Google, Karmarama and Monster, share their top-line philosophy on this important aspect of marketing capability.

Key Headlines

What does a test and learn culture look like?

  • Experimentation is central to the culture of some organisations, especially digital disruptors like Google, but for most it is counter-cultural.
  • The best way of creating an innovation culture is via experimentation.
  • Success demands small, fluid multi-disciplinary teams. Management must foster ideas and approaches that challenge the conventional way of thinking and then back them in terms of testing them.
  • And the more experimentation/ test and learn organisations do, the more it becomes part of their everyday way of doing business, the more innovative they become.
  • A ‘workaround’ if all else fails is to split out the innovation business from the main business.

“Creative and data teams need to remember that they are strategic partners and sometimes the data has to lead.”
Lawrence Weber Karmarama timecode 27.00

What benefits will businesses see?

  • Experimentation is the only way you will get causal links, everything else is correlation.
  • Innovation – and the need for experimentation that goes with it – has become a business essential. Do it, or have it done to you.
  • Without innovation and experimentation businesses can quickly become inwardly focused and progressively less relevant. An added benefit of experimentation is that it confuses competitors.

What part does failure play?

  • The essence of an innovation culture is one in which it is ‘safe to fail’.
  • You have to expect that more than two-thirds of tests will fail. If your failure rate isn’t high you are not pushing innovation.
  • You need to accept failure; it often produces more learning than success.
  • ‘Safe to fail’ experimentation empowers employees, harnessing their talent, creativity and enthusiasm.

Illustration showing the idea that failure is an essential part of a sucessful test and learn culture

“When I was at Expedia we accepted that 2 out of every three tests would fail”
Andrew Warner Monster timecode 13.35

How do you achieve a test and learn culture?

  • Use pace – rapid fire. Think sprint.
  • It’s most productive to focus innovations to test on the user/ consumer experience.
  • Be ambitious. Google sets the bar high for success. They are fast to kill off (‘sunset’) products that don’t meet their standards, even if they often reach reasonable scale. But they always extract maximum learnings from products they sunset and often re-purpose elements of them in future successes.
  • The best experiments happen in the real world. They focus on observable behavioural responses rather reported behaviour or intentions from market research.
  • Beware focusing just on short-term response – it’s important to think longer-term too, so evidence of things like brand impact, customer lifetime value, and broader impact on people and society, should be considered alongside short-term response measures.

“You need a dumping strategy for your experiments if they don’t meet your goals, and also an amplification strategy for when they are very successful”
Alison Lomax Google timecode 9.21


(Chair) Alison Lomax Head of Brand Solutions Google
Andrew Warner VP Marketing Monster Europe
Lawrence Weber Managing Partner Innovation Karmarama

<< Back to Learning Resources

“Just what the industry needs, great collaboration between clients and agencies on the topics that drive business growth.”

Bridget Angear, Joint Chief Strategy Officer at AMV BBDO

“It’s great to see the IPA in the UK bring the whole industry and particularly the trade bodies together to focus on effectiveness. This new Marketing Effectiveness initiative will enable people across the industry to work together to build on best practice.”

David Wheldon, Chief Marketing Officer, RBS

“Effectiveness is a team sport, so it was great to see the industry in the widest sense, come together. In an increasingly diverse and fragmented world, only by using all parts of the brain will we solve effectiveness challenges and design our campaigns to deliver short and long term value. That’s why what happens next is important – if the IPA can help facilitate progress on this with a long-term initiative around Marketing Effectiveness, we’ll definitely crack it.”

Bart Michels, Global CEO Kantar Added Value and Country Leader Kantar UK

“The time spent at #EffWeek was extraordinarily effective. It was great to hear the diverse views from all areas of the industry. All tied together with the common themes of accountability and effectiveness.”

Andrew Canter, Global CEO, BCMA

“It has been a privilege to be part of the inaugural Effectiveness Week. The agenda is one which we at O2 UK feel passionately about. To see and hear perspectives across the industry demonstrates how the breadth of marketing effectiveness is increasingly being valued within businesses. Data, insight, social, customer experience, test and learn, ROI, these are all fundamentals and were covered expansively at the event”.

Sandra Fazackerley, Marketing & Consumer, Telefónica UK Limited

“The full week of effectiveness events brought into clear focus the need for marketers to use data and insight to achieve the key business objectives of growth and profits. Marketers today are in a better position to quantify their knowledge of customers and measure the ability of investments in marketing to increase brand and shareholder value.”

Chris Combemale, Group CEO, DMA