Chief Executive at international strategic consultancy Rouser, JP Hanson, talks about the pitfalls that organisations trying to measure their marketing effectiveness face, and argues for a strategic and structured approach to avoid them.


In The Kingdom of God is Within You, Russian literary giant Leo Tolstoy wrote that “the most difficult subjects can be explained to the most slow-witted man if he has not formed any idea of them already; but the simplest thing cannot be made clear to the most intelligent man if he is firmly persuaded that he knows already, without a shadow of a doubt, what is laid before him.”

It is nigh on impossible to, for example, explain to a manager that the notion that employee satisfaction drives performance is a halo effect (performance more likely drives employee satisfaction – nobody wants to work for a loser). To paraphrase Upton Sinclair, getting someone to understand something when their salary depends upon them not understanding it is difficult.

Similarly, many marketers claim to know, without a shadow of a doubt, what marketing effectiveness entails. Though experts within the field are quick to label this a Dunning-Kruger effect, the fact remains that there isn’t a universal definition of the term. This means that we end up with a situation close to that which Sinclair described; there are financial incentives for not embracing external interpretations of what effectiveness is or certain measurements of it. To illustrate, it stands to reason that someone who has an incentive to buy or sell digital media is not only likely to promote a definition of effectiveness that is close to efficiency, they are also likely to promote multi touch attribution models.


The effectiveness perception gap

As a result, brands often find themselves unsure of the best course of action. While readers of this space are familiar with the important work of Les Binet and Peter Field, many are not. Where there is a lack of defined verbiage, there is room for perception to outmaneuver reality, and for a gap to appear.

According to the 2018 Nielsen CMO report, social media and search were perceived as having the highest effectiveness of all channels, with 69% of respondents (in each case) claiming they were either “extremely” or “very” effective. A mere 43% considered TV to be effective. In reality, TV is the most effective media channel on the planet by quite some distance, at least by the definition of effectiveness that many, including Binet and Field, employ. But herein also lies the problem. If we do not agree on what we mean by effectiveness, how can we agree on what communications options, or tactics in general, are most effective?


A strategic framework to bridge the gap

Any marketing effectiveness effort, consequently, must start with the establishment of a framework. Not until we have defined what we mean can we identify and agree upon what is required to measure and forecast.

The definition should be tailored to the individual company and relevant not only to marketing but also the wider business strategy. Alignment acts as a step toward management buy-in; marketing measurement in practice is often about validating work to people who don’t believe in it. Getting the finance department on board throughout the process, and ensuring that the marketing effectiveness targets set are aligned with the overall needs, will build organizational credibility.

Additionally, an established effectiveness framework will help prevent silo formations. Organizations are chain-linked. When each link is managed separately, not only does the company tend to get stuck in a low-effectiveness state, but there are fewer incentives for managers to invest resources in making their link better if other managers are not. Some management teams try to counteract this by awarding budget relative to performance, but end up invoking Goodhart’s Law. E-mail marketing teams, for example, might spam customers to get more conversions and, thereby, more budget. A framework prevents such scenarios, and for organizations that are active across international markets, it improves compliance. If local branches are held to the same established process, there is far less risk that they ignore global directives.


Definition of a benchmark with profit in mind

The next step is to identify a benchmark metric that is appropriate for the strategic intent. The goal of effectiveness is to create a clear understanding of what is and isn’t happening, and a leading KPI allows for more accurate testing. Measuring too many things at once is not only expensive, but often leads to contradicting data and analysis paralysis. Of course, proxies and alternative models can be used to solidify the benchmark, but it is important not to use data the way a drunk uses a lamppost – for support rather than illumination.

It is imperative that the effectiveness benchmark metric takes the long-term into consideration, which means focusing on things other than ROMI. Any marketer looking to spend company money will have return demands thrust upon them whether they like it or not, but measuring effectiveness through efficiency leads to numerous false conclusions and a dangerous bias towards the short-term.

In IPA research, very large market share effects were found in only 3% of short-term cased analysed. For cases exceeding 30 months, it was 38%. ROMI-focused activities also often target consumers with established affiliations to the brand and imminent purchase intentions at the cost of brand growth, long-term base sales and margins. Unsurprisingly, long-term cases (6 months or more) drove an eye-opening 460% more market share growth than short-term cases did. In other words, not only is there is clear incompatibility between maximising efficiency and maximising effectiveness, by measuring the latter using the former one gravely undermines the long-term profitability potential of the brand.

And, when it all comes down to it, the most important thing for marketing is to drive profit. Then again, I would say that – I’m a business-first strategist who works with effectiveness frameworks and whose salary, consequently, depends on it being true.


This article was written for, and first published by, the IPA’s EffWorks.