Incrementality & Halo Effect
Digital ads have changed the advertising landscape specifically because of how easy it is to quantifiable performance. Cookie tracking has long been the backbone of reporting, but it has its problems:
1. The industry is over-reporting non-incremental campaigns
2. We’re not valuing non-tracked benefits. Since cookie tracking is degrading with privacy policy we must now look to bolster our attribution toolkit.
What is the halo effect?
Example of a benefit not tracked. Halo measurement consists of quantifying both direct and indirect effects from any marketing intervention. There is no way to pinpoint specific conversions, however, you can uncover the untraceable benefits of your marketing with the halo effect. The beauty of the test is that it doesn’t require any cookies, and is easy to implement!
How can you measure the halo effect?
Halo measurement focuses on estimating incremental conversions across channels and campaigns. You can read more about incremental conversions in our recent blog post.
Average cost per acquisition considerations often mask the reality of your incremental cost per acquisition. The name of the game is to run a number of experiments in order to quantify how incremental your marketing initiatives really are:
[Average Cost Per Acquisition] = [Ad Spend] / [Conversions] where [Conversions] reflect all conversions directly associated with a specific marketing effort.
[Incremental Cost Per Acquisition] = [Ad Spend] / [Incremental Conversions] where [Incremental Conversions] reflect those conversions that would not otherwise occur without any ad spend.
In the below example, five experiments were run in order to estimate halo effect across channels and categories.
Fig. 1 | Paid Media Performance With Or Without Ad Spend
Google Brand Search: organic picked up part of paid search with still 300 conversions when pausing paid search, and an incremental 200 conversions when enabling Brand Search. Therefore, the average CPA of $40 was translated into an Incremental CPA of $100.
Google and Facebook Prospecting: while performance seems poor at first based on the high average CPA ($100 and $77 respectively), it turns out the vast majority of conversions are incremental, hence an incremental CPA of $103.
Google and Facebook Remarketing: the low average CPA ($25 and $28 respectively) mask a much higher Incremental CPA ($100 and $102 respectively) indicating limited incrementality from remarketing - as expected.
Thanks to these experiments, Blackbird was able to revisit our client’s media mix in order to align incremental CPAs across categories, and ultimately produce better user experience, and superior paid media performance. More specifically, while the average CPA varies from $25 to $100 depending on categories, the incremental CPA is now mostly aligned around $100.
Fig. 2 | Average CPAs aren’t aligned, however Incremental CPAs mostly are!
How to test the halo effect
Using a simple yet robust test design and statistical analysis framework, advertisers can measure the true incrementality of their marketing initiatives and therefore refine their campaign mix and deliver a better experience to their customers, and better numbers for the business.
For this specific scenario, that is measuring the effect of a marketing intervention across two groups over time, Google has shared a superb R package using Bayesian structural time-series models which helps determine performance difference across control and test groups after a given intervention based on relationship before the intervention (i.e relative lift). In practice, all you need to run this script is Control and Test groups, as well as some data pre and post intervention. An easy way to make this happen is to run an experiment in specific geos-only.
Fig. 3 | State-level breakdown with six test states in test group, the rest in control group
Fig. 4 | Incrementality Visualization
Fig. 5 | Incrementality test results example
In this particular case, pausing Google Brand Search resulted in a 25% decrease in combined Paid+Organic conversions with a 99% confidence interval. The probability of obtaining this effect by chance is very small (Bayesian one-sided tail-area probability p = 0.001). This means the causal effect can be considered statistically significant. Now that we’ve determined conversion volume with and without ad spend, we can calculate our incremental CPA, and refine our media mix accordingly!
Source
INFERRING CAUSAL IMPACT USING BAYESIAN STRUCTURAL TIME-SERIES MODELS
