Global vs. Local Optimization: When “Better” Gets You Further from Optimal

A/B testing is probably the cornerstone in marketing and arguably the most central process that all other marketing decisions follow. It's pretty common to end up with a champion ad, LP, KW set, whatever, and be unable to find a better option despite near continuous testing. 

In our latest analytics installment, we’re asking the question, “When does following better numbers give you a worse result?” 

Usually what marketers do when testing is insert variations to the existing program and track the differences in performance until we find something better. They aim to find improvements relative to a current baseline rather than exploring entirely new designs or strategies. This approach may lead to settling for local optima—solutions that are better than the current state but not necessarily the best possible solution in the broader landscape.

In this example, we start on the left, find better performance (noted in pink) and stop testing as we approach the brown point. If we kept on testing, we would find the global maximum. This conceptually works best with continuous variables but it's true of any incremental change.

In the above chart, we see change in one variable translate to one metric of performance.  What makes this dynamic tricky is that real life has many interdependent variables. This creates a performance landscape closer to what we see below.  The best creative and best landing page pair could combine to perform worse than an alternative.  

Let’s take two wildly successful eCommerce examples that have achieved global optimization in very different ways.

There are probably not many entities that are A/B tested as much as Amazon’s homepage. I am confident that every image and section has been extensively tested. 

Now consider Apple’s approach to eCommerce: 

If Apple was explicitly testing to maximize CVR on one page, they probably would miss out on peak powered by brand affinity. For Amazon, DR is the entire game – and the ability to buy what you want, quickly, serves to build Amazon’s brand as the eCommerce DR king.  If you A/B everything you’ll end up looking like Amazon but miss Apple’s potentially global max. 

So what’s the takeaway here? Just remember that in the right context, optimizations and tests that lead to bad immediate outcomes might actually be getting you closer to long-term account health.