Imagine a straightforward A/B test, between a "red" recipe and a "yellow" recipe. There are different nuances and aspects to the test recipes, but for the sake of simplicity the design team and the testing team just codenamed them "red" and "yellow". The two test recipes were run against each other, and the results came back. The data was partially analysed, and a long list of metrics was produced. Which one is the most important? Was it bounce rate? Exit rate? Time on page? Does it really matter?
Let's take a look at the data, comparing the "yellow" recipe (on the left) and the "red" recipe (on the right).
Let's take a look at the data, comparing the "yellow" recipe (on the left) and the "red" recipe (on the right).
As I said, there's a large number of metrics. And if you consider most of them, it looks like it's a fairly close-run affair.
The yellow team on the left had
28% more shots
8.3% more shots on target
22% fewer fouls (a good result)
Similar possession (4% more, probably with moderate statistical confidence)
40% more corners
less than half the number of saves (it's debatable whether more or fewer saves is better, especially if you look at the alternative to a save)
More offsides and more yellow cards (1 vs 0).
So, by most of these metrics, the yellow team (or the yellow recipe) had a good result. They might even have done better.
However, the main KPI for this test is not how many shots, or shots on target. The main KPI is goals scored, and if you look at this one metric, you'll see a different picture. The 'red' team (or recipe) achieved seven goals, compared to just one for the yellow team.
In A/B testing, it's absolutely vital to understand in advance what the KPI is. Key Performance Indicators are exactly that: key. Critical. Imperative. There should be no more than two or three KPIs and they should match closely to the test plan which in turn, should come from the original hypothesis. If your test recipe is designed to reduce bounce rate, there is little point in measuring successful leads generated. If you're aiming for improved conversion, why should you look at time on page? These other metrics are not-key performance indicators for your test.
Sadly, Brazil's data on the night was not sufficient for them to win - even though many of their metrics from the game were good, they weren't the key metrics. Maybe a different recipe is needed.
Hi David,
ReplyDeleteThis is a good metaphor to explain what a KPI is. Mourinho's mantra!
However, I would highly discourage - if that was your point - to limit yourself only to testing things that can be measured by KPIs. After all, A/B testing is about learning how to win the World Cup not about winning it. In testing you can win regardless of the final score as long as the result is conclusive. And to get conclusive results you must pick the metrics that you intended to change. That of course can but doesn't have be a KPI.
We like to formulate hypothesis this way:
I predict that version B will impact X by Y because of Z.
As long as you are not going to make your country fall into disgrace, you can test:
I predict that playing more long balls into a box (B) will give us three more corners per game (X by Y) because defending team will try to clear the ball (Z).
Of course your team might suck at executing corners so generating more corners will have no impact on KPIs (or maybe negative impact). But, what would be the point of practicing corners if you are not generating any chances?
In this scenario, the outcome of the test is to see if you should start practicing corners.
hope this makes sense :)
cheers,
Lukasz
Hi Lukasz,
DeleteThanks for your comment - I apologise for not replying sooner.
Sometimes, online testing is about learning how to "win" (if you want to test to understand user behaviour), but sometimes, it is just about winning (sometimes, we're aiming to move the needle on revenue, sales and margins). If your analysis of previous data shows that a page is underperforming, then fixing the page will lead to better financial performance, and you may learn a thing or two that will help improve other pages on your site too.
I particularly liked your comment about my team taking corners - it's very, very true that we can sometimes focus on metrics that have no connection whatsoever to the KPIs of the site (or the test). If we can say with confidence that the number of corners we get has no impact on how many goals we score, then yes - absolutely - stop counting corners. And, as you comment, if we find that more corners means more goals, then start practising!!
So, as Einstein said, "Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted."