Segmenting Test Data Intelligently
Sometimes, a simple 'did it win?' will provide your testing stakeholders with the answer they need. Yes, conversion was up by 5% and we sold more products than usual, so the test recipe was clearly the winner. However, I have noticed that this simple summary is rarely enough to draw a test analysis to a close. There are questions about 'did more people click on the new feature?' and 'did we see better performance from people who saw the new banner?'. There are questions about pathing ('why did more people go to the search bar instead of going to checkout?') and there are questions about these users. Then we can also provide all the in-built data segments from the testing tool itself. Whichever tool you use, I am confident it will have new vs return users; users by geographic region; users by traffic source; by landing page; by search term... any way of segmenting your normal website traffic data can be unleashed onto your test data and fill up those slides with pie charts and tables.
After all, segmentation is key, right? All those out-of-the-box segments are there in the tool because they're useful and can provide insight.
Well, I would argue that while they can provide more analysis, I'm not sure about more insights (as I wrote several years ago). And I strongly suspect that the out-of-the-box segments are there because they were easy to define and apply back when website analytics was new. Nowadays, they're there because they've always been there, and because managers who were there at the dawn of the World Wide Web have come to know and love them (even if they're useless. The metrics, not the managers).
Does it really help to know that users who came to your site from Bing performed better in Recipe B versus Recipe A? Well, it might - if the traffic profile during the test run was typical for your site. If it is, then go ahead and target Recipe B for users who came from Bing. And please ask your data why the traffic from Bing so clearly preferred Recipe B (don't just leave it at that).
Is it useful to know that return users performed better in Recipe C compared to Recipe A?
Not if most of your users make a purchase on their first visit: they browse the comparison sites, the expert review sites and they even look on eBay, and then they come to your site and buy on their first visit. So what if Recipe C was better for return users? Most of your users purchase on their first visit, and what you're seeing is a long-tail effect with a law of diminishing returns. And don't let the argument that 'All new users become return users eventually' sway you. Some new users just don't come back - they give up and don't try again. In a competitive marketplace where speed, efficiency and ease-of-use are now basic requirements instead of luxuries, if your site doesn't work on the first visit, then very few users will come back - they'll find somewhere easier instead.
And, and, and: if return users perform better, then why? Is it because they've had to adjust to your new and unwieldy design? Did they give up on their first visit, but decide to persevere with it and come back for more punishment because the offer was better and worth the extra effort? This is hardly a compelling argument for implementing Recipe C. (Alternatively, if you operate a subscription model, and your whole website is designed and built for regular return visitors, you might be on to something). It depends on the size of the segments. If a tiny fraction of your traffic performed better, then that's not really helpful. If a large section of your traffic - a consistent, steady source of traffic - performed better, then that's worth looking at.
So - how do we segment the data intelligently?
It comes back to those questions that our stakeholders ask us: "How many people clicked?" and "What happened to the people who clicked, and those who didn't?" These are the questions that are rarely answered with out-of-the-box segments. "Show me what happened to the people who clicked and those who didn't" leads to answers like, "We should make this feature more visible because people who clicked it converted at a 5% higher rate." You might get the answer that, "This feature gained a very high click rate, but made no impact [or had a negative effect] on conversion." This isn't a feature: it's a distraction, or worse, a roadblock.
The best result is, "People who clicked on this feature spent 10% more than those who didn't."
And - this is more challenging but also more insightful - what about people who SAW the new feature, but didn't click? We get so hung up on measuring clicks (because clicks are the currency of online commerce) that we forget that people don't read with their mouse button. Just because somebody didn't click on the message doesn't mean they didn't see it: they saw it and thought, "Not interesting," "not relevant" or "Okay, that's good to know but I don't need to learn more". The message that says, "10% off with coupon code SAVETEN - Click here for more" doesn't NEED to be clicked. And ask yourself "Why?" - why are they clicking, why aren't they? Does your message convey sufficient information without further clicking, or is it just a headline that introduces further important content. People will rarely click Terms and Conditions links, after all, but they will have seen the link.
So we're going to need to have a better understanding of impressions (views) - and not just at a page level, but at an element level. Yes, we all love to have our messages, features and widgets at the top of the page, in what my high school Maths teacher called "Flashing Red Ink". However, we also have to understand that it may have to be below the fold, and there, we will need to get a better measure of how many people actually scrolled far enough to see the message - and then determine performance for those people. Fortunately, there's an abundance of tools that do this; unfortunately, we may have to do some extra work to get our numerators and denominators to align. Clicks may be currency, but they don't pay the bills.
So: segmentation - yes. Lazy segmentation - no.
