As you begin the process of designing an online test, the first thing you'll need is a solid test hypothesis. My previous post outlined this, looking at a hypothesis, HIPPOthesis and hippiethesis. To start with a quick recap, I explained that a good hypothesis says something like, "IF we make this change to our website, THEN we expect to see this improvement in performance BECAUSE we will have made it easier for visitors to complete their task." Often, we have a good idea about what the test should be - make something bigger, have text in red instead of black... whatever.
Stating the hypothesis in a formal way will help to draw the ideas together and give the test a clear purpose. The exact details of the changes you're making in the test, the performance change you expect, and the reasons for the expected changes will be specific to each test, and that's where your web analytics data or usability studies will support your test idea. For example, if you're seeing a large drop in traffic between the cart page and the checkout pages, and your usability study shows people aren't finding the 'continue' button, then your hypothesis will reflect this.
In between the test hypothesis and the test execution are the key questions. These are the key questions that you will develop from your hypothesis, and which the test should answer. They should tie very closely to the hypothesis, and they will direct the analysis of your test data, otherwise you'll have test data that will lack a focus and you'll struggle to tell the story of the test. Think about what your test should show - what you'd like it to prove - and what you actually want to answer, in plain English.
Let's take my offline example from my previous post. Here's my hypothesis: "If I eat more chocolate, then I will be able to run faster because I will have more energy."
It's good - but only as a hypothesis (I'm not saying it's true, or accurate, but that's why we test!). But before I start eating chocolate and then running, I need to confirm the exact details of how much chocolate, what distance and what times I can achieve at the moment. If this was an ideal offline test, there would be two of me, one eating the chocolate, and one not. And if it was ideal, I'd be the one eating the chocolate :-)
So, the key questions will start to drive the specifics of the test and the analysis. In this case, the first key question is this: "If I eat an additional 200 grams of chocolate each day, what will happen to my time for running the 100 metres sprint?"
It may be 200 grams or 300 grams; the 100m or the 200m, but in this case I've specified the mass of chocolate and the distance. Demonstrating the 'will have more energy' will be a little harder to do. In order to do this, I might add further questions, to help understand exactly what's happening during the test - perhaps questions around blood sugar levels, body mass, fat content, and so on. Note at this stage that I haven't finalised the exact details - where I'll run the 100 metres, what form the chocolate will take (Snickers? Oreos? Mars?), and so on. I could specify this information at this stage if I needed to, or I could write up a specific test execution plan as the next section of my test document.
In the online world I almost certainly will be looking at additional metrics - online measurements are rarely as straightforward as offline. So let's take an online example and look at it in more detail.
"If I move the call-to-action button on the cart page to a position above the fold, then I will drive more people to start the checkout process because more people will see it and click on it."
And the key questions for my online test?
"How is the click-through rate for the CTA button affected by moving it above the fold?"
"How is overall cart-to-complete conversion affected by moving the button?"
"How are these two metrics affected if the button is near the top of the page or just above the fold?"
As you can see, the key questions specify exactly what's being changed - maybe not to the exact pixel, but they provide clear direction for the test execution. They also make it clear what should be measured - in this case, there are two conversion rates (one at page level, one at visit level). This is perhaps the key benefit of asking these core questions: they drive you to the key metrics for the test.
"Yes, but we want to measure revenue and sales for our test."
Why? Is your test meant to improve revenue and sales? Or are you looking to reduce bounce rate on a landing page, or improve the consumption of learn content (whitepapers, articles, user reviews etc) on your site? Of course, your site's reason-for-being is to general sales and revenue. Your test data may show a knock-on improvement on revenue and sales, and yes, you'll want to make sure that these vital site-wide metrics don't fall off a cliff while you're testing, but if your hypothesis says, "This change should improve home page bounce rate because..." then I propose that it makes sense to measure bounce rate as the primary metric for the test success. I also suspect that you can quickly tie bounce rate to a financial metric through some web analytics - after all, I doubt that anyone would think of trying to improve bounce rate without some view of how much a successful visitor generates.
So: having written a valid hypothesis which is backed by analysis, usability or other data (and not just a go-test-this mentality from the boss), you are now ready to address the critical questions for the test. These will typically be, "How much....?" and "How does XYZ change when...?" questions that will focus the analysis of the test results, and will also lead you very quickly to the key metrics for the test (which may or may not be money-related).
I am not proposing to pack away an extra 100 grams of chocolate per day and start running the 100 metres. It's rained here every day since Christmas and I'm really not that dedicated to running. I might, instead, start on an extra 100 grams of chocolate and measure my body mass, blood cholesterol and fat content. All in the name of science, you understand. :-)
Stating the hypothesis in a formal way will help to draw the ideas together and give the test a clear purpose. The exact details of the changes you're making in the test, the performance change you expect, and the reasons for the expected changes will be specific to each test, and that's where your web analytics data or usability studies will support your test idea. For example, if you're seeing a large drop in traffic between the cart page and the checkout pages, and your usability study shows people aren't finding the 'continue' button, then your hypothesis will reflect this.
In between the test hypothesis and the test execution are the key questions. These are the key questions that you will develop from your hypothesis, and which the test should answer. They should tie very closely to the hypothesis, and they will direct the analysis of your test data, otherwise you'll have test data that will lack a focus and you'll struggle to tell the story of the test. Think about what your test should show - what you'd like it to prove - and what you actually want to answer, in plain English.
Let's take my offline example from my previous post. Here's my hypothesis: "If I eat more chocolate, then I will be able to run faster because I will have more energy."
So, the key questions will start to drive the specifics of the test and the analysis. In this case, the first key question is this: "If I eat an additional 200 grams of chocolate each day, what will happen to my time for running the 100 metres sprint?"
It may be 200 grams or 300 grams; the 100m or the 200m, but in this case I've specified the mass of chocolate and the distance. Demonstrating the 'will have more energy' will be a little harder to do. In order to do this, I might add further questions, to help understand exactly what's happening during the test - perhaps questions around blood sugar levels, body mass, fat content, and so on. Note at this stage that I haven't finalised the exact details - where I'll run the 100 metres, what form the chocolate will take (Snickers? Oreos? Mars?), and so on. I could specify this information at this stage if I needed to, or I could write up a specific test execution plan as the next section of my test document.
In the online world I almost certainly will be looking at additional metrics - online measurements are rarely as straightforward as offline. So let's take an online example and look at it in more detail.
"If I move the call-to-action button on the cart page to a position above the fold, then I will drive more people to start the checkout process because more people will see it and click on it."
And the key questions for my online test?
"How is the click-through rate for the CTA button affected by moving it above the fold?"
"How is overall cart-to-complete conversion affected by moving the button?"
"How are these two metrics affected if the button is near the top of the page or just above the fold?"
As you can see, the key questions specify exactly what's being changed - maybe not to the exact pixel, but they provide clear direction for the test execution. They also make it clear what should be measured - in this case, there are two conversion rates (one at page level, one at visit level). This is perhaps the key benefit of asking these core questions: they drive you to the key metrics for the test.
"Yes, but we want to measure revenue and sales for our test."
Why? Is your test meant to improve revenue and sales? Or are you looking to reduce bounce rate on a landing page, or improve the consumption of learn content (whitepapers, articles, user reviews etc) on your site? Of course, your site's reason-for-being is to general sales and revenue. Your test data may show a knock-on improvement on revenue and sales, and yes, you'll want to make sure that these vital site-wide metrics don't fall off a cliff while you're testing, but if your hypothesis says, "This change should improve home page bounce rate because..." then I propose that it makes sense to measure bounce rate as the primary metric for the test success. I also suspect that you can quickly tie bounce rate to a financial metric through some web analytics - after all, I doubt that anyone would think of trying to improve bounce rate without some view of how much a successful visitor generates.
So: having written a valid hypothesis which is backed by analysis, usability or other data (and not just a go-test-this mentality from the boss), you are now ready to address the critical questions for the test. These will typically be, "How much....?" and "How does XYZ change when...?" questions that will focus the analysis of the test results, and will also lead you very quickly to the key metrics for the test (which may or may not be money-related).
I am not proposing to pack away an extra 100 grams of chocolate per day and start running the 100 metres. It's rained here every day since Christmas and I'm really not that dedicated to running. I might, instead, start on an extra 100 grams of chocolate and measure my body mass, blood cholesterol and fat content. All in the name of science, you understand. :-)
No comments:
Post a Comment