"Let's run a test!" comes the instruction from senior management. Let's improve this page's performance, let's make things better, let's try something completely new, let's make a small change... let's do it like Amazon or eBay. Let's run an A/B test.
In a future post, I'll cover where to test, what to test, and what to look for, but in this post, I'd like to cover how to test. Are you going to test totally new page designs, or just minor changes to copy, text, calls-to-action and pictures? Which is best?
It depends. If you're under pressure to show an improvement in performance with your tests, such as fixing a broken sales funnel, then you are probably best testing small, steady changes to a page in a careful, logical and thoughtful way. Otherwise, you risk seriously damaging your financial performance while the test is running, and not achieving a successful, positive result. By making smaller changes in your test recipes, you are more likely to get performance that's closer to the original recipe - and if your plan and design were sound, then it should also be an improvement :-)
If you have less pressure on improving performance, and iterating seems irritating, then you have the opportunity to take a larger leap into the unknown - with the increased risk that comes with it. Depending on your organisation, you may find that there's pressure from senior management to test a completely new design and get positive results (the situation worsens when they expect to get positive results with their own design which features no thought to prior learnings). "Here, I like this, test it, it should win." At least they're asking you to test it first, instead of just asking you to implement it.
Here, there's little thought to creating a hypothesis, or even iterating, and it's all about creating a new design - taking a large leap into the unknown, with increased risk. Yes, you may hit a winner and see a huge uplift from changing all those page elements; painting the site green and including pictures of the products instead of lifestyle images, but you may just find that performance plummets. It's a real leap into the unknown!
The diagram above represents the idea behind iterative and creative testing. In iterative testing (the red line), each test builds on the ideas that have been identified and tested previously. Results are analysed, recommendations are drawn up and then followed, and each test makes small but definite improvements on the previous. There's slow but steady progress, and performance improves with time.
The blue line represents the climber jumping off his red line and out into the unknown. There are a number of possible results here, but I've highlighted two. Firstly the new test, with the completely untested design, performs very badly, and our climber almost falls off the mountain completely. Financial performance is poor compared to the previous version, and is not suitable for implementation. It may be possible to gain useful learnings from the results (and this may be more than, "Don't try this again!") but this will take considerable and careful analysis of the results.
Alternatively, your test result may accelerate you to improved performance and the potential for even better results - the second blue climber who has reached new heights. It's worth pointing out at this stage that you should analyse the test results as carefully as if it had lost. Otherwise, your win will remain an unknown and your next test may still be a disaster (even if it's similar to the new winner). Look at where people clicked, what they saw, what they bought, and so on. Just because your creative and innovative design won doesn't mean you're off the hook - you still need to work out why you won, just as carefully as if you'd lost.
So, are you iterating or creating? Are you under pressure to test out a new design? Are you able to make small improvements and show ROI? What does your testing program look like - and have you even thought about it?
In a future post, I'll cover where to test, what to test, and what to look for, but in this post, I'd like to cover how to test. Are you going to test totally new page designs, or just minor changes to copy, text, calls-to-action and pictures? Which is best?
It depends. If you're under pressure to show an improvement in performance with your tests, such as fixing a broken sales funnel, then you are probably best testing small, steady changes to a page in a careful, logical and thoughtful way. Otherwise, you risk seriously damaging your financial performance while the test is running, and not achieving a successful, positive result. By making smaller changes in your test recipes, you are more likely to get performance that's closer to the original recipe - and if your plan and design were sound, then it should also be an improvement :-)
If you have less pressure on improving performance, and iterating seems irritating, then you have the opportunity to take a larger leap into the unknown - with the increased risk that comes with it. Depending on your organisation, you may find that there's pressure from senior management to test a completely new design and get positive results (the situation worsens when they expect to get positive results with their own design which features no thought to prior learnings). "Here, I like this, test it, it should win." At least they're asking you to test it first, instead of just asking you to implement it.
Here, there's little thought to creating a hypothesis, or even iterating, and it's all about creating a new design - taking a large leap into the unknown, with increased risk. Yes, you may hit a winner and see a huge uplift from changing all those page elements; painting the site green and including pictures of the products instead of lifestyle images, but you may just find that performance plummets. It's a real leap into the unknown!
The diagram above represents the idea behind iterative and creative testing. In iterative testing (the red line), each test builds on the ideas that have been identified and tested previously. Results are analysed, recommendations are drawn up and then followed, and each test makes small but definite improvements on the previous. There's slow but steady progress, and performance improves with time.
The blue line represents the climber jumping off his red line and out into the unknown. There are a number of possible results here, but I've highlighted two. Firstly the new test, with the completely untested design, performs very badly, and our climber almost falls off the mountain completely. Financial performance is poor compared to the previous version, and is not suitable for implementation. It may be possible to gain useful learnings from the results (and this may be more than, "Don't try this again!") but this will take considerable and careful analysis of the results.
Alternatively, your test result may accelerate you to improved performance and the potential for even better results - the second blue climber who has reached new heights. It's worth pointing out at this stage that you should analyse the test results as carefully as if it had lost. Otherwise, your win will remain an unknown and your next test may still be a disaster (even if it's similar to the new winner). Look at where people clicked, what they saw, what they bought, and so on. Just because your creative and innovative design won doesn't mean you're off the hook - you still need to work out why you won, just as carefully as if you'd lost.
So, are you iterating or creating? Are you under pressure to test out a new design? Are you able to make small improvements and show ROI? What does your testing program look like - and have you even thought about it?
No comments:
Post a Comment