One of the unplanned topics from one of my xChange 2013 huddles was how to get an online testing program up and running, and how to build its momentum. We were discussing online testing more broadly, and this subject came up. Getting a test program up and running is not easy, but during our discussion a few useful hints and tips emerged, and I wanted to add to them here.
Selling plain web analytics isn't easy, but once you have a reporting and analytics program up and running, and you're providing recommendations which are supported by data and seeing improvements in your site's performance, then the next step will probably be to propose and develop a test. Why test?
On the one hand, if your ideas and recommendations are being wholeheartedly received by the website's management team, then you may never need to resort to a test. If you can show with data (and other sources, such as survey responses or other voice-of-customer sources) that there's an issue on your site, and if you can use your reporting tools to show what the problem probably is - and then get the site changed based on your recommendations - and then see an improvement, then you don't need to test. Just implement!
However, you may find that you have a recommendation, backed by data, that doesn't quite get universal approval. How would the conversation go?
"The data shows that this page needs to be fixed - the issue is here, and the survey responses I've looked at show that the page needs a bigger/smaller product image."
"Hmm, I'm not convinced."
"Well, how about we try testing it then? If it wins, we can implement; if not, we can switch it off."
"How does that work, then?"
This is idealised, I know. But you get the idea, and then you can go on to explain the advantages of testing compared to having to implement and then roll back (when the sales figures go south).
The discussions we had during xChange showed that most testing programs were being initiated by the web analytics team - there were very few (or no) cases where management started the discussion or wanted to run a test. As web professionals, supporting a team with sales and performance targets, we need to be able to use all the online tools available to us - including testing - so it's important that we know how to sell testing to management, and get the resources that it needs. From management's perspective, analytics requires very little support or maintenance (compared to testing) - you tag the site (once, with occasional maintenance) and then pay any subscriptions to the web analytics provider, and pay for the staff (whether that's one member of staff or a small team). Then - that's it. No additional resource needed - no design, no specific IT, no JavaScript developers (except for the occasional tag change, maybe). And every week, the mysterious combination of analyst plus tags produces a report showing how sales and traffic figures went up, down or sideways.
By contrast, testing requires considerable resource. The design team will need to provide imagery and graphics, guidance on page design and so on. The JavaScript developers will need to put mboxes (or the test code) around the test area; the web content team will also need to understand the changes and make them as necessary. And that's just for one test. If you're planning to build up a test program (and you will be, in time) then you'll need to have the support teams available more frequently. So - what are the benefits of testing? And how do you sell them to management, when they're looking at the list of resources that you're asking for?
1. Testing provides the opportunity to do that: test something that the business is already thinking of changing. A change of banners? A new page layout? As an analyst, you'll need to be ahead of the change curve to do this, and aware of changes before they happen, but if you get the opportunity then propose to test a new design before it goes live. This has the advantage of having most of the resource overhead already taken into account (you don't need to design the new banner/page) but it has one significant disadvantage: you're likely to find that there's a major bias towards the new design, and management may just go ahead and implement anyway, even if the test shows negative results for it.
2. A good track record of analytics wins will support your case for testing. You don't have to go back to prior analysis or recommendations and be as direct as, "I told you so," but something like, "The changes made following my analysis and recommendations on the checkout pages have led to an improvement in sales conversion of x%." is likely to be more persuasive. And this brings me neatly on to my next suggestion.
3. Your main aim in selling testing is to ensure you can get the money for testing resources, and for implementation. As I mentioned above, testing takes time, resource and manpower - or, to put it another way, money. So you'll need to persuade the people who hold the money that testing is a worthwhile investment. How? By showing a potential return on that investment.
"My previous recommendation was implemented and achieved a £1k per week increase in revenue. Additionally, if this test sees a 2% lift in conversion, that will be equal to £3k per week increase in revenue."
It's a bit of a gamble, as I've mentioned previously in discussing testing - you may not see a 2% lift in conversion, it may go flat or negative. But the main focus for the web channel management is going to be money: how can we use the site to make more money. And the answer is: by improving the site. And how do we know if we're improving the site? Because we're testing our ideas and showing that they're better than the previous version.
You do have the follow-up argument (if it does win), that, "If you don't implement this test win it will cost..." because there, you'll know exactly what the uplift is and you'll be able to present some useful financial data (assuming that yesterday's winner is not today's loser!). Talk about £, $ or Euros... sometimes, it's the only language that management speak.
4. Don't be afraid to carry out tests on the same part of a page. I know I've covered this previously - but it reduces your testing overhead, and it also forces you to iterate. It is possible to test the same part of a page without repeating yourself. You will need to have a test program, because you'll be testing on the same part of a page, and you'll need to consult your previous tests (winners, losers and flat results) to make sure you don't repeat them. And on the way, you'll have chance to look at why a test won, or didn't, and try to improve. That is iteration, and iteration is a key step from just testing to having a test program.
5. Don't be afraid to start by testing small areas of a page. Testing full-page redesigns is lengthy, laborious and risky. You can get plenty of testing mileage out of testing completely different designs for a small part of a page - a banner, an image, wording... remember that testing is a management expense for the time being, not an investment, and you'll need to keep your overheads low and have good potential returns (either financial, or learning, but remember that management's primary language is money).
6. Document everything! As much as possible - especially if you're only doing one or two tests at a time. Ask the code developers to explain what they've done, what worked, what issues they faced and how they overcame them. It may be all code to you, but in a few months' time, when you're talking to a different developer who is not familiar with testing and test code, your documentation may be the only thing that keeps your testing program moving.
Also - and I've mentioned this before - document your test designs and your results. Even if you're the only test analyst in your company, you'll need a reference library to work from, and one day, you might have a colleague or two and you'll need to show them what you've done before.
So, to wrap up - remember - it's not a problem if somebody agrees to implement a proposed test. "No, we won't test that, we'll implement it straight away." You made a compelling case for a change - subsequently, you (representing the data) and management (representing gut feeling and intuition) agreed on a course of action. Wins all round.
Setting up a testing program and getting management involvement requires some sales technique, not just data and analysis, so it's often outside the analyst's usual comfort zone. However, with the right approach to management (talk their language, show them the benefits) and a small but scaleable approach to testing, you should - hopefully - be on the way to setting up a testing program.
Sometimes, launching a test program is like defying gravity.
Selling plain web analytics isn't easy, but once you have a reporting and analytics program up and running, and you're providing recommendations which are supported by data and seeing improvements in your site's performance, then the next step will probably be to propose and develop a test. Why test?
On the one hand, if your ideas and recommendations are being wholeheartedly received by the website's management team, then you may never need to resort to a test. If you can show with data (and other sources, such as survey responses or other voice-of-customer sources) that there's an issue on your site, and if you can use your reporting tools to show what the problem probably is - and then get the site changed based on your recommendations - and then see an improvement, then you don't need to test. Just implement!
However, you may find that you have a recommendation, backed by data, that doesn't quite get universal approval. How would the conversation go?
"The data shows that this page needs to be fixed - the issue is here, and the survey responses I've looked at show that the page needs a bigger/smaller product image."
"Hmm, I'm not convinced."
"Well, how about we try testing it then? If it wins, we can implement; if not, we can switch it off."
"How does that work, then?"
 |
| The ideal 'we love testing' management meeting. Image credit. |
The discussions we had during xChange showed that most testing programs were being initiated by the web analytics team - there were very few (or no) cases where management started the discussion or wanted to run a test. As web professionals, supporting a team with sales and performance targets, we need to be able to use all the online tools available to us - including testing - so it's important that we know how to sell testing to management, and get the resources that it needs. From management's perspective, analytics requires very little support or maintenance (compared to testing) - you tag the site (once, with occasional maintenance) and then pay any subscriptions to the web analytics provider, and pay for the staff (whether that's one member of staff or a small team). Then - that's it. No additional resource needed - no design, no specific IT, no JavaScript developers (except for the occasional tag change, maybe). And every week, the mysterious combination of analyst plus tags produces a report showing how sales and traffic figures went up, down or sideways.
By contrast, testing requires considerable resource. The design team will need to provide imagery and graphics, guidance on page design and so on. The JavaScript developers will need to put mboxes (or the test code) around the test area; the web content team will also need to understand the changes and make them as necessary. And that's just for one test. If you're planning to build up a test program (and you will be, in time) then you'll need to have the support teams available more frequently. So - what are the benefits of testing? And how do you sell them to management, when they're looking at the list of resources that you're asking for?
1. Testing provides the opportunity to do that: test something that the business is already thinking of changing. A change of banners? A new page layout? As an analyst, you'll need to be ahead of the change curve to do this, and aware of changes before they happen, but if you get the opportunity then propose to test a new design before it goes live. This has the advantage of having most of the resource overhead already taken into account (you don't need to design the new banner/page) but it has one significant disadvantage: you're likely to find that there's a major bias towards the new design, and management may just go ahead and implement anyway, even if the test shows negative results for it.
2. A good track record of analytics wins will support your case for testing. You don't have to go back to prior analysis or recommendations and be as direct as, "I told you so," but something like, "The changes made following my analysis and recommendations on the checkout pages have led to an improvement in sales conversion of x%." is likely to be more persuasive. And this brings me neatly on to my next suggestion.
3. Your main aim in selling testing is to ensure you can get the money for testing resources, and for implementation. As I mentioned above, testing takes time, resource and manpower - or, to put it another way, money. So you'll need to persuade the people who hold the money that testing is a worthwhile investment. How? By showing a potential return on that investment.
"My previous recommendation was implemented and achieved a £1k per week increase in revenue. Additionally, if this test sees a 2% lift in conversion, that will be equal to £3k per week increase in revenue."
It's a bit of a gamble, as I've mentioned previously in discussing testing - you may not see a 2% lift in conversion, it may go flat or negative. But the main focus for the web channel management is going to be money: how can we use the site to make more money. And the answer is: by improving the site. And how do we know if we're improving the site? Because we're testing our ideas and showing that they're better than the previous version.
You do have the follow-up argument (if it does win), that, "If you don't implement this test win it will cost..." because there, you'll know exactly what the uplift is and you'll be able to present some useful financial data (assuming that yesterday's winner is not today's loser!). Talk about £, $ or Euros... sometimes, it's the only language that management speak.
4. Don't be afraid to carry out tests on the same part of a page. I know I've covered this previously - but it reduces your testing overhead, and it also forces you to iterate. It is possible to test the same part of a page without repeating yourself. You will need to have a test program, because you'll be testing on the same part of a page, and you'll need to consult your previous tests (winners, losers and flat results) to make sure you don't repeat them. And on the way, you'll have chance to look at why a test won, or didn't, and try to improve. That is iteration, and iteration is a key step from just testing to having a test program.
5. Don't be afraid to start by testing small areas of a page. Testing full-page redesigns is lengthy, laborious and risky. You can get plenty of testing mileage out of testing completely different designs for a small part of a page - a banner, an image, wording... remember that testing is a management expense for the time being, not an investment, and you'll need to keep your overheads low and have good potential returns (either financial, or learning, but remember that management's primary language is money).
6. Document everything! As much as possible - especially if you're only doing one or two tests at a time. Ask the code developers to explain what they've done, what worked, what issues they faced and how they overcame them. It may be all code to you, but in a few months' time, when you're talking to a different developer who is not familiar with testing and test code, your documentation may be the only thing that keeps your testing program moving.
Also - and I've mentioned this before - document your test designs and your results. Even if you're the only test analyst in your company, you'll need a reference library to work from, and one day, you might have a colleague or two and you'll need to show them what you've done before.
So, to wrap up - remember - it's not a problem if somebody agrees to implement a proposed test. "No, we won't test that, we'll implement it straight away." You made a compelling case for a change - subsequently, you (representing the data) and management (representing gut feeling and intuition) agreed on a course of action. Wins all round.
Setting up a testing program and getting management involvement requires some sales technique, not just data and analysis, so it's often outside the analyst's usual comfort zone. However, with the right approach to management (talk their language, show them the benefits) and a small but scaleable approach to testing, you should - hopefully - be on the way to setting up a testing program.
No comments:
Post a Comment