What happens if your test wins?

Online testing can be difficult, and getting a winner can be especially hard.  But what happens when your test wins?  Do you have a plan in place?

When a test loses, we tend to analyse the results more carefully than when it wins - to quote Wikipedia's entry on loss aversion and prospect theory, "Losses hurt more than gains feel good."  And if a test loses, there's no immediate action required (apart from switching the test off).  But if it wins, then there'll be the follow-up questions "Why did it win?" (occasionally) and "Can we implement it?" (almost 100% of the time).

The easiest way to implement a test winner is to use your testing tool to simply transfer 100% of traffic to the winning recipe, and a number of web optimisation professionals that I've spoken to use this process.  It's straightforward, However, there are a number of potential drawbacks to this, which you may need to consider:

-If you're using the testing tool to support a winning implementation, you can't use that testing code for further tests.
-In order to display the winner, you'll be relying on a server call to and from the testing tool server (which may slow down page load time), 
-If your contract with your test provider means that you're billed on server calls, you'll be incurring a fee each time the page loads (probably not significant compared to your total revenue and income, but something you may need to consider).

If these are significant issues for you to consider - and to be fair, they may not be important for you - then you'll need to have another route to getting your win implemented.  A good personal network within your company will certainly  help at this stage, as will some understanding of how your site is put together.  You don't need to be completely conversant in HTML, AJAX and JavaScript (although a basic understanding will certainly help you make valid contributions to discussions around implementation - and testing generally) but being able to see how your test win could be implemented will certainly help.


Here are some potentially easy implementations:


Swapping images - if you can show that lifestyle images work better than product-on-a-white-background - is easy enough, and can be implemented in an ongoing way.






Headlines - and text changes - should be straightforward, for hardcoded sites (who uses them?) and those that pull in content from a server-side include.
I know I've said this before (so I apologise, slightly) but CTAs are a great opportunity for implementable test wins.

Making changes or additions that can be incorporated into a template change and rolled out across the site - providing your test win should be implemented globally - is doable.

More complex implementations include redesigning pages - moving content around may be doable (again, this depends on the page template, so beware).

However, adding in new snippets of code on specific pages or totally redesigning a page; creating exceptions for specific pages is trickier, and is likely to cause your HTML/JS development team a headache.  

Don't forget: apart from avoiding negative changes to your site, and informing future design,  the return on investment of your testing program is only going to come from implemented winners.  If implementing your winner is going to take your development team three months to code, then that resource cost will have to be weighed against the revenue and conversion lifts from the implementation.  There's nothing wrong with so-called "simple" tests - you can build momentum that way - and when you start achieving wins, you can scale up and start asking for more development resource.