What happens if your test wins?

Online testing can be difficult, and getting a winner can be especially hard.  But what happens when your test wins?  Do you have a plan in place?

When a test loses, we tend to analyse the results more carefully than when it wins - to quote Wikipedia's entry on loss aversion and prospect theory, "Losses hurt more than gains feel good."  And if a test loses, there's no immediate action required (apart from switching the test off).  But if it wins, then there'll be the follow-up questions "Why did it win?" (occasionally) and "Can we implement it?" (almost 100% of the time).

The easiest way to implement a test winner is to use your testing tool to simply transfer 100% of traffic to the winning recipe, and a number of web optimisation professionals that I've spoken to use this process.  It's straightforward, However, there are a number of potential drawbacks to this, which you may need to consider:

-If you're using the testing tool to support a winning implementation, you can't use that testing code for further tests.
-In order to display the winner, you'll be relying on a server call to and from the testing tool server (which may slow down page load time), 
-If your contract with your test provider means that you're billed on server calls, you'll be incurring a fee each time the page loads (probably not significant compared to your total revenue and income, but something you may need to consider).

If these are significant issues for you to consider - and to be fair, they may not be important for you - then you'll need to have another route to getting your win implemented.  A good personal network within your company will certainly  help at this stage, as will some understanding of how your site is put together.  You don't need to be completely conversant in HTML, AJAX and JavaScript (although a basic understanding will certainly help you make valid contributions to discussions around implementation - and testing generally) but being able to see how your test win could be implemented will certainly help.


Here are some potentially easy implementations:


Swapping images - if you can show that lifestyle images work better than product-on-a-white-background - is easy enough, and can be implemented in an ongoing way.






Headlines - and text changes - should be straightforward, for hardcoded sites (who uses them?) and those that pull in content from a server-side include.
I know I've said this before (so I apologise, slightly) but CTAs are a great opportunity for implementable test wins.

Making changes or additions that can be incorporated into a template change and rolled out across the site - providing your test win should be implemented globally - is doable.

More complex implementations include redesigning pages - moving content around may be doable (again, this depends on the page template, so beware).

However, adding in new snippets of code on specific pages or totally redesigning a page; creating exceptions for specific pages is trickier, and is likely to cause your HTML/JS development team a headache.  

Don't forget: apart from avoiding negative changes to your site, and informing future design,  the return on investment of your testing program is only going to come from implemented winners.  If implementing your winner is going to take your development team three months to code, then that resource cost will have to be weighed against the revenue and conversion lifts from the implementation.  There's nothing wrong with so-called "simple" tests - you can build momentum that way - and when you start achieving wins, you can scale up and start asking for more development resource. 

 

"A/B and MVT doesn't work" on websites

A few months ago, I wrote about my experience of how modern power-tools just don't work, and how my power-drill wasn't working as well as other people said it would.  Well, today, I've just read that A/B and multivariate testing does not work on e-Commerce sites.  And I think it's true.  After all, I read it online, so it must be true.  And it matches with another recent experience I had:  it seems my car is broken.

I've had my car for a number of years - it's a very nice car and I was fortunate enough to have it from new.  It seems to drive well, but I've found that there are a number of fundamental issues with it.

Image credit

The speedometer doesn't work.  The speed readout (both digital and analogue) agree with each other to about 1%, but the maximum speed on the speedo is 150 mph, and I've never managed to achieve those speeds. I took it for a drive recently, and even though my maximum speed was 60 mph, which I achieved for a short while, it still took me 30 mins to travel 20 miles.  No - the speedometer is just plain useless.

The miles per gallon calculation on the dashboard display is inaccurate, and certainly doesn't match the driver's guide (yes, I read the driver's guide).  It keeps changing all the time - what's that all about?  Even the 'average' fuel efficiency figure changes.  Seriously?  Another important part of my car is not working properly.  The dealer says that the figures are based on "average use" (whatever that is), and over a long period of time it will average out, but I'm not prepared to wait that long.  I want to see my driving data match the actual data all the time.

Image credit

The rear-view- and wing-mirrors only show me where I've just been, and - worse still - the image is reversed (transposed left to right, horizontally).  Honestly, just how am I supposed to make sense of the image - or see where I'm going - by looking in them?

I have my own satnav - it's a mid-range TomTom model, so it's good and reliable - but when I use it in my car, the satnav predicted arrival time is wrong.  Worse still, the satnav (which has full maps of Europe) won't integrate with the car's engine and drive it for me, so I am stuck with still having to actually drive the car myself.  And did I mention that the speed readout on the satnav doesn't match the car's speedometer?

Image credit

I have driven the car in heavy traffic and along long stretches of motorways; it's covered country lanes and suburban housing estates - but it's only occasionally performed as well as it's supposed to in the manual, and it only rarely displays MPG figures that match my own forecasts and predictions.

To be honest, the only things that really work as I expect them to are the odometer (mileometer) and the clock.

I've tried monitoring engine temperature (the engine is cold when it starts in winter and is generally warmer in summer); oil pressure (unaffected); brake pad usage (it goes up when I travel through busier streets) and so on, but none of these metrics have been particularly useful at helping me drive better, faster or further.

I've attempted to forecast my future mileage, fuel efficiency and average speed based on the distances I covered yesterday, last week and last month.  It's almost impossible - it seems that it varies day by day.  Having said that, we went on a family holiday round North Wales last August, and I know we're not going this year, so I can expect a decrease in my average mileage for this year.  And my fuel efficiency will be lower because I won't be driving as many of the long, steady dual-carriageway runs that I did last year.  But these are just estimates really, based only on actual events that happened last year but won't happen this year.

My car is on a three-year lease, and the lease expires next summer.  I will probably swap it for a different car, but I'd like to know how that car's fuel efficiency and speed performance will compare with my current one.  Not just what the rear view mirrors look like, or the maximum speed on the speedometer, or even what the driver's manual says. 

Perhaps I should continue to drive as I usually do for the next 6-12 months, and ask someone to drive around, following me, in my potential purchase.  Then, next summer, maybe we can compare their actual performances and choose between keeping my car or swapping it?  Who knows?

I've used car performance as an analogy to online testing before:
How long to run an A/B test for? (2013)
Who holds the steering wheel? (2011)