Monday, 15 April 2013

A/B Testing: Where to Test?

You've bought the software, you've even read the manual and a few books or blogs about testing, and now you're ready to test.  Last time, I discussed how to design your test, and in this post, I'd like to look at where to test.  Which pages are you going to test on?  There's no denying that some tests are easier to build, develop and write the code for, and some pages will be trickier (especially if they're behind secure firewalls or if the page is largely hard-coded with little scope for inserting JavaScript), but there's definitely a group of pages that are good for testing.

Why?  Because an improvement in the financial performance of some of the key pages of your site will have a dramatic impact on the overall performance of your site.

Here are a few good examples of places where testing is likely to be financially productive:

1.  Landing pages with a high bounce rate

Bounce rate is defined as the number of people who land on your site and then click away without visiting any other pages, divided by the total number who landed.  More technically, it's the number of single-page-visits divided by the total number of entries.  Landing pages - especially your home page or a campaign landing page - are some of the mostly highly trafficked pages on your site.  For this reason, small improvements in bounce rate or on click-through rates on landing page calls to action will help to move your financials.  In particular, if your cost per acquisition is high, or the page has a high entrance rate combined with a high bounce rate, then improving page performance here will help improve your financial figures.

2.  Leaky funnels 

If you have a linear payment process (and who doesn't?) then you can monitor page-to-page conversion in a linear way.  If one page is "leaking" - i.e. people are leaving when they reach that particular page, then that's a definite area to look at.  Revisit the page yourself, and generate some ideas to help improve the page's performance.  Why are people leaving?  What's missing?  What's getting in the way?  Where are they going - are they leaving the site or going back to another page on your site?  Which page?  WHY?

3.  Pages with high exit rates
People have to leave your site - it's a matter of fact.  The question is - are they leaving at appropriate exit points, or are they leaving too early?  Some pages on your site are destination pages, and that's not just the 'thank you for your order' page.    There are other pages where visitors are able to identify product features, find out what they want to know, or download a PDF.  These are all acceptable exit pages, and a high exit rate on these pages is probably not a bad thing.  Just to explain - the exit rate is the number of exits from a page, divided by the number of page views for the page, typically expressed as a percentage.

However, other pages are navigation pages - section pages, category pages, header pages, hub pages, whatever you choose to call them.  The page purpose here is to get people deeper into the site, and if people are leaving on these pages, then visitors are not fulfilling their visit purpose because the pages aren't working properly.   This is similar to the leaky funnel for a non-linear path, but in the same way, it indicates that something on the page isn't optimal.

4.  In response to customer comments. 
If you have a survey or feedback mechanism on your site, then take time to read the comments that your visitors have left. Visitors won't necessarily answer your design questions, but their comments can either support am existing test idea you have, she'd light on an issue you've identified with your traffic analysis, or provide you with new test ideas. And they aren't usually hesitant about telling you where the weaknesses in your site are, so be prepared to face some fierce criticism about your site.

The anonymity of a customer survey often leads some visitors to tell you exactly what they think about your site - so don't take it personally! Comments will vary from 'Your site is great' through to 'your site is dreadful' but may take in, 'I can't find the link to track my order,' and 'I can't find spare batteries for my camera' which will help focus your testing efforts.

So, review your stats; check your campaign metrics and listen to what your customers are telling you - you're bound to find some ideas for improving your site, and for testing your own solutions to the problems you've found.  Would you agree?  Do you have other ways of generating test ideas?

In my next post in this series, I intend to look at how long to run a test for and explain statistical significance, confidence and when to call a winner.