Multi Variate Testing - Online Panacea?

I've discussed multi variate testing previously - outlining the theory, the ideas, the maths and ways in which it can be done.  But, in my discussions with other web analytics and optimisation professionals, it seems that MVT isn't really being used all that widely.  This surprised me at first - after all, the number of tools vendors and suppliers who offer MVT is growing all the time, and I assumed from their sales material that it was the next level of A/B testing and the future of online optimisation.  Additionally, it's often marketed as an online panacea, that will highlight the way forwards for your ecommerce business, and bring in double-digit growth (in whichever metric you'd care to measure).

However, out of a dozen or so online professionals that I've spoken to in EMEA, only one had tried it, and had obtained mixed results.  So, why isn't it being taken up and used as widely as I'd expected?  Here are some possibilities:

1.  It's difficult to code
2.  It's difficult to identify MVT opportunities
3.  It's quicker to do an A/B test
4.  It's difficult to explain to the Boss

Let's look at take a look at a simple example of MVT, which will hopefully address the first two challenges that online optimisation professionals face.  I say 'simple', but it's easier than most test ideas because it concerns making some straightforward changes to a web page:  taking things away.

Our content pages; our product pages; our shopping and ecommerce pages are all full of the most important content we can produce for our visitors - glossy images; descriptive text; eye-catching call-to-action buttons; all working together to produce the perfect digital shopping experience.  Or perhaps they aren't.  Perhaps it's a huge mish-mash of competing elements, some of which are helping, and some of which are distracting users and putting them off.  So:  what's working, and what isn't?

Let's take an example from maplin.co.uk  - they sell a wide range of electronics and electrical items.  I've selected one at random, a keyring torch.  I've highlighted below various parts of the page which could be removed as part of a test (I should probably say at this point that this test will require access to the global template for product description pages - if this isn't going to work for you, read ahead to another example). 

Click on the image to see a larger version.

 The product page is very similar to many other ecommerce pages (similar layouts are used on various sites to sell clothes, furniture, games, toys... you name it).  But what's the value of each component, and how do they work together?  I've covered interactions between elements in MVT previously.  The easiest way of working out the optimum combination of elements is to selectively remove them in a multi-variate test.

Here's the recipe definition for each of the various combinations that are possible:

Recipe	Reviews	Social	Tabs	Banner
A	Yes	Yes	Yes	Yes
B	Yes	Yes	Yes	No
C	Yes	Yes	No	Yes
D	Yes	Yes	No	No
E	Yes	No	Yes	Yes
F	Yes	No	Yes	No
G	Yes	No	No	Yes
H	Yes	No	No	No
I	No	Yes	Yes	Yes
J	No	Yes	Yes	No
K	No	Yes	No	Yes
L	No	Yes	No	No
M	No	No	Yes	Yes
N	No	No	Yes	No
O	No	No	No	Yes
P	No	No	No	No

Note that Recipe A is the control state (with all elements present) and Recipe P is removing everything; there are then the various combinations of the four elements in between the two.  (If you're feeling mathematical, you can review how the patterns for each of the four elements changes in a binary-type way - 1000, 1001, 1010, etc. and how the table has certain symmetries).  The number of recipes can be calculated by the number of options for each element (yes or no means 2 options), raised to the power of the number of elements (four elements) so 2⁴= 2 x 2 x 2 x 2 = 16 recipes.


So:  sixteen recipes like this is simply not realistic for a normal A/B/C/D/n test.  The traffic requirements are far too high, and you'd probably be waiting six months for results.  However, because the elements are independent (you don't have to have the reviews included to have the social bar), we can carry out a multivariate test which has only a sample of these recipes, selected to ensure even coverage of the four elements, and which will (with the appropriate tools) enable you to work out the optimal combination, even if you didn't test it.
This example was on a product information page, and as I mentioned above, if you want to test here, your coders will need access to the global template file so that you can run the test across all product information pages.  There are, however, single-page options that would work just as well:
- landing pages for online/offline marketing campaigns
- your home page
- checkout pages

In these cases, each page is (typically) built for a specific purpose and with specific content, so you have much more flexibility on what you can test.  For example, should you have a "Chat online" option and a telephone number on your landing page, as well as an option for online feedback?  Are all three really needed?

This testing has some key advantages: 

1.  You can test a large number of element changes on the page in one go
2.  You can understand (with accurate analysis) the contribution each element makes to page performance
3.  There's no new content required from the design or marketing teams - you're only handling existing content - so no reliance on them for images or content.
4.  It's usually easier to remove page elements with code than it is to insert them, so your code developers will be happier
5.  It's relatively easy to explain what you've tested to the Boss.
6.  In this case, it's definitely quicker than A/B testing, and the more elements you choose to test, the larger the advantage becomes. 

It also has some key requirements:

A.  You're going to need to be able to interpret the results.  This will require some careful analysis and understanding of the maths behind multi-variate testing, in order to work out what each element is contributing (in a positive or negative way).  Many of the tools that are available (here's a list of some of them) offer and promise this kind of analysis, but I'm not aware of it being widely used, so it may be prudent to discuss your requirements with your account manager (I don't work for a tool provider).  You don't really want to get to the end of a test and discover that you have spent eight weeks collectin a mountain of data that you can't climb... that would really require some explaining to the boss.

B.  You're going to need more traffic than a typical A/B test, even if you're using a mathemetical method (such as the Taguchi method) to reduce the recipe requirements, so be prepared to wait longer than usual for your results.


I hope in this blog post I've been able to encourage you to think about using MVT, and shown you how to overcome some of the initial hurdles to getting an MVT idea together - and hopefully into execution.  Please do let me know (either in the comments, or by contacting me) how your efforts go!