Website Compromization

Test data, just like any other data, is open to interpretation.  The more KPIs you have, the more the analysis can be pointed towards one winning test recipe or another.  I've discussed this before, and used my long-suffering imaginary car salespeople to show examples of this.

Instead of a clear-cut winner, which is the best on all cases, we often find that we have to select the recipe which is the best for most of the KPIs, or the best for the main KPI, and appreciate that maybe it's not the best design overall.  Maybe the test recipe could be improved if additional design changes were made - but there isn't time to test these extra changes before the marketing team need to get their new campaign live (or the IT team need to deploy the winner in their next launch).  

Do we have enough time to actually identify the optimum design for the site?  Or the page?  Or the element we're testing?  

Anyways - is this science, or is it marketing?  Do we need to make everything on the site perfectly optimized?  Is 'better than control' good enough, or are we aiming for 'even better'?

What do we have?  Is this site optimization, a compromise, or compromization?

Or maybe you have a test result that shows that your users liked a new feature - they clicked on it, they purchased your product.  Does this sound like a success story?  It does, but only until you realise that the new feature you promoted has diverted users' attention away from your most profitable path.  

For example - your new banner promotes new sports laces for your new range of running shoes... so users purchase them but spend less on the actual running shoes.  And the less expensive shoes have a lower margin, so you actually make less profit. Are you trying to sell new laces, or running shoes?

Or you have a new feature that improves the way you sort your search results, with "Featured" or "Recommend" or "Most Relevant" now serving up results that are genuinely what customers want to see.  The problem is, they're the best quality but lowest-priced products in your inventory, so your conversion rate is up by 10% but your average order value is down by 15%.  What do you do?

Are you following customer experience optimization, or compromization?

Sometimes, you'll need to compromise. You may need to sell the new range of shiny accessories with a potential loss of overall profit in order to break into a new market.  You may decide that a new feature should not be launched because although it clearly improves overall customer experience and sales volumes, it would bring down revenue by 5%.  But testing has shown what the cost of the new feature would be (and perhaps a follow-up test with some adjustments would lead to a drop in revenue of only 2%... would you take that?).    In the end, it's going to be a matter of compromization.