Web Analytics: Reporting, Analysing, Testing and Forecasting

Reporting, Forecasting, Testing and Analysing

As a website analyst, my working role means that I will take a large amount of online data, sift it, understand it, sift it again and search for the underlying patterns, changes, trends and reasons for those changes.  It's a bit like panning for gold.  In the same way as there's a small amount of gold in plenty of river, there's plenty of data to look at, it's just a case of finding what's relevant, useful and important, and then telling people about it.  Waving it under people's noses; e-mailing it round; printing it out and so on – that’s reporting.

However, if all I do with the data I have is report it, then all I'm probably doing is something similar to reporting the weather for yesterday.  I can make many, many different measurements about the weather for yesterday, using various instruments, and then report the results of my measurements.  I can report the maximum temperature; the minimum temperature; the amount of cloud coverage in the sky; the rainfall; the wind speed and direction and the sunrise and sunset times.  But, will any of this help me to answer the question, "What will the weather be like tomorrow?" or is it just data?  Perhaps I'll look at the weather the day before, and the day before that.  Are there trends in any of the data?  Is the temperature rising?  Is the cloud cover decreasing?  In this way, I might be able to spot trends or patterns in the data that would lead me to conclude that yes, tomorrow is likely to be warmer and clearer than yesterday.  Already my reporting is moving towards something more useful, namely forecasting.

The key difference between the weather and online data, hopefully, is that when we come to analyse business data (marketing data or web data), I'm in a position to change the inputs of today’s data.  I can't do much today with my measurements to influence tomorrow's weather, but online I can change my website’s content, text, layout or whatever, and hopefully make some changes to my online performance.  No amount of measuring or reporting is going to change anything – not the weather, not sales performance.  Only changes to the site will lead to changes to the results.  Then, I can not only forecast tomorrow's online performance, but also make changes to try to improve it.

No change means that there's no way to determine what works and what doesn't.  I've been asked to try and determine, "What does good look like?" but unless I make some guesses at what good might be, and test them out on the website, I'll never know.  What I should be able to do, though, is forecast what future performance will look like - this is the advantage of having a website that doesn't change much.  Providing most of the external factors (for example traffic sources, marketing spend, product pricing) stay the same, I should be able to forecast what performance will be like next week.  Unfortunately, the external factors rarely stay the same, which will make forecasting tricky - but it'll be easier than forecasting performance for a changing website!

Consider the following situation:  here's my online promotion, and I've simplified it (I've removed the text, and really simplified it) and I've reduced it to a colour and a shape.  So I launch my campaign with Red Triangle, and measurements show that it is worth 500 points per day (I'm not discussing whether that's clicks, sales, quotes, telephone calls or what - it's a success metric and I've scored it 500 points per day).

       500 points per day

If I make no changes to the promotion, then I'll keep using Red Triangle, and theoretically it'll keep scoring 500 points each day.  However, I might change it to something else, for example, I might test Green Circle. 

300 points per day

Now, Green Circle scores 300 points per day, over a week.  Is that good?  Well, Red Triangle scored 500 points per day, so you might think it'd be worth changing it back.  There's a barrier here, in that if I do change it back to Red Triangle, I have to admit that I made a mistake, and that my ideas weren't as good as I thought they were.  Perhaps I'll decide that I can't face going back to Red Triangle, and I'll try Blue Square instead.
 

 


 200 points per day


But what if Blue Square scores only 200 points each day?  Do I keep running it until I'm sure it's not as good, or do I carry out a test of statistical significance?  Perhaps it'll recover?  One thing is for sure; I know what good looks like (it's a Red Triangle at the moment) but I'll have to admit that my two subsequent versions weren't as good; this is a real mental shift - after all, doesn't optimising something mean making it better?  No, it's not scientific and I should probably start testing Red Circles and Green Triangles, but based on the results I've actually obtained, Red Triangle is the best. 

Maybe I shouldn't have done any testing at all.  After all, Green Circle would cost me 200 points per day, and Blue Square costs me 300 points per day.  And I've had to spend time developing the creative and the text - twice.

Now, I firmly believe that testing is valuable in and of itself.  I’m a scientist, with a strong scientific background, and I know how important testing has been, and will continue to be, to the development of science.  However, one of the major benefits of online marketing and sales is that it's comparatively easy to swap and change - to carry out tests and to learn quickly.  It’s not like changing hundreds of advertising posters at bus stops up and down the country – it’s simply a case of publishing new content on the site.  Even sequential tests (instead of A/B tests) like my example above with the coloured shapes, will provide learning.  What's imperative, though, is that the learning is carried forwards.  Having discovered that Red Triangle is the best of the three shapes tried so far, I would not start the next campaign with a variation of Blue Square.  Learning must be remembered, not forgotten.

Having carried out tests like this, it becomes possible to analyse the results.  I’ve done the measuring and reporting, and it looks like this:  Red Triangle = 500 pts/day, Green Circle = 300 pts/day, Blue Square = 200 pts/day.

Analysing the data is the next step.  In this case, there genuinely isn’t much data to analyse, so I would recommend more testing.  I would certainly recommend against Green Circle and Blue Square, and would propose testing Yellow Triangle instead, to see if it’s possible to improve on Red Triangle’s performance.  It all sounds so easy, and I know it isn’t, especially when there’s a development cycle to build Yellow Triangle, when Green Circle is already on the shelf, and Blue Square is already up and running.  However, that’s my role – to review and analyse the data and recommend action.  There are occasions when there are other practical reasons for not following the data, and flexibility is key here.

In fact, for me, I’m always looking at what to do on the website next – the nugget of gold which is often a single sentence that says, “This worked better than that, therefore I recommend this…” or “I recommend doing this, because it provided an uplift of 100 points per day".  That’s always the aim, and the challenge, when I’m analysing data.  Otherwise, why analyse?  My role isn’t to report yesterday’s weather.  At the very least, I’m looking to provide a forecast for tomorrow’s weather, and ideally, I’d be able to recommend if an umbrella will be needed tomorrow afternoon, or sun tan lotion.  Beyond that, I’d also like to be able to suggest where to seed the clouds to make it rain, too!