Analysis is Easy, Interpretation Less So

Every time we open a spreadsheet, or start tapping a calculator (yes, I still do), or plot a graph, we start analysing data.  As analysts, it is probably most of what we do all day.  It's not necessarily difficult - we just need to know which data points to analyse, which metrics we divide by each other (do you count exit rate per page view, or per visit?) and we then churn out columns and columns of spreadsheet data.  As online or website analysts, we plot the trends over time, or we compare pages A, B and C, and we write the result (so we do some reporting at the end as well).

Analysis. Apparently.

As business analysts, it's not even like we have complicated formulae for our metrics - we typically divide X by Y to give Z, expressed to two decimal places, or possibly as a percentage.  We're not calculating acceleration due to gravity by measuring the period of a pendulum (although it can be done), with square roots, fractions, and square roots of fractions.

Analysis - dare I say it - is easy.

What follows is the interpretation of the data, and this can be a potential minefield, especially when you're presenting to stakeholders. If analysis is easy, then sometimes interpretation can really be difficult.

For example, let's suppose revenue per visit went up by 3.75% in the last month.  This is almost certainly a good thing - unless it went up by 4% in the previous month, and 5% in the same month last year.  And what about the other metrics that we track?  Just because revenue per visit went up, there are other metrics to consider as well.  In fact, in the world of online analysis, we have so many metrics that it's scary - and so accurate interpretation becomes even more important.

Okay, so the average-time-spent-on-page went up by 30 seconds (up from 50 seconds to 1 minute 20).  Is that good?  Is that a lot?  Well, more people scrolled further down the page (is that a good thing - is it content consumption or is it people getting well and truly lost trying to find the 'Next page' button?) and the exit rate went down.  

Are people going back and forth trying to find something you're unintentionally hiding?  Or are they happily consuming your content and reading multiple pages of product blurb (or news articles, or whatever)?  Are you facilitating multiple page consumption (page views per visit is up), or are you sending your website visitors on an online wild goose chase (page views per visit is up)?  Whichever metrics you look at, there's almost always a negative and positive interpretation that you can introduce.

This comes back, in part, to the article I wrote last month - sometimes two KPIs is one too many.  It's unlikely that everything on your site will improve during a test.  If it does, pat yourself on the back, learn and make it even better!  But sometimes - usually - there will be a slight tension between metrics that "improved" (revenue went up), metrics that "worsened" (bounce rate went up) and metrics that are just open to anybody's interpretation (time on page; scroll rate; pages viewed per visit; usage of search; the list goes on).  In these situations, the metrics which are open to interpretation need to be viewed together, so that they tell the same story, viewed from the perspective of the main KPIs.  For example, if your overall revenue figures went down, while time on page went up, and scroll rate went up, then you would propose a causal relationship between the page-level metrics and the revenue data:  people had to search harder for the content, but many couldn't find it so gave up.

On the other hand, if your overall revenue figures went up, and time on page increased and exit rate increased (for example), then you would conclude that a smaller group of people were spending more time on the page, consuming content and then completing their purchase - so the increased time on page is a good thing, although the exit rate needs to be remedied in some way.  The interpretation of the page level data has to be in the light of the overall picture - or certainly with reference to multiple data points.  

I've discussed average time on page before.  A note that I will have to expand on sometime:  we can't track time on page for people who exit the page.  It's just not possible with standard tags. It comes up a lot, and unless we state it, our stakeholders assume that we can track it:  we simply can't.  

So:  analysis is easy, but interpretation is hard and is open to subjective viewpoints.  Our task as experienced, professional analysts is to make sure that our interpretation is in line with the analysis, and is as close to all the data points as possible, so that we tell the right story.

In my next posts in this series, I go on to write about how long to run a test for and explain statistical significance, confidence and when to call a test winner.

Other articles I've written on Website Analytics that you may find relevant:

Web Analytics - Gathering Requirements from Stakeholders

Analysis is Easy, Interpretation Less So
Telling a Story with Web Analytics Data
What is "Direct Traffic"?
Reporting, Analysing, Testing and Forecasting
Pages with Zero Traffic

Is That A Lot?

No matter how well we research and present our numerical data, there is always one question that we will probably always face:  "Is that a lot?"  Does it show a lack of understanding on the part of our audience, or did we just not make it perfectly clear that our recommendation is earth-shattering, game-changing and generally just awesome?  There are various reasons why our data isn't being received with the awe that it deserves; here are some ways of addressing the gaps.

External Comparison
If you want to give an effective image of how many people visited your site (either normally, or in response to a marketing campaign) then it may be useful to compare them to an external figure.  For example, if you saw 90,000 people respond to your marketing campaign, you might get asked if that's a lot.  One answer:  it's equal to the capacity of Wembley Stadium in London. 
 
As another example, 8 million people fly in an airliner each day.  Is that a lot?  On the one hand, it's about 0.11% of the total population of the world.  On the other hand, it's almost equal to the population of London (8.67 million).  Is that a lot?

Naturally, it helps to have a list of populations for various cities, towns and countries if you want to keep using external comparisons.

Internal Comparison
Probably more effective than external comparison, this uses your current data on your website, sales, revenue, whichever, and calls out how current performance compares to other parts of your site.  For example, you might compare sales or traffic for shoes with shirts, trousers and socks; or perhaps you'd compare SUVs with sports, hatchbacks and estate cars.

This is most effective if you find that traffic to the different internal sections of your site changes (e.g. seasonally) but isn't going to work well if there's little change in the relative traffic to each part (e.g. if shirts always has more traffic than shoes, and shoes are more popular than trousers, etc.).  You could also express this as a share of total traffic: "Menswear traffic rose from 30% to 40% of total site traffic this week" (which also eliminates the overall variation in site traffic - whether you want to make use of that effect or not).

Trending
- This was the highest for six months
- ...the lowest for eight months
- ... the second highest this year
- ...making it the third lowest in the last five years

If you're going to pursue this strategy, then it also helps to have a reason why things were high six months ago, or low eight months ago e.g. "This month was the lowest for 15 months, when one of our competitors had a massive sale and undercut us for three consecutive weeks." or "This month was the highest for six months, when we had the pre-Christmas sale."   This helps connect the data to real-life events and brings the data to life.  "Do you remember that time when our site was really busy?  Well, it's even busier than that."

- The UK Meteorological Office do this with their "since records began" expression, and according to NASA, July 2016 was the world's hottest month since records began.

 - The UK census showed a population boom that was also the largest since records began.

 - TV data shows that the Rio Olympics in 2016 got the smallest TV audience in Brazil since the 2004 games.  The reason is that more people streamed the games online:  it's always good to have a reason why a metric jumps or falls sharply (read more in my article about moving from reporting to insight).


As you can see, these kinds of 'highest since/lowest since' statements really make great headlines, so don't be afraid of using them if you want to instil a sense of urgency into your reporting or analysis.
If it's been a fairly average month, and hasn't been the biggest/best/worst/lowest month since Christmas/Thanksgiving/Easter/ever, then you could always do a comparison with the previous period.  Year on year, or month over month comparisons are widely used - especially year-on-year (YoY) which conveniently removes any seasonal effects (if it was Back to School this year, it will have been Back to School last year too). 

Trends, of course, are vey easily represented as graphs - line charts or bar charts, depending on your personal preference.  Here's an example I've used in the past, showing the current year trend, and last year's trend.  I thought it was fairly intuitive, and with a bit of stakeholder education (I showed them what it was and what it meant), it became the standard way of showing YoY trends, and the current trend.  The bars are last year; the line is this year.  The colour of the line matched the colour of the particular part of the site being discussed (e.g. blue could be men's wear, pink could be ladies' wear - beware of using red and green, as these are shortcuts for 'bad' and 'good' respectively).

Financial Metrics

If you really want to make your stakeholders take action, connect your recommendations and analysis to the money.  Nobody's sure if 19,354 visitors is a lot, but everybody knows how much £19,354 is, or how much $19,354 will buy you.  Whether you go for a trended view, or an external or internal comparison, you can still say, "We made $54,218 this week.  Is that a lot?  It's 15% more than the week before, but 4% less than the same week last year."  Suddenly everybody's paying attention; and if you're lucky, they'll ask you what you recommend doing about it.  Have your answers ready!

I've written before about actionable analysis - connecting any metric to a KPI or to a financial figure immediately makes analysis more actionable.

Conclusion
So, when you tell your manager that the figure is $150, and your manager decides it's time to emulate Admiral Kirk by asking "Is that a lot?" you can be ready with a comparative or trended view of the data to say, "Well, it may not buy you a gold watch, but it'll get you two bus tickets to the whales on the other side of San Francisco".

Admiral Kirk asks, "Is that a lot?"

Data, Analysis, Insight and Wisdom

Good web analysts love producing 'actionable insights' - it's the way we add value to the business we're in; it makes our managers happy - it's like finding hidden treasure.  But what are actionable insights (five years ago I asked who makes them actionable - the analyst or the manager) and how can get better at finding them and sharing them?

Web analytics starts with data - this could be various  kinds of data depending on the business model you're following.  So, in order to keep things industry-neutral, I'm going to focus on an unrelated area, and see what we can learn from it.  Yes, I'm going back to my old favourite:  reporting and analysing the weather.

In meteorology, scientists gather all kinds of data from the atmosphere.  They are interested in collecting multiple types of data - or "data points" - from multiple sources in various ways.  And the good news is that this data is quantitative (it can be given a number).

A thermometer will tell you the air and ground temperature - how comfortable things are at the moment
A barometer - the air pressure where you are at the moment
A hygrometer will measure humidity (or possibly rainfall) - depending on conditions
An anemometer - is used to measure wind speed and direction , and will tell you which way things are going to change and where to look for what's coming next.

Each instrument will tell you different things about how things are at the moment.  The anemometer can tell you which way things are going to change, and to some extent, which way to look to see what's going to happen next.  The data that these devices will give you will almost always be numerical (or partly numerical), and certainly abstract.  Each one individually will give you a partial picture of the current situation.  None of them by themselves will actually tell you anything meaningful:  is it raining?  Yes, but has it started, has it stopped, is it getting heavier or lighter?  The temperature may be 16^o C but what time of day is it; what time of year is it and is the temperature going up or down?

What is needed here is some analysis.

Analysis is the art of combining the data sources to tell you something more meaningful, with a wider view, and painting a better picture.

One of the easiest forms of analysis is comparison.  It's hot today, but is it hotter than yesterday, this time last week, or this time last year?  Meteorologists typically compare year on year - there's little benefit in comparing sunny May with rainy April (in the UK, in theory).  But comparing May 2016 with May 2015 will tell you if we're having a good spring season.

And comparison leads naturally to trending.  It might be raining more today than it was yesterday, but how does that pattern compare over a longer time period?  And if you want to present this data to a wider audience, you'll either compile a table of data, or produce a graph of your data.  And the analysis is already starting - comparing two forms of data (typically time and another measurement) and producing comparable data (and possibly even trends).

Another form of analysis is statistical analysis - comparing averages, ranges, populations and so on.  Providing you and your audience are agreed on which average you're taking, what it means and what its potential drawbacks are, this can be a very useful form of analysis.

A note:  analysis is not just plotting graphs.  No, really, it isn't.  A spreadsheet can plot graphs, but analysis requires brainpower.  Therefore, plotting graphs (by itself) isn't analysis.  It can help to direct your analysis, and tell you what the data is saying, but a graph is just a set of lines on a page.  Plotting good, meaningful graphs is an exercise by itself, and data visualization is a whole subject of its own.  And a sidenote to this note: there are times when a simple, basic bar chart will be more informative and drive more action than any trendy visualisation with arrows, flowcharts and nodes.  Good doesn't mean visually impressive.

Good analysis breaks down the data into meaningful and relevant sections that will start to tell you more than just the individual data points.  Analysis will combine data points:  for example, imagine combining temperature data combined with geographic data, compared to average data:

This data has been presented in a very accessible way, and you can see at a glance that the southern half of the UK had a slightly-wetter-than-average January, whereas the northern half of the UK, and especially Scotland, was much drier than average.  

This is analysis clearly presented.  However, it isn't insight:  I haven't explained why the rainfall varied so much.  And if you're looking to explain why the rainfall in January was less than in June (for example), then you can easily point to annual trends:  the rain in January is always less than in June.

Insight

Insight is the next step from analysis, and insight will often show you WHY something is happening.  Yes, I know you won't fully answer "why" visitors behave in the way they do just by consulting quantitative data, but it's a start - and additionally, you'll be able to answer why a number went up, down or sideways.  You'll know you're beginning to show insight when you've stopped drawing graphs and tables, and started writing in sentences.  And not just describing what the data is saying, either, but explaining what's actually happening and the underlying causes.  "Total sales this week fell from 100k to 74k" isn't analysis.  "Total sales fell from 100k to 74k due to the conclusion of the summer sale and a drop in men's shoe sales; last year we continued the sale for an extra month and consistently achieved 100k+ sales for an additional three weeks with no loss of profit."

Or, to keep within the weather theme: "Rainfall in the south and east was above average throughout June due to a series of Atlantic storms which passed over continental Europe; in previous years these storms have tracked much further south."

Insight is about using the data to tell you about something that's happened before, and what happened next.  For example, we don't watch the weather channel to see what the weather was like yesterday or earlier this morning.  We may watch the weather channel to see what the weather's like now in another part of the country (or the world), but more often we want to know what the weather's going to be like tomorrow.  Good analysis will enable you to generate insights, extrapolate data and forecast future performance.

The regions of the UK Shipping Forecast, for which the BBC produces regular weather forecasts.

Should I buy (or pack) an umbrella or sunblock?  Which way do I point my windmill?  How do I trim my sails?  Do I go fishing tonight or wait until dawn?  When do I gather my crops?  How you use the data and then generate the insight depends on the audience.  This is life or death for some people.

Online, there's a clearer connection between actions and consequences - if you increase your online advertising spend, you should see more traffic coming to your site (and if you don't, start analysing and find out why, and what you should do about it).  With the weather - you can't make it rain, but you can work out why it rained (or didn't), when it's going to rain again (because you know it will), and what steps to take in order to make the best of the weather.  If you work in a team or a situation where the brand, marketing and advertising decisions for the online channel are made by an offline team with TV, radio and press expertise, you may find yourself in this kind of situation:  do not despair!

Wisdom

Some insights can be demonstrated repeatedly, and described succinctly so that they eventually become gathered wisdom:

"The north wind doth blow, and we shall have snow."
"Red sky at night, shepherd's delight; red sky at morning, shepherd's warning"

In online marketing, it could be something like, "Always show the discounted price in red", (honestly, I wrote that before I discovered that somebody genuinely thought it was a good idea) or "Never show a banner with two different products" (I'm making this stuff up).

No amount of data will automatically produce wisdom.  Big data (however big that might mean) will not spontaneously transform into insight and wisdom when it reaches a critical mass, in the same way that no amount of charcoal will produce diamonds even though they're made of the same stuff.

Actionable

Data, analysis and insight are useful tools and worthwhile aims - providing that they are actionable.  In my examples, I've been talking about using weather data to inform decisions, such as whether to wear a sunhat or a raincoat.  In this case, the data on temperature and rainfall are critical.  In online analysis (or in any kind of data analysis) it's vital that the analysis and insight are focused on the key performance indicators - that's what will make it actionable.  Talking about traffic to the landing page or the product information page will be trivial unless you can connect that data point to a key data point which drives the business - such as conversion, margin or revenue.  When you gather the data which enables you to tie your analysis and insight to a KPI, then your insight is far more likely to be actionable (I say this as your recommendation may be profitable but not feasible).

"My analysis shows that if we direct traffic from the landing page to page B instead of page A, then we will see an increase in conversion because 65% of people who see page B add an item to cart, compared to 43% for page A."  You can almost hear the sound of the checkout ringing.

"If we change our call to action from 'Buy Now' to 'Find out more', then the click through rate will go up."  Yes... and then what?  The click-through rate is probably a good data point to start with, but how much will it go up by, and what will the impact be on the website's KPIs?

Conclusion 

  If data analysis (sales revenue, time, banner description and click-through rate) indicates that sales revenue drops when you mix your products in your banners, because people ignore the higher priced product and only buy the cheaper one, then this can move from data to analysis to insight to wisdom.  It may take repeated observations to get there (was it a one-off, does it apply to all products, does it only happen in summer?), but it shows how you can move from data to analysis to actionable insights.

Other articles I've written on Website Analytics that you may find relevant:

Web Analytics - Gathering Requirements from Stakeholders

Analysis is Easy, Interpretation Less So
Telling a Story with Web Analytics Data
What is "Direct Traffic"?
Reporting, Analysing, Testing and Forecasting
Pages with Zero Traffic

Telling a Story with Web Analytics Data

Management demands actionable insights - not just numbers, but KPIs, words, sentences and recommendations.  It's therefore essential that we, as web analysts and optimisers, are able to transform data into words - and better still, stories.  Consider a report with too much data and too little information - it reads like a science report, not a business readout:

Consider a report concerning four main characters;
Character A: female, aged 7 years old.  Approximately 1.3 metres tall.
Character B:  male, aged 5 years old.
Character C: female, aged 4 years old.
Character D:  male, aged 1 year old.

The main items in the report are a small cottage, a 1.2 kw electric cooker, 4 pints of water, 200 grams of dried cereal and a number of assorted iron and copper vessels, weighing 50-60 grams each.

After carrying out a combination of most of the water and dried cereal, and in conjunction with the largest of the copper vessels, Character B prepared a mixture which reached around 70 degrees Celsius.  He dispensed this unevenly into three of the smaller vessels in order to enable thermal equilibrium to be formed between the mixture and its surroundings.  Characters B, C and D then walked 1.25 miles in 30 minutes, averaging just over 4 km/h.  In the interim, Character A took some empirical measurements of the chemical mixture, finding Vessel 1 to still be at a temperature close to 60 degrees Celsius, Vessel 2 to be at 70 degrees Fahrenheit and Vessel 3 to be at 315 Kelvin, which she declares to be optimal.

The report continues with Character A consuming all of the mixture in Vessel 3, then single-handedly testing (in some case destruction testing) much of the furniture in the small cottage.

The problem is:  there's too much data and not enough information. 

The information is presented in various formats - lists, sentences and narrative.


Some of it the data is completely irrelevant (the height of Character A, for example)
Some of it is misleading (the ages of the other characters lacks context);
Some of it is presented in a mish-mash of units (temperatures are stated four times, with three different units).
The calculation of the speed of the walking characters is not clear - the distance is given in miles; the time is given in minutes; and the speed in kilometres per hour (if you are familiar with the abbreviation km/h).

Of course, this is an exaggeration, and as web analytics professionals, we wouldn't do this kind of thing in our reporting. 

Visitors are called visitors, and we consistently refer to them as visitors (and we ensure that this definition is understood among our readers)
Conversion rates are based on visitors, even though this may require extra calculation since our tools provide figures based on visits (or sessions)
Percentage of traffic coming from search is quoted as visitors (not called users), and not visits (whether you use visitors or visits is up to you, but be consistent)
Would you include number of users who use search?  And the conversion rate for users of search?
And when you say 'Conversion', are you consistently talking about 'user added an item to cart', or 'user completed a purchase and saw the thank-you page'?
Are you talking about the most important metrics?
 
So - make sure, for starters, that your units and data and KPIs are consistent, contextual, or at least make sense. And then:  add the words to the numbers.  It's only the start to say that: "We attracted 500 visitors with paid search, at a total cost of £1,200."  Go on to talk about the cost per visitor, break it down into key details by talking about the most expensive keywords, and the ones that drove the most traffic.  But then tell the story:  there's a sequence of events between user seeing your search term, clicking on your ad, visiting your site, and [hopefully] converting.  Break it down into chronological steps and tell the story!

There are various ways to ensure that you're telling the story; my favourites are to answer these types of questions:
"You say that metric X has increased by 5%.  Is that a lot?  Is that good?"
 "WHY has this metric gone up?"
"What happened to our key site performance indicators (profit, revenue, conversion) as a result?"
and my favourite:
"What should we do about it?"

There are, of course, various ways to hide the story, or disguise results that are not good (i.e. do not meet sales or revenue targets) - I did this in my anecdote at the start. However, management tend to start looking at incomplete data, or data that's obscure or irrelevant, and go on to ask about the data that's "missing"... so the truth will out, so it's better to show the data, tell the whole story, and highlight why things are below par. 

It's our role to highlight when performance is down - we should be presenting the issues (nobody else has the tools to do so) and then going on to explain what needs to be done - this is where actionable insights become invaluable.  In the end, we present the results and the recommendations and then let the management make the decision - I blogged about this some time ago - web analytics: who holds the steering wheel?

In the case of Characters A, B, C and D, I suggest that Characters B and C buy a microwave oven, and improve their security to prevent Character A from breaking into their house and stealing their breakfast.  In the case of your site, you'll need to use the data to tell the story.

Other articles I've written on Website Analytics that you may find relevant:

Web Analytics - Gathering Requirements from Stakeholders

Analysis is Easy, Interpretation Less So
Telling a Story with Web Analytics Data
What is "Direct Traffic"?
Reporting, Analysing, Testing and Forecasting
Pages with Zero Traffic

Web Analytics - A Medical Emergency

One of my favourite TV programmes at the moment is Casualty.  Or perhaps it's Holby City (they're both the same, really).  A typical episode unfolds with all the drama and angst between the main characters, which is suddenly broken up by the paramedics unloading a patient from an ambulance.  Perhaps the patient is the victim of a fire, or a road traffic accident, or another emergency.  Whatever it is, the paramedics come in, wheeling the patient along, giving a brief description of who they've got, the main symptoms, and start rattling off a list of numbers.  "Patient is RTA victim, aged 56, BP is 100 over 50, pulse is 58 and weak, 100 mls of adrenaline given..." the list goes on.  The senior consultant who is receiving the patient hasn't really got time to be asking questions like, "Is that bad?" and certainly not, "Is this important?"  The questions he's already asking himself are, "What can we do to help this patient?" and "What's been done already?"

Regular readers will already know where I'm going with this analogy, so I'll try to keep it brief.  In a life-or-death situation (and no, web analysts are hardly ever going to have that degree of responsiblity) there isn't really time to start asking and answering the trivial questions.  The executive dashboard, the report or the update need to state what the results are at the moment, and how this looks against target, normal or threshold, and what action needs to be taken.  The executive, in a similar way to the Formula 1 driver I mentioned last time, hasn't got time to look through all the data, decide what's important and what isn't, and what needs to be looked at.

As an aside, I should comment that reporting dying figures to an executive is likely to lead to a series of questions back to the analyst, so be ready to answer them.  Better still, including a commentary that states the reasons for a change in the figures and the action that's being taken to address them.  Otherwise, all you'll achieve is an unfortunate way of generating action from the content team, who probably won't be too pleased to receive a call from a member of the executive team, asking why their figures are dying, and will want to know why you didn't tell them first.

Another skill comes in determining the key figures to report - the vital statistics.  The paramedics know that time is of the essence and keep it deliberately brief and to the point.  No waffle.  Clear.  The thresholds for each KPI are already understood - after all, they have the advantage that all medical staff know what typical temperature, pulse, blood pressure and blood sugar levels are.  As a web analyst (or a business analyst), you'll need to gain agreement from your stakeholders on what these are.  Otherwise you may find yourself reporting the height and weight of a patient who has severe blood loss, where the metrics are meaningless and don't reflect the current situation.  If you give a number for a KPI, and the reply is, "Is that a lot?" then you have some work to do - and I have some answers for you too.


Now, all I've covered so far is the reporting - the paramedics' role.  If we were (or are) web reporters, then that would be the sum of our role: to look at the site, take the measurements, blurt out all the relevant figures and then go back to our desks.  However, as web analysts, we now need to take on the role of the medical consultant, and start looking at the stats - the raw data - and working out why they're too high (or too low), and most importantly, what to do about them.  Could you imagine the situation where the consultant identifies the cause of the problem - say an infection in the lungs - and goes over to the patient, saying, "That's fine Mr Smith, we have found the cause of your breathlessness.  It's just a bacterial infection in your left lung."  There would then be a hesitant pause, until the patient says something like, "Can you treat it?" or "What can you do for me?".  

Good web analysts go beyond the reporting, through to identifying the cause of any problems (or, if your patient is in good health, the potential for improvements) and then working out what can be done to improve them.  This takes time, and skill, and a good grasp of the web analytics tool you're using.  You may have to look at your website too - actually look at the pages and see what's going on.  Look at the link text; the calls to action; read the copy, and study the images.  Compare this with the data you've obtained from your analytics tools.  This may not provide all the answers, so you may have to persevere.  Go on to look at traffic sources - the referrers, the keywords, the campaign codes.  Track down the source of the problem - or the likely causes - and follow the data to its conclusion, even if it takes you outside your site to a search engine and you start trying various keywords in Google to see how your site ranks, and what your PPC actually looks like.


Checking pages on a site is just the equivalent of a doctor actually looking at his patient.  He may study the screens and take a pulse and measure blood pressure or take the patient's temperature, but unless he actually looks at the patient - the patient's general appearance, any wounds, scars, marks, rashes or whatever else - he'll be guessing in the dark.  This isn't House (another medical drama that I never really took to), this is real medicine.  Similarly, doctors may consider environmental factors - what has the patient eaten, drunk, inhaled, come into contact with?  What's going on outside the body that might affect something inside it?

There's plenty of debate about the difference between reporting and analysis - in fact I've commented on this before - but I think the easiest answer I could give now is the difference between the paramedic and the doctor.  What do you think?

Web Analytics: Reporting, Analysing, Testing and Forecasting

Reporting, Forecasting, Testing and Analysing

As a website analyst, my working role means that I will take a large amount of online data, sift it, understand it, sift it again and search for the underlying patterns, changes, trends and reasons for those changes.  It's a bit like panning for gold.  In the same way as there's a small amount of gold in plenty of river, there's plenty of data to look at, it's just a case of finding what's relevant, useful and important, and then telling people about it.  Waving it under people's noses; e-mailing it round; printing it out and so on – that’s reporting.

However, if all I do with the data I have is report it, then all I'm probably doing is something similar to reporting the weather for yesterday.  I can make many, many different measurements about the weather for yesterday, using various instruments, and then report the results of my measurements.  I can report the maximum temperature; the minimum temperature; the amount of cloud coverage in the sky; the rainfall; the wind speed and direction and the sunrise and sunset times.  But, will any of this help me to answer the question, "What will the weather be like tomorrow?" or is it just data?  Perhaps I'll look at the weather the day before, and the day before that.  Are there trends in any of the data?  Is the temperature rising?  Is the cloud cover decreasing?  In this way, I might be able to spot trends or patterns in the data that would lead me to conclude that yes, tomorrow is likely to be warmer and clearer than yesterday.  Already my reporting is moving towards something more useful, namely forecasting.

The key difference between the weather and online data, hopefully, is that when we come to analyse business data (marketing data or web data), I'm in a position to change the inputs of today’s data.  I can't do much today with my measurements to influence tomorrow's weather, but online I can change my website’s content, text, layout or whatever, and hopefully make some changes to my online performance.  No amount of measuring or reporting is going to change anything – not the weather, not sales performance.  Only changes to the site will lead to changes to the results.  Then, I can not only forecast tomorrow's online performance, but also make changes to try to improve it.

No change means that there's no way to determine what works and what doesn't.  I've been asked to try and determine, "What does good look like?" but unless I make some guesses at what good might be, and test them out on the website, I'll never know.  What I should be able to do, though, is forecast what future performance will look like - this is the advantage of having a website that doesn't change much.  Providing most of the external factors (for example traffic sources, marketing spend, product pricing) stay the same, I should be able to forecast what performance will be like next week.  Unfortunately, the external factors rarely stay the same, which will make forecasting tricky - but it'll be easier than forecasting performance for a changing website!

Consider the following situation:  here's my online promotion, and I've simplified it (I've removed the text, and really simplified it) and I've reduced it to a colour and a shape.  So I launch my campaign with Red Triangle, and measurements show that it is worth 500 points per day (I'm not discussing whether that's clicks, sales, quotes, telephone calls or what - it's a success metric and I've scored it 500 points per day).

       500 points per day

If I make no changes to the promotion, then I'll keep using Red Triangle, and theoretically it'll keep scoring 500 points each day.  However, I might change it to something else, for example, I might test Green Circle. 

300 points per day

Now, Green Circle scores 300 points per day, over a week.  Is that good?  Well, Red Triangle scored 500 points per day, so you might think it'd be worth changing it back.  There's a barrier here, in that if I do change it back to Red Triangle, I have to admit that I made a mistake, and that my ideas weren't as good as I thought they were.  Perhaps I'll decide that I can't face going back to Red Triangle, and I'll try Blue Square instead.
 

 


 200 points per day


But what if Blue Square scores only 200 points each day?  Do I keep running it until I'm sure it's not as good, or do I carry out a test of statistical significance?  Perhaps it'll recover?  One thing is for sure; I know what good looks like (it's a Red Triangle at the moment) but I'll have to admit that my two subsequent versions weren't as good; this is a real mental shift - after all, doesn't optimising something mean making it better?  No, it's not scientific and I should probably start testing Red Circles and Green Triangles, but based on the results I've actually obtained, Red Triangle is the best. 

Maybe I shouldn't have done any testing at all.  After all, Green Circle would cost me 200 points per day, and Blue Square costs me 300 points per day.  And I've had to spend time developing the creative and the text - twice.

Now, I firmly believe that testing is valuable in and of itself.  I’m a scientist, with a strong scientific background, and I know how important testing has been, and will continue to be, to the development of science.  However, one of the major benefits of online marketing and sales is that it's comparatively easy to swap and change - to carry out tests and to learn quickly.  It’s not like changing hundreds of advertising posters at bus stops up and down the country – it’s simply a case of publishing new content on the site.  Even sequential tests (instead of A/B tests) like my example above with the coloured shapes, will provide learning.  What's imperative, though, is that the learning is carried forwards.  Having discovered that Red Triangle is the best of the three shapes tried so far, I would not start the next campaign with a variation of Blue Square.  Learning must be remembered, not forgotten.

Having carried out tests like this, it becomes possible to analyse the results.  I’ve done the measuring and reporting, and it looks like this:  Red Triangle = 500 pts/day, Green Circle = 300 pts/day, Blue Square = 200 pts/day.

Analysing the data is the next step.  In this case, there genuinely isn’t much data to analyse, so I would recommend more testing.  I would certainly recommend against Green Circle and Blue Square, and would propose testing Yellow Triangle instead, to see if it’s possible to improve on Red Triangle’s performance.  It all sounds so easy, and I know it isn’t, especially when there’s a development cycle to build Yellow Triangle, when Green Circle is already on the shelf, and Blue Square is already up and running.  However, that’s my role – to review and analyse the data and recommend action.  There are occasions when there are other practical reasons for not following the data, and flexibility is key here.

In fact, for me, I’m always looking at what to do on the website next – the nugget of gold which is often a single sentence that says, “This worked better than that, therefore I recommend this…” or “I recommend doing this, because it provided an uplift of 100 points per day".  That’s always the aim, and the challenge, when I’m analysing data.  Otherwise, why analyse?  My role isn’t to report yesterday’s weather.  At the very least, I’m looking to provide a forecast for tomorrow’s weather, and ideally, I’d be able to recommend if an umbrella will be needed tomorrow afternoon, or sun tan lotion.  Beyond that, I’d also like to be able to suggest where to seed the clouds to make it rain, too!