Wednesday, 30 January 2013

Testing: Iterating or Creating?

"Let's run a test!" comes the instruction from senior management.  Let's improve this page's performance, let's make things better, let's try something completely new, let's make a small change...  let's do it like Amazon or eBay.  Let's run an A/B test.

In a future post, I'll cover where to test, what to test, and what to look for, but in this post, I'd like to cover how to test.  Are you going to test totally new page designs, or just minor changes to copy, text, calls-to-action and pictures?  Which is best?

It depends.  If you're under pressure to show an improvement in performance with your tests, such as fixing a broken sales funnel, then you are probably best testing small, steady changes to a page in a careful, logical and thoughtful way.  Otherwise, you risk seriously damaging your financial performance while the test is running, and not achieving a successful, positive result.  By making smaller changes in your test recipes, you are more likely to get performance that's closer to the original recipe - and if your plan and design were sound, then it should also be an improvement :-)



If you have less pressure on improving performance, and iterating seems irritating, then you have the opportunity to take a larger leap into the unknown - with the increased risk that comes with it.  Depending on your organisation, you may find that there's pressure from senior management to test a completely new design and get positive results (the situation worsens when they expect to get positive results with their own design which features no thought to prior learnings).  "Here, I like this, test it, it should win."  At least they're asking you to test it first, instead of just asking you to implement it.

Here, there's little thought to creating a hypothesis, or even iterating, and it's all about creating a new design - taking a large leap into the unknown, with increased risk.  Yes, you may hit a winner and see a huge uplift from changing all those page elements; painting the site green and including pictures of the products instead of lifestyle images, but you may just find that performance plummets.  It's a real leap into the unknown!


The diagram above represents the idea behind iterative and creative testing.  In iterative testing (the red line), each test builds on the ideas that have been identified and tested previously.  Results are analysed, recommendations are drawn up and then followed, and each test makes small but definite improvements on the previous.  There's slow but steady progress, and performance improves with time.

The blue line represents the climber jumping off his red line and out into the unknown.  There are a number of possible results here, but I've highlighted two.  Firstly the new test, with the completely untested design, performs very badly, and our climber almost falls off the mountain completely.  Financial performance is poor compared to the previous version, and is not suitable for implementation.  It may be possible to gain useful learnings from the results (and this may be more than, "Don't try this again!") but this will take considerable and careful analysis of the results.

Alternatively, your test result may accelerate you to improved performance and the potential for even better results - the second blue climber who has reached new heights.  It's worth pointing out at this stage that you should analyse the test results as carefully as if it had lost.  Otherwise, your win will remain an unknown and your next test may still be a disaster (even if it's similar to the new winner).  Look at where people clicked, what they saw, what they bought, and so on.  Just because your creative and innovative design won doesn't mean you're off the hook - you still need to work out why you won, just as carefully as if you'd lost.

So, are you iterating or creating?  Are you under pressure to test out a new design?  Are you able to make small improvements and show ROI?  What does your testing program look like - and have you even thought about it?

Tuesday, 29 January 2013

What is Direct Traffic?

It is a pain. It is anonymous and it is often a sizeable proportion of your traffic.  Yet the received wisdom and standard description is that it is traffic which came from users typing in your URL directly in their browser.  What is it? "Direct traffic".

Direct Traffic:  What it Isn't

It isn't search, and it can't be confirmed as traffic clicking a link from another website. It almost counts as a miscellaneous bucket. Here I would like to summarise what it might be, what it isn't and how to improve reporting for it.  Is it as good as Avinash Kaushik says, when he says that direct traffic is a good thing and we should look to improve it?  I have found, in my experience, that it's not.  If direct traffic was traffic without a referring URL and I could be assured that it was people typing in my URL in the browser bar, then I'd be happy.  But my experience shows otherwise - and historically it's been a matter of some consternation as I look to find out what it really is.

If it were the result of the five factors that Avinash suggests, then I'd be very pleased.  Here they are:

    1. People who are your existing customers / past purchasers, they'll type url and come to the site or via bookmarks.
    2. People familiar with your brand. They need a solution and your name pops up into their head and they type.
    3. People driven by word of mouth. Someone recommends your business / solution to someone else and boom they show up at the site. Uninvited, but we love them!
    4. People driven by your offline campaigns. Saw an ad on TV, heard one on radio, saw a billboard and were motivated enough to typed the url and show up.
    5. Free, non-campaign, traffic.

Direct Traffic:  What It Often Is

However a better definition of direct traffic is 'referrer not known' or 'referrer information lost'.  It can be lost in a number of ways, but the main ones are javascript redirects, or redirects that go via an ad agency's server.  As a website analyst, I was often asked to track online marketing campaigns that went live 'last Monday', and sure, there was a spike in traffic, which came from adagency.com (insert your ad agency here), because the ad agency were also tracking impressions and clicks, and kindly stamped themselves all over the traffic source. 

"Yes we've seen an increase in traffic.  Yes, it matches the timing of your online campaign.  No, I can't tell you where it came from.  No, I don't have the telephone number for your online agency.  Do you?"

Other examples are flash applications, documents (such as Word or Excel documents that have links in them), and some automated traffic, like spiders or bots (and not the 'good' ones, which don't process tags.

The way around it?

If you always ensure that your inbound campaign links are passing campaign or source information, or both, through URL query strings (whether that's ?cid=online, or ?utm= or ?marketing=online) then you will still be able to capture the referring site information, even if the brower isn't passing it.  Google Analytics has a semi-automatic process that will allow you to build your own campaign URLs.


You won't usually find "Direct Traffic" in a promotional or merchandising
screenshot for a tool... it probably means the tool hasn't classed the traffic
as anything else.
Alternatively, you can start to look at the visitor data for the direct traffic.  Previously, I have used Adobe's Discover tool to isolate "direct" traffic and start segmenting it.  Where, geographically, does it come from?  For example, does the traffic increase match an offline campaign in London?  Were people really typing in your URL?

No?  How about drilling down even further?  I have found cases where direct traffic was actually an automated checking tool - an example would be Gomez - which was pinging the site every 15 minutes, without fail, day and night, and always from the same IP addresses. 

It took some detective work (start by Googling the IP address) to track an IP address back to the performance monitoring tool, but once it was done, it was easy enough to block the IP - after making sure that we weren't paying the tool providers.  Alternatively, if you are, and you do see their traffic in your reports, there are ways to screen them out from your analytics (without blocking them from the page).



The results?  Direct traffic went down.   Total traffic went down.  So far, so good for site targets.  However, we did some beneficial results, which supported another piece of work I was doing:  reducing bounce rate.  The automated traffic was bouncing off various pages of the site (in particular, the home page) at a constant, relentless and nagging rate.  Once we blocked the automated traffic that we were sure came from the performance tracking tool, the bounce rate fell.  Dramatically.

So, I reduced the volume of direct traffic - but I improved reporting quality and traffic quality.  If you're looking to improve direct traffic performance (but not quantity ;-) then I suggest the steps I took.  This will help you to improve your conversion figures for the segment, and overall, and that's got to be a good thing.


Thursday, 17 January 2013

Why I Like Snow

 I'm writing this on a wintry evening, with an almost-certain forecast of heavy snow for tomorrow.  I've mentioned this a few times on Facebook, and posted screenshots of weather forecasts from the Met Office and the BBC Weather site to share my excitement at the prospect of a large fall of snow tomorrow.  Some of my friends share my love of snow.  Some don't.  In fact, some positively dislike it; I expect, given the opportunity, they'd shake their fists at the sky and disapprovingly wave their fingers at the clouds at the first suggestion that there may be a few flakes falling in the near future.


The media often report snow in terms of traffic delays, travel disruption and so on.  Consequently, many people have a negative view of snow (and its apparent 'partner in crime' - ice).  Might I suggest that if you have a negative view of snow for this reason, that you allow extra time for your journey, give up on the idea of driving at normal speeds and pay attention to the change in road conditions?  Leave extra space between you and the car in front, drive carefully, accelerate steadily, brake smoothly and steer a little less dramatically?  If we all did this, then we'd all travel more slowly but we'd all arrive safely, and with fewer accidents.  

So, having addressed the main negative reaction to the snow and the disruption it causes, I would like to turn to the questions, "Why do you like snow?  What's the obsession?  What's so good about snow anyway?"

Here's my reply.




1.  It's photogenic.  Very photogenic.  During the annual time of long, dark nights, with overcast and grey days; lifeless and leafless trees, and general dullness, the arrival of snow heralds a widespread brightening and improvement in the landscape.  Even on a cloudy day, snow can brighten the landscape considerably.

2.  It's fun.  Sledging (which I only discovered a couple of years ago); snowballs, snowmen...  sometimes there's no point trying to take snow seriously.

3.  And this is the most important to me:  it's a great visual reminder of some Biblical truths.  In order from 'resonates with me a bit' through to 'hits home every time I see snow', here are some Scriptural principles and the Bible verses that I see when the snow falls.

a)  Snow is a good description of what angels look like.  Matthew 28 reads:

"After the Sabbath, at dawn on the first day of the week, Mary Magdalene and the other Mary went to look at the tomb.  There was a violent earthquake, for an angel of the Lord came down from heaven and, going to the tomb, rolled back the stone and sat on it. His appearance was like lightning, and his clothes were white as snow."

On a sunny day, just a glance at snow can be dazzling - that's why skiers wear sunglasses (or ski goggles) even in winter.  What do angels look like?  Answer:  they wear clothes that are as bright and white as snow.

b)  Snow comes down from heaven, and is a reminder that God keeps his promises.  Isaiah 55:10,11

"As the rain and the snow come down from heaven, and do not return to it without watering the earth and making it bud and flourish, so that it yields seed for the sower and bread for the eater, so is my word that goes out from my mouth:  It will not return to me empty, but will accomplish what I desire and achieve the purpose for which I sent it."

God set the rainbow in the sky as a reminder that he keeps his promises (in particular, He's not going to flood the whole world again), and sends rain and snow to remind us that His word, which he also sends from heaven.  Snow doesn't just evaporate its way back to the clouds.  It waters the earth, and makes things grow.  So it is with God's word - when He sends it, it fulfils its purpose.

c)  This is my most favoured one; there are plenty more references to snow in the Bible (and none of them, by the way, are negative), but this one is one that resonates most strongly with me - it hits me where I live.  Isaiah 1:18 says:

“Come now, let us settle the matter,” says the Lord.  “Though your sins are like scarlet, they shall be as white as snow; though they are red as crimson, they shall be like wool."

I'm not perfect.  In fact, if you were to look at me as a weather forecast, it would be dull and gloomy with black clouds.  However, in this verse, God invites Isaiah to sit down and talk.  God knows that Isaiah isn't perfect - in fact his record is stained with scarlet and crimson, which are deep, dark dyes - but God says he will wash them out and Isaiah's sins - his mistakes and wrongdoings - will be as white as snow.  Pure, clean and white.  God washes whiter than Persil, and puts Isaiah (and me) in a position where he's able to sit down and reason with God.  It's an invitation to talk, where Isaiah is as white as snow... which is as white as the angels.

So, why do I like snow?  Because it reminds me that with my sins forgiven, I'm as white as snow.  And it's fun, bright and cheerful, and makes for great photographs.