Testing: Iterating or Creating?

"Let's run a test!" comes the instruction from senior management.  Let's improve this page's performance, let's make things better, let's try something completely new, let's make a small change...  let's do it like Amazon or eBay.  Let's run an A/B test.

In a future post, I'll cover where to test, what to test, and what to look for, but in this post, I'd like to cover how to test.  Are you going to test totally new page designs, or just minor changes to copy, text, calls-to-action and pictures?  Which is best?

It depends.  If you're under pressure to show an improvement in performance with your tests, such as fixing a broken sales funnel, then you are probably best testing small, steady changes to a page in a careful, logical and thoughtful way.  Otherwise, you risk seriously damaging your financial performance while the test is running, and not achieving a successful, positive result.  By making smaller changes in your test recipes, you are more likely to get performance that's closer to the original recipe - and if your plan and design were sound, then it should also be an improvement :-)

If you have less pressure on improving performance, and iterating seems irritating, then you have the opportunity to take a larger leap into the unknown - with the increased risk that comes with it.  Depending on your organisation, you may find that there's pressure from senior management to test a completely new design and get positive results (the situation worsens when they expect to get positive results with their own design which features no thought to prior learnings).  "Here, I like this, test it, it should win."  At least they're asking you to test it first, instead of just asking you to implement it.

Here, there's little thought to creating a hypothesis, or even iterating, and it's all about creating a new design - taking a large leap into the unknown, with increased risk.  Yes, you may hit a winner and see a huge uplift from changing all those page elements; painting the site green and including pictures of the products instead of lifestyle images, but you may just find that performance plummets.  It's a real leap into the unknown!

The diagram above represents the idea behind iterative and creative testing.  In iterative testing (the red line), each test builds on the ideas that have been identified and tested previously.  Results are analysed, recommendations are drawn up and then followed, and each test makes small but definite improvements on the previous.  There's slow but steady progress, and performance improves with time.

The blue line represents the climber jumping off his red line and out into the unknown.  There are a number of possible results here, but I've highlighted two.  Firstly the new test, with the completely untested design, performs very badly, and our climber almost falls off the mountain completely.  Financial performance is poor compared to the previous version, and is not suitable for implementation.  It may be possible to gain useful learnings from the results (and this may be more than, "Don't try this again!") but this will take considerable and careful analysis of the results.

Alternatively, your test result may accelerate you to improved performance and the potential for even better results - the second blue climber who has reached new heights.  It's worth pointing out at this stage that you should analyse the test results as carefully as if it had lost.  Otherwise, your win will remain an unknown and your next test may still be a disaster (even if it's similar to the new winner).  Look at where people clicked, what they saw, what they bought, and so on.  Just because your creative and innovative design won doesn't mean you're off the hook - you still need to work out why you won, just as carefully as if you'd lost.

So, are you iterating or creating?  Are you under pressure to test out a new design?  Are you able to make small improvements and show ROI?  What does your testing program look like - and have you even thought about it?

What is Direct Traffic?

It is a pain. It is anonymous and it is often a sizeable proportion of your traffic.  Yet the received wisdom and standard description is that it is traffic which came from users typing in your URL directly in their browser.  What is it? "Direct traffic".

Direct Traffic:  What it Isn't

It isn't search, and it can't be confirmed as traffic clicking a link from another website. It almost counts as a miscellaneous bucket. Here I would like to summarise what it might be, what it isn't and how to improve reporting for it.  Is it as good as Avinash Kaushik says, when he says that direct traffic is a good thing and we should look to improve it?  I have found, in my experience, that it's not.  If direct traffic was traffic without a referring URL and I could be assured that it was people typing in my URL in the browser bar, then I'd be happy.  But my experience shows otherwise - and historically it's been a matter of some consternation as I look to find out what it really is.

If it were the result of the five factors that Avinash suggests, then I'd be very pleased.  Here they are:

1. People who are your existing customers / past purchasers, they'll type url and come to the site or via bookmarks.
2. People familiar with your brand. They need a solution and your name pops up into their head and they type.
3. People driven by word of mouth. Someone recommends your business / solution to someone else and boom they show up at the site. Uninvited, but we love them!
4. People driven by your offline campaigns. Saw an ad on TV, heard one on radio, saw a billboard and were motivated enough to typed the url and show up.
5. Free, non-campaign, traffic.

Direct Traffic:  What It Often Is

However a better definition of direct traffic is 'referrer not known' or 'referrer information lost'.  It can be lost in a number of ways, but the main ones are javascript redirects, or redirects that go via an ad agency's server.  As a website analyst, I was often asked to track online marketing campaigns that went live 'last Monday', and sure, there was a spike in traffic, which came from adagency.com (insert your ad agency here), because the ad agency were also tracking impressions and clicks, and kindly stamped themselves all over the traffic source. 

"Yes we've seen an increase in traffic.  Yes, it matches the timing of your online campaign.  No, I can't tell you where it came from.  No, I don't have the telephone number for your online agency.  Do you?"

Other examples are flash applications, documents (such as Word or Excel documents that have links in them), and some automated traffic, like spiders or bots (and not the 'good' ones, which don't process tags.

The way around it?

If you always ensure that your inbound campaign links are passing campaign or source information, or both, through URL query strings (whether that's ?cid=online, or ?utm= or ?marketing=online) then you will still be able to capture the referring site information, even if the brower isn't passing it.  Google Analytics has a semi-automatic process that will allow you to build your own campaign URLs.

You won't usually find "Direct Traffic" in a promotional or merchandising
screenshot for a tool... it probably means the tool hasn't classed the traffic
as anything else.

Alternatively, you can start to look at the visitor data for the direct traffic.  Previously, I have used Adobe's Discover tool to isolate "direct" traffic and start segmenting it.  Where, geographically, does it come from?  For example, does the traffic increase match an offline campaign in London?  Were people really typing in your URL?

No?  How about drilling down even further?  I have found cases where direct traffic was actually an automated checking tool - an example would be Gomez - which was pinging the site every 15 minutes, without fail, day and night, and always from the same IP addresses. 

It took some detective work (start by Googling the IP address) to track an IP address back to the performance monitoring tool, but once it was done, it was easy enough to block the IP - after making sure that we weren't paying the tool providers.  Alternatively, if you are, and you do see their traffic in your reports, there are ways to screen them out from your analytics (without blocking them from the page).

The results?  Direct traffic went down.   Total traffic went down.  So far, so good for site targets.  However, we did some beneficial results, which supported another piece of work I was doing:  reducing bounce rate.  The automated traffic was bouncing off various pages of the site (in particular, the home page) at a constant, relentless and nagging rate.  Once we blocked the automated traffic that we were sure came from the performance tracking tool, the bounce rate fell.  Dramatically.

So, I reduced the volume of direct traffic - but I improved reporting quality and traffic quality.  If you're looking to improve direct traffic performance (but not quantity ;-) then I suggest the steps I took.  This will help you to improve your conversion figures for the segment, and overall, and that's got to be a good thing.

Other articles I've written on Website Analytics that you may find relevant:

Web Analytics - Gathering Requirements from Stakeholders

Analysis is Easy, Interpretation Less So
Telling a Story with Web Analytics Data
What is "Direct Traffic"?
Reporting, Analysing, Testing and Forecasting
Pages with Zero Traffic

Why I Like Snow

 I'm writing this on a wintry evening, with an almost-certain forecast of heavy snow for tomorrow.  I've mentioned this a few times on Facebook, and posted screenshots of weather forecasts from the Met Office and the BBC Weather site to share my excitement at the prospect of a large fall of snow tomorrow.  Some of my friends share my love of snow.  Some don't.  In fact, some positively dislike it; I expect, given the opportunity, they'd shake their fists at the sky and disapprovingly wave their fingers at the clouds at the first suggestion that there may be a few flakes falling in the near future.

The media often report snow in terms of traffic delays, travel disruption and so on.  Consequently, many people have a negative view of snow (and its apparent 'partner in crime' - ice).  Might I suggest that if you have a negative view of snow for this reason, that you allow extra time for your journey, give up on the idea of driving at normal speeds and pay attention to the change in road conditions?  Leave extra space between you and the car in front, drive carefully, accelerate steadily, brake smoothly and steer a little less dramatically?  If we all did this, then we'd all travel more slowly but we'd all arrive safely, and with fewer accidents.  

So, having addressed the main negative reaction to the snow and the disruption it causes, I would like to turn to the questions, "Why do you like snow?  What's the obsession?  What's so good about snow anyway?"

Here's my reply.

1.  It's photogenic.  Very photogenic.  During the annual time of long, dark nights, with overcast and grey days; lifeless and leafless trees, and general dullness, the arrival of snow heralds a widespread brightening and improvement in the landscape.  Even on a cloudy day, snow can brighten the landscape considerably.

2.  It's fun.  Sledging (which I only discovered a couple of years ago); snowballs, snowmen...  sometimes there's no point trying to take snow seriously.

3.  And this is the most important to me:  it's a great visual reminder of some Biblical truths.  In order from 'resonates with me a bit' through to 'hits home every time I see snow', here are some Scriptural principles and the Bible verses that I see when the snow falls.

a)  Snow is a good description of what angels look like.  Matthew 28 reads:

"After the Sabbath, at dawn on the first day of the week, Mary Magdalene and the other Mary went to look at the tomb.  There was a violent earthquake, for an angel of the Lord came down from heaven and, going to the tomb, rolled back the stone and sat on it. His appearance was like lightning, and his clothes were white as snow."

On a sunny day, just a glance at snow can be dazzling - that's why skiers wear sunglasses (or ski goggles) even in winter.  What do angels look like?  Answer:  they wear clothes that are as bright and white as snow.

b)  Snow comes down from heaven, and is a reminder that God keeps his promises.  Isaiah 55:10,11

"As the rain and the snow come down from heaven, and do not return to it without watering the earth and making it bud and flourish, so that it yields seed for the sower and bread for the eater, so is my word that goes out from my mouth:  It will not return to me empty, but will accomplish what I desire and achieve the purpose for which I sent it."

God set the rainbow in the sky as a reminder that he keeps his promises (in particular, He's not going to flood the whole world again), and sends rain and snow to remind us that His word, which he also sends from heaven.  Snow doesn't just evaporate its way back to the clouds.  It waters the earth, and makes things grow.  So it is with God's word - when He sends it, it fulfils its purpose.

c)  This is my most favoured one; there are plenty more references to snow in the Bible (and none of them, by the way, are negative), but this one is one that resonates most strongly with me - it hits me where I live.  Isaiah 1:18 says:

“Come now, let us settle the matter,” says the Lord.  “Though your sins are like scarlet, they shall be as white as snow; though they are red as crimson, they shall be like wool."

I'm not perfect.  In fact, if you were to look at me as a weather forecast, it would be dull and gloomy with black clouds.  However, in this verse, God invites Isaiah to sit down and talk.  God knows that Isaiah isn't perfect - in fact his record is stained with scarlet and crimson, which are deep, dark dyes - but God says he will wash them out and Isaiah's sins - his mistakes and wrongdoings - will be as white as snow.  Pure, clean and white.  God washes whiter than Persil, and puts Isaiah (and me) in a position where he's able to sit down and reason with God.  It's an invitation to talk, where Isaiah is as white as snow... which is as white as the angels.

So, why do I like snow?  Because it reminds me that with my sins forgiven, I'm as white as snow.  And it's fun, bright and cheerful, and makes for great photographs.

Other articles I've written based on Biblical principles

10 things I learned from not quite reading the Bible in a year
Advent and a Trip to London
Advent: Names and Titles
Reading Matthew 1
My reading of Matthew 2
The Parable of the 99 Sheep

Doctor Who Asylum of the Daleks

"Well come on then.  You've got me.  At long last.  It's Christmas!  Here I am!"
The Doctor announces his return.

After the longest gap that we've had to wait since the relaunch of Doctor Who in 2005, the Doctor has returned, for a five-episode run, and then a Christmas special.  I'll try to review each episode in turn, and I will not feature spoilers of future episodes, and I won't even trail the next episode in my reviews.  I will also not refer to any of the other speculation, news and comment about Doctor Who that features in the media.  I won't talk about expected character deaths or new companions - or anything else.  I'll take each episode at face value and comment on the 50 minutes of television.

And so series seven of Doctor Who begins, and it opens with the Daleks.  Cue the sighing and cheering in equal measure - an entire episode devoted almost entirely to the one-eyed exterminators.

However, I have to say that this episode is one of the very best Dalek episodes - in my opinion.

The episode begins on the Dalek's home planet of Skaro, which is probably a more exciting event for long-time viewers of the earlier series than it was for me. Nevertheless, the scene is a derelict and declining planet, and a strange woman who claims that her daughter, Hannah, has been kidnapped by the Daleks.  She's been able to summon the Doctor to ask his assistance.  Does something seem amiss?

"If Hannah's in a Dalek prison camp, tell me: why aren't you?"
"I escaped."
[laughs]  "No.  Nobody escapes the Dalek camps."
As the Doctor points out, few people are able to avoid the Daleks, and even fewer are able to send him a message.  Sure enough, the strange woman grows a Dalek eye stalk from her forehead, and a Dalek weapon from her arm, and zaps the Doctor with a stun ray.

Similarly, the soon-to-be-divorced Ponds are stunned by human-Dalek-pretenders and captured by the Daleks.  I don't know if it was intentional, or it was just the make-up that Amy Pond was wearing, but she looked much older and very tired during the opening scenes.

What did I make of the human-Dalek-pretenders?  I'm really not sure.  A clever way of snaring the Doctor and his companions, for sure, and for giving the Daleks human hands - and, let's be fair, a much better way than the mutant pigs which the Daleks used when they were in Manhattan.  Overall, I think they were a reasonable addition to the Dalek repertoire... up to this point.

The Doctor and his companions are whisked off to a ship orbiting the Daleks' asylum planet, where their maddest and baddest Dalek comrades are sent to die.  The insane, battle-scarred and the uncontrollable are sent there, as it transpires that killing a Dalek contravenes the Daleks' own perception of 'divine hatred' or a concept of beauty... not a pretty thought.  The Doctor and the Ponds are summoned to appear in front of a Dalek parliament, which is populated with the latest Steven Moffat Daleks (red, white, yellow, blue etc), alongside some of the gold/brown "new series Daleks" from the Russell T Davies era.

Now hang on a minute:  when we were introduced to the new Steven Moffat Daleks, they unilaterally decided that the previous incarnation were genetically impure and proceeded to exterminate them on the spot (see Victory of the Daleks).  And yet here they are, side by side?  Hmm, I was none to impressed by this decision.  I thought the red version of the new Daleks would be the drones/troops/cannon fodder...  I was not totally convinced by this decision.  But still, nice to see lots of Daleks all waggling their eye stalks around at anything and everything that looked exterminatable.  My guess is that Mr Moffat is either responding to fans' criticism of his 'Power Ranger Daleks' or 'Crayola Daleks', or the BBC have spotted a merchandising opportunity and asked him to keep the old guard in.  Having said that, the new Daleks do look a lot better with their colour schemes toned down - the red Daleks, for instance, are more maroon than post-box red, and this definitely helps.

The central guest character in this episode is Oswin Oswald, a survivor from a ship that crash-landed on the asylum and who has avoided capture for almost a year.  She's surviving capture despite frequent, almost-nightly attacks by the Daleks, and is staying sane by listening to opera music (featuring a performance by the Doctor on the triangle that 'got buried in the mix') and by making souffles (although the Doctor wonders where she's getting the milk).  She's a genius who has hacked into the Dalek defences and security cameras - helping to keep them out.

There is a protective shield around the planet, but the discovery of Oswin's spacecraft, crashed on the surface, shows the impenetrable shield is not as impenetrable as they Daleks thought.  And, if something can get through it and get in, then it is possible that something can get out... including a tsunami of uncontrollable and crazy Daleks; that's why the Daleks have summoned the Doctor - to help destroy the asylum (before the inmates escape).  The shield can only be lowered from within the asylum, and the Doctor suggests sending a small task force, before realising that he's actually volunteering himself, known to the Daleks as the "Predator of the Daleks", and his companions.  It seems the Daleks are too afraid to go for themselves.  

The mission plan is simple:  the Predator of the Daleks is to be fired down to the planet, he then lowers the impenetrable shields so that the Dalek's spacecraft can destroy it with missiles.  This seems like a good idea to Rory, until he gets included in the plan as well:

Doctor:  "You're going to fire me at a planet.  That's your plan?  I get fired at a planet and expected to fix it?"
Rory:  "To be honest, that is slightly your MO."
Doctor:  "Don't be fair to the Daleks when they're firing me at a planet.  "What do you want with them [Amy and Rory]?"
Dalek:  "It is known that the Doctor requires companions."
Rory:  "Ohhh brilliant."

The Doctor and his companions are given a wristband to protect them from 'the nanocloud' - a Dalek invention which enables them to assimilate all life forms -  living or dead - and turn them into the Dalek Pretender life forms.  I'm sorry, but for me this was just a bit too much Vashta Nerada for me, especially when the skeletons of dead astronauts (still wearing their white coats, no less) became Dalek Pretenders.  No, this was a bit of a recycled story idea and I was not too impressed with this new usage of the Dalek humanoid zombies.  However, while Amy and the Doctor escape from the roomful of Vashta Nerada - I mean Dalek drones - Amy loses the wristband which protects her from the nanocloud, and must now work to avoid being slowly converted into a Dalek drone.

Rory becomes separated from Amy and the Doctor as they are fired onto the asylum planet, and has to find his own way around:  in fact, he lands in a room full of Daleks.  And here, the Daleks were particulary good in some suspenseful moments, as they start to re-energise as Rory walked among them.  I couldn't help think that Rory was a bit dim in not working out what the Daleks were trying to say when he started, "Eggs. Eggs..." Hmm... I wonder.  Exx--- ter-- min--- ate!

There are some very good action scenes, one of which involves a Dalek trying to take out the Doctor by self-destructing.  An unusual kamikaze tactic from the Daleks, and perhaps a brief comment on suicide bombers?  Probably not, but a good way of taking out a room filled with Daleks in one go, with a wave of the sonic screwdriver and a good shove!

The rest of the story features lots of running around, looking to deactivate the shields, avoid the Daleks and rescue Oswin.  There's some extremely well written development around the soon-to-be-divorced Ponds: it turns out that Rory wants to have children, and Amy certainly doesn't (River was too traumatic for her).  By clever manipulation, the Doctor is able to maneouvre them both into admitting their true feelings for each other... he walks past a security camera, screen visible to the Ponds, and straightens his bow-tie, "Situation fixed."  Although the main characters still suffer from narrative immunity (they must survive to the end of the episode because the story says they must) and there isn't much suspense around them, there is a very clever twist.  In fact, it is an extremely good twist - and it concerns the character of Oswin Oswald.

The Doctor has had his suspicions about her true nature since the beginning - how does she survive, how is she making all the souffles and how is she able to hack the Dalek network?  In a shocking reveal, it turns out that the Dalek inmates captured her and, recognising her genius, transformed her into one of them.  Oswin is a Dalek.  She doesn't know that, and certainly doesn't acknowledge it, and the revelation is as much a shock to her as it is it to us (well, me, anyway).  In twist upon twist, Oswin is able to wipe the Doctor's file from the Daleks' memory, so that they no longer have any recollection of him (and those which are on the planet cease trying to exterminate him).

To wrap up:  Oswin lowers the shields around the asylum planet, and the Dalek spaceship sends in enough missiles to destroy an entire planetary system.  The Doctor is able to overclock a site-to-site teleporter sufficiently to beam himself and the Ponds back to the Dalek spacecraft, just before the planet blows up.

And, as the Doctor points out, he is very good at teleporters, and he beams the heroic trio back, directly inside the Tardis on the Dalek spaceship.  Poking his head around the door, he greets the Daleks with a broad smile.  But the effect of Oswin's actions is felt across the whole Dalek race:  none of the Daleks know who the Doctor is.  There he is, showboating in front of the Dalek parliament, and they demand to know who he is.

"Titles are not meaningful.  Doctor who?"
"Doc-tor who?  Doc-tor who?"

And with that, the Doctor takes Rory and Amy back to their house, where they will be resuming their marriage (much to Rory's delight), and then departs, smiling to himself about "Doctor who...?"

There are clearly big plans to move the Doctor's arc forward considerably, based on this episode.  The Daleks are no longer a direct threat to the Doctor, and the question... sorry, I mean The Question has now been asked.  I wish I could recall the rest of the blurb at the end of the last series, about The Question, but it was big and meaningful.

This episode gets five stars from me - for intelligent use of the Daleks alone, and for the story twist.  More to follow!

My other Doctor Who Reviews

Doctor Who: Asylum of the Daleks
Doctor Who: Sea Devils
Doctor Who: Warriors of the Deep
The Space Babies/The Devil's Chord

Other sci-fi TV reviews:

Star Trek Picard - Season 1 and Season 2
The Book of Boba Fett

Blood Sweat and Tears (GlaxoSmithKline)

It only seems fair that as GlaxoSmithKline (GSK) continue to produce ever more images for their anti-doping advertising campaign, that I should try to keep up with them.

Their latest and final range, "Blood, Sweat and Tears," now features with two Olympic gold medal winners, Beth Tweddle and Sophie Troiano.  GSK have even put them on their own Flickr site. They've drawn some criticism (in fact, the whole range has) for their inaccurate usage of chemistry, and in some cases, totally nonsensical chemistry in their advertising, but I'm still happy to keep parodying them, in an affectionate but not pedantic way.   Having said that, it does seem strange that a multinational chemical and pharmaceutical company hasn't bothered to display its scientific knowledge in its advertising, and has left the science to a group of non-scientific marketing folks.

It is worth pointing out that GSK is the Official Laboratory Services Provider for the London 2012 Olympic and Paralympic Games, but that they are not actually carrying out the testing - just providing the facilities. These labs, facilities and equipment are provided to enable expert analysts from King's College to independently operate a World Anti-Doping Agency (WADA) accredited laboratory during the Olympic Games.

After that brief aside, here is what will probably be my final poster, celebrating the vast majority of non-Olympians who also believe in training, running and playing sports in a doping-free environment.  It's meant to be humorous, not political, and I'm not trying to promote or discredit any manufacturers of anything in particular (such as 'high-energy' soft drinks).

Chemistry Advertising: GlaxoSmithKline Chemistry Again

I'd like to follow up on my previous post about how GlaxoSmithKline (GSK) have recently been promoting their anti-doping testing technology for the Olympic games.  They've done this with some very impressive 'chemistry' adverts featuring British athletes. 

 Now that the athletes are winning medals, they've changed the message to one about 'blood, sweat and tears', see below (taken from GSK's FB page) :

After my first set of chemistry images based on GlaxoSmithKline's advertis, I think it's only fair that I try to keep pace with these new developments, so I've produced a few more of my own.  They're designed to recognise those of us who aren't Olympic standard athletes, but who still believe in drug-free sports and improving our performances through hard work and practice.  What do you think?

Chemistry Advertising: Glaxo Smith Kline

A different slant on Chemistry cartoons this time.  I've recently noticed (with enjoyment) that GlaxoSmithKline (GSK) have recently been promoting their anti-doping testing technology for the Olympic games.  They've done this with some very impressive 'chemistry' adverts featuring British athletes (some of them medal winners).  GSK have put these on their Facebook page, and so I've borrowed them, and produced some of my own alternatives.


Mine aren't meant to be offensive, just comical parodies.  I'm not intending to criticise GSK, just borrow their 'chemical plus picture' motif.  I'm not even going to criticise the chemistry of the 'molecules' they've designed... I'm just going to smile and participate in the chemistry advertising as well.


Here are GSK's (taken from their FB page) :

And here, just to raise a smile for the 'every man' who also doesn't believe in taking drugs to enhance his performance, are mine.

What do you think?