"A/B and MVT doesn't work"

A few months ago, I wrote about my experience of how modern power-tools just don't work, and how my power-drill wasn't working as well as other people said it would.  Well, today, I've just read that A/B and multivariate testing does not work on e-Commerce sites.  And I think it's true.  After all, I read it online, so it must be true.  And it matches with another recent experience I had:  it seems my car is broken.

I've had my car for a number of years - it's a very nice car and I was fortunate enough to have it from new.  It seems to drive well, but I've found that there are a number of fundamental issues with it.

The speedometer doesn't work.  The speed readout (both digital and analogue) agree with each other to about 1%, but the maximum speed on the speedo is 150 mph, and I've never managed to achieve those speeds. I took it for a drive recently, and even though my maximum speed was 60 mph, which I achieved for a short while, it still took me 30 mins to travel 20 miles.  No - the speedometer is just plain useless.

The miles per gallon calculation on the dashboard display is inaccurate, and certainly doesn't match the driver's guide (yes, I read the driver's guide).  It keeps changing all the time - what's that all about?  Even the 'average' fuel efficiency figure changes.  Seriously?  Another important part of my car is not working properly.  The dealer says that the figures are based on "average use" (whatever that is), and over a long period of time it will average out, but I'm not prepared to wait that long.  I want to see my driving data match the actual data all the time.

Wing mirror showing rear view.

The rear-view- and wing-mirrors only show me where I've just been, and - worse still - the image is reversed (transposed left to right, horizontally).  Honestly, just how am I supposed to make sense of the image - or see where I'm going - by looking in them?

I have my own satnav - it's a mid-range TomTom model, so it's good and reliable - but when I use it in my car, the satnav predicted arrival time is wrong.  Worse still, the satnav (which has full maps of Europe) won't integrate with the car's engine and drive it for me, so I am stuck with still having to actually drive the car myself.  And did I mention that the speed readout on the satnav doesn't match the car's speedometer?

I have driven the car in heavy traffic and along long stretches of motorways; it's covered country lanes and suburban housing estates - but it's only occasionally performed as well as it's supposed to in the manual, and it only rarely displays MPG figures that match my own forecasts and predictions.

To be honest, the only things that really work as I expect them to are the odometer (mileometer) and the clock.

I've tried monitoring engine temperature (the engine is cold when it starts in winter and is generally warmer in summer); oil pressure (unaffected); brake pad usage (it goes up when I travel through busier streets) and so on, but none of these metrics have been particularly useful at helping me drive better, faster or further.

I've attempted to forecast my future mileage, fuel efficiency and average speed based on the distances I covered yesterday, last week and last month.  It's almost impossible - it seems that it varies day by day.  Having said that, we went on a family holiday round North Wales last August, and I know we're not going this year, so I can expect a decrease in my average mileage for this year.  And my fuel efficiency will be lower because I won't be driving as many of the long, steady dual-carriageway runs that I did last year.  But these are just estimates really, based only on actual events that happened last year but won't happen this year.

My car is on a three-year lease, and the lease expires next summer.  I will probably swap it for a different car, but I'd like to know how that car's fuel efficiency and speed performance will compare with my current one.  Not just what the rear view mirrors look like, or the maximum speed on the speedometer, or even what the driver's manual says. 

Perhaps I should continue to drive as I usually do for the next 6-12 months, and ask someone to drive around, following me, in my potential purchase.  Then, next summer, maybe we can compare their actual performances and choose between keeping my car or swapping it?  Who knows?

I've used car performance as an analogy to online testing before:
How long to run an A/B test for? (2013)
Who holds the steering wheel? (2011)

Numbers 1,2,3,4 from 100 to 150

Following on from last month's post about achieving the numbers from 1 to 50 (and then from 1 to 100) using just the digits 1,2,3,4 and the maths operators + - / *  and powers and factorials, I've gone a step further (and possibly a step too far) - from 101 to 150.  This is proving far more difficult - as you would expect - and so far I've only managed around 30 of the 50.  If the numbers below are blank, it's because I haven't solved them yet :-)

Major thanks to Denis and to lookagain from the Free Math Help Forum for their many solutions and for helping me fill the gaps that I had.  Denis's solutions are in purple, lookagain's are indicated in blue.  You'll see that in two cases, we've had to use the FLOOR function (i.e. the result of an operation is rounded down to the nearest integer) - we're trying to avoid using this where possible.  If you have any better solutions, please let me know!

101 = (4! / .3) + 21

102 = 3! * (21-4)

103 = 4√.1∗.2 + 3

104 = (4! + 2) * (3+1)

105 = 21 * [3 + SQRT(4)]

106 = (3+2)! - 14

107 = 4!.2−13

108 = 4 * (3 ^ (2 + 1))

109 =  4!−1.2−3!

110 = (31 + 4!) * 2

111= (1+4)! - 3^2

112 = (1+3!) *4^2

113 = (4! / .2) -1 - 3!

114 = (1+4)! – (3 * 2)

115 = (1+4)! – (3 + 2)

116 = (3+2)! – (1 * 4)

117 = (3+2)! – (4 - 1)

118 = 124 - 3!

119 = (4+1)! – (3 - 2)

120 = (4+1)! * (3 - 2)

121 = (4+1)! + (3 - 2)

122 = (3! * 21) – 4

123 = (42 – 1) * 3

124 = (3+2)! + (1 * 4)

125 = (1+4)! + 3 + 2

126 = 42 * 3 * 1

127 = (42 * 3) + 1

128 = 1 *(2 ^ (3+4))

129 = (42 + 1) * 3

130 = (3! * 21) + 4

131 = 1+4!.2+3!

132 = (31 + 2) * 4

133 = 4! / .2  + 13

134 = (3+2)! + 14

135 = (21+4!) * 3

136 = 132 + 4

137 = FLOOR (412/3)

138 = (24 - 1) * 3!

139 = 142 - 3  

140 = (1 + 3 + 4!) / .2

141 = (3! * 4!) – (2 + 1)

142 = (3! * 4!) – (2 * 1)

143 = (3! * 4!)  – (2 - 1)

144 = (3 * 4) ^ (2 * 1) = (3 * 2)! / (4 + 1)

145 = (13^2) - 4!

146 = (3! * 4!) + (2 * 1)

147 = (3! * 4!) + 2 + 1

148 = 142 + 3!

149 = (4! + 3!) /0.2  - 1

150 = 3! * (21 + 4)

Online Optimisation is a Game of Chess

It occurred to me recently that there are some similarities between playing a game of Chess and optimising a website through testing (apart from the considerable differences between them).

 Many years ago, when I started this blog, I envisaged it as a place where I would type up my chess games, with commentary, lessons and points for improvement.  A quick glance through my archive of blog posts will quickly show that the blog really hasn't matched its original plan.  It evolved and changed, in particular in 2011, when I shared my first few posts about web analytics and I realised that I could attract significantly more readers by blogging and sharing about web analytics than I have ever managed with Chess.  I figure that Chess is a much more mature subject, with many more people who are considerably more experienced than me; whereas web analytics (and especially my main area of interest, testing) is a much younger field, and there's scope for sharing ideas and experiences which are still novel and interesting.  However, there are some similarities between the two - here are a few thoughts.


Strategies and Tactics

Black prepares a tactic.

Plans in Chess can be broadly separated into strategies (long-term aims) and tactics (short term two-or-three move plans).  The long-term aim is to checkmate your opponent's king, and throughout the game that will become the more prominent goal.  In the short term, as you progress through the opening moves, you'll identify potential opportunities to capture your opponent's pieces, to put your pieces on good squares and to restrict your opponent's chances of beating you.  Some of these are short term, some are long term.  For example, it may be possible to win one of your opponent's bishops with a cunning trap (if your opponent is not vigilant).  This will make it easier to achieve your long-term goal of winning the game by checkmating your opponent.

Testing offers the same range of opportunities:  are you looking for quick wins (who isn't?) or are you planning to redesign an entire page, or even an entire site?  Will you be implementing wins as soon as you have confirmed results, or are you going to iterate and try to do even better?  Are you compiling wins to launch in one big bang, or are you testing, implementing and then repeating?  How far ahead are you planning?  Neither approach is necessarily better, as long as you're planning, and everybody is agreed on the plan!


Aims and Goals

White threatens checkmate.

As I've just mentioned, in Chess there's a clear goal:  checkmate your opponent's king, by threatening to capture his king and leaving him with no way to escape.  The aim for your online testing program may not be so clear cut, and if it isn't, then it might be time to find one single aim for it.  You might call it the mission statement for your optimisation program, but whatever you call it, it's important that everybody who's involved in your program understands the purpose of the testing team.  The same applies to each test, as I've mentioned before - each test should have a clear aim, that everybody understands and that can be measured in some clearly-defined way.

Values and KPIs

Each piece in Chess has a certain nominal value; the values aren't precise, but they provide a meaningful comparison of the strength and ability of each piece.  For example, the rook is worth five pawns, the knight and bishop are worth three pawns each, and the queen is worth nine.  This enables players to quickly evaluate a position in a game and say which player is winning, or if the position is roughly equal.  It's slightly more complicated than that, as you have to take into consideration the position of the pieces and so on, but a quick comparison of the total material value that each player has will give a good idea of who's winning.

Rooks are worth five pawns; bishops are worth three pawns and queens are worth nine.

This also means that it's possible to determine if a plan or a strategy is likely to win and is worth pursuing.  It may be possible to trap your opponent's rook (worth five pawns), but if doing so will mean losing two knights (each worth three pawns) and a pawn, then the trap is not really beneficial to you.  If, on the other hand, you could trap your opponent's king (winning the game) at the cost of two rooks and a knight, then that's definitely worth doing.
 
The key performance indicator for a game of Chess is your opponent's king, and if you can measure how close you are to capturing (or checkmating) your opponent's king, then you can see how close you are to winning the game.  You also need to keep your own king safe, but that's where the analogy breaks down :-)

White wins despite less material.

In online testing, your plan, strategy and tests all need to have KPIs.  Once you've established your long-term aim, you can set KPIs against each test which are connected to achieving the long-term aim.  If you want to improve the return on investment (ROI) of your online marketing, you could look at the landing page... improve the bounce rate and the exit rate, and encourage more people to move further into your site and view your products.  Alternatively, you could look at improving conversion of visitors from cart (basket) to checkout.  Or perhaps the flow of visitors through your checkout process.  Providing you can tie each of your tests and your tactics to the overall strategy: "Improve ROI for online marketing" then you can measure whether or not it's succeeding.

Classifying your KPIs in order of importance is also important - as we saw in Chess, if you can win your opponent's pieces but lose several of yours in the process, then it's probably not a good idea.  In testing, what would you do if your test recipe had a worse bounce rate but higher overall conversion (a situation that's not impossible)?  Which is the more important metric - conversion, or bounce rate?  Would your answer be the same if it was improved conversion but lower revenue (people not spending as much per order)?  Are you going to capture your opponent's knight but lose your queen?

Win, lose or draw?

Queen takes King: checkmate!

In Chess, there are clear rules that determine the outcome of a game.  Either one player wins (so the other loses), or it's a draw, and there are various ways of drawing:  including by agreement (both players decide neither can win); by stalemate (one player cannot make any legal moves) or a drawn position where it's clear that neither player has enough material to checkmate the other.


I hate losing at Chess.  However, the truth be told, I'm not much above average as a Chess player, and I'm the weakest player at my club (this has not deterred me, and I still play for fun).  This means that I get plenty of chances to analyse my losses, and see where I could improve.  Do I make the same mistakes in future games?  Not usually, no.  

However, I'm not satisfied to stay as the weakest player in my club - I just see this as an opportunity to do some giant-slaying in my future matches.  I read books, I visit Chess websites, I practice against other people and against computers, but I especially review my own games.  Occasionally, I win.  And do I analyse my winning games?  Absolutely - I may have already seen during the game that my opponent missed a chance to beat me, but did I also miss a chance to win more easily?

In online optimisation, the rules for calling a test a win, lose or draw are still up for debate, and they vary between companies.  And so they should.  For example, how long should you run your test? Each company will have its own testing program with its own tactics and strategies, and its own requirements.  Do you want to have 99.9% confidence that a test will win, or do you want some directional test data to support something you already believed based on other data sources?  How quickly do you want to run the next test, or implement the winner?  Providing that the rules for calling the win, lose or draw are agreed in advance, I might even suggest that they could vary between tests. This is, of course, totally different from Chess, where the centuries-old rules of the game clearly state the requirements for a win or a draw.  Otherwise though, I think it's fair to say that KPIs, metrics and strategy have their approximate equivalents in pieces, pawns and plans - and that thinking and planning are definitely the way forward!

Chess cartoons taken from the 1971 printing of  Chess for Children, originally published 1960.

Mathematical Film Reviews

Here are some simple film reviews, expressed in mathematical form.  If you've seen some of the older films (or read the books), then you won't need to watch the newer ones (or perhaps they'll help you decide if you'd like to watch them).

Equilibrium = Fahrenheit 451 + 1984 + The Matrix

After Earth = Oblivion + Will Smith's son

Noah = Transformers + Merlin + Titanic -Bible

Live Die Repeat = Mission Impossible + Groundhog Day + 12:01 + Starship Troopers

White House Down = Olympus has Fallen

Earth to Echo = ET + Close Encounters + Batteries Not Included

Lucy = Limitless + Scarlett Johansson

The Philadelphia Experiment = Quantum Leap + Titanic + Time Tunnel

Looper = Back to the Future + Die Hard

X-Men: Days of Future Past = Back to the Future + The Incredibles

Guardians of the Galaxy = The Avengers Assemble + Star Wars IV

I'm not saying that all the films in Hollywood are derivative (in fact there are some very original films out there), I'm just pointing out some recent similarities I've found.  To be honest, many of the films I've mentioned here are actually favourites of mine.

On the subject of very original films, I'd like to share a brief word about 2001 A Space Odyssey.  It's widely regarded as a genre-defining ground-breaking film, so you might be surprised to learn that even though I'm a sci-fi fan, I didn't watch it until 2015 (even though it was released in 1968, years before I was born).

I don't know why I hadn't watched it until now.  Now, I wish I hadn't bothered.

For example, if you just want to watch a computer play chess, then watch Kasparov and the Machine.

If you like model spaceships and spacecraft, go to a hobby modeller shop
If you like actual spacecraft, watch the footage of the Sputnik launch and the early Apollo missions.
If you like listening to the Blue Danube Waltz, buy it on CD (or tape, or even vinyl).

If you like a murder mystery, go and watch (or read) Agatha Christie or Arthur Conan Doyle
If you like stories with a bizarre ending, watch The Italian Job or Planet of the Apes
If you like long drawn-out stories, watch Les Miserables (released 1925, at 5hr 59 mins), or the 1927 film Napoleon (at 5hr 30 mins).

All better in comparison, and all better for watching.  2001 A Space Odyssey's only claim to fame is that it did a lot of things first.  It's just a shame that it made so many mistakes at the same time, and while doing so many things first, it forgot about the basics like story-telling, pace, narrative, dialogue and just having things make sense.

Can a 747 take off from a conveyor belt runway?

This question is currently going round on Facebook; one of my friends posted it, I answered it, and a few hours later entered a lengthy and circular debate in the comments section.  Here, with more space than a Facebook comments section, I'd like to pose the question, answer it and then address some of the misconceptions.

Here's the question:  Imagine a 747 is sitting on a conveyor belt, as wide and long as a runway.  The conveyor belt is design to exactly match the speed of the wheels, moving in the opposite drection.  Can the plane take off?

Shared originally by Aviwxchasers.Com a news and media site from the US.

My friend shared the post, and answered, "No - there's no way to generate lift."
My original answer:  

"I think you're assuming that the 747 will be moving because its wheels are being driven. The forward thrust of the aircraft comes from its engines, not from making its wheels turn (like a car). So the engines push the aircraft forward and the wheels just slide and skid along the conveyor belt , while the engines push the aircraft to take-off speed. The wheels don't have to turn to allow the aircraft to move."

However, this wasn't deemed sufficient by some other commentators, who (to summarise) posted the following questions or objections.

1. The conveyer would counteract the forward movement produced by the engines' thrust.  The conveyor matches the speed of the wheels in the opposite direction, so there can be no forward motion.
2. Lift is created by air flow over the wings. It doesn't matter how much thrust you have , if doesn't generate airflow over the wing, it won't fly.
3. The method of propulsion should be irrelevant. If the speed of the conveyor matches the speed of the wheels in the opposite direction, there can be no forward motion.
4. Take the vehicle out of the equation, it's all about the wheels and the conveyor. The faster the wheels turn, the faster the conveyor goes. Stick anything you want on top of the wheels, the principles are the same. Newton's third law
5. Indeed it's not about the wheels it's about forward movement (thrust from the engine) needed to take off but that forward movement is being counteracted by the conveyer    

Here, I propose to look at these arguments individually and collectively, and explain why the points which are raised aren't sufficient to stop the aircraft from moving and taking off.

1.  The conveyor would counteract the forward movement produced by the engines' thrust.  The conveyoy matches the speed of the wheels in the opposite direction, so there can be no forward motion.

1A.  The conveyor will only stop the wheels' rotation from generating forward movement.  However, it is not the wheels which are being driven - this is not a car or a truck.  The forward movement of the plane is not a result of the wheels successfully pushing against the conveyor belt; the forward movement of the plane comes from the push of the engines.  The result is that the plane will move forwards (the engine pushes against the air flowing through it, the air pushes back - Newton's Third Law) and the wheels will spin and skid down the track.

It's not "The plane moves forwards because the wheels go round", it's, "The wheels go round because the plane goes forwards [over a surface which has sufficient friction] ."


2. Lift is created by air flow over the wings. It doesn't matter how much thrust you have , if doesn't generate airflow over the wing, it won't fly.

True.  But there is thrust, and it is generating airflow over the wing.  The conveyor belt does not have the ability to resist the movement of the plane, only to counteract the turning of the wheels.  

The plane can move forwards -even along the ground - without its wheels turning.  The engine isn't driving the wheels.

3.  The method of propulsion should be irrelevant. If the speed of the conveyor matches the speed of the wheels in the opposite direction, there can be no forward motion.
and 
4. Take the vehicle out of the equation, it's all about the wheels and the conveyor. The faster the wheels turn, the faster the conveyor goes. Stick anything you want on top of the wheels, the principles are the same. Newton's third law

3A and 4A.  Comment 3 was a response to my question, "What happens if we replace the 747 with a space rocket, aligned horizontally on the conveyor belt runway, on wheels?"  and the comment suggests a misunderstanding about what's causing the forward motion of the plane.  The thrust of a space rocket is completely independent of the any wheels (they normally fly fine without them) and the wheels will just get dragged along the conveyor belt, without turning. 

Yes, the faster the wheels turn, the faster the conveyor belt goes.  But the wheels don't have to turn for the plane to move (as I said in response to 2).  The conveyor belt does not have some property which prevents something from skidding along it, just from preventing any forward motion due to wheels turning on it. 

You can only take the vehicle out of the equation when you realise that the vehicle isn't a car, with the limitations that a car has.

Newton's third law applies to rotating wheels and conveyor belts.  It also applies to the aircraft engines and the air flowing through them, or to space rockets and the fuel burning inside them.  I wonder if the Starship Enterprise (on wheels) would have this problem?

5.  It's all about forward movement (thrust from the engine) needed to take off but that forward movement is being counteracted by the conveyer  

The forward movement is not counteracted by the conveyor.  The conveyor just stops rotating wheels from generating any forward movement.  But if the wheels aren't rotating (because they're not being driven, such as in a car) then there's no movement to resist.

In conclusion, I'd like to offer this video from "Mythbusters" which shows a light aircraft attempting to take off against a conveyor belt (which in this case is being pulled by a pickup truck to match the plane's speed).  To quote one of the engineers, "People just can't wrap their heads around the fact that the plane's engine drives the propellor, not the wheels."

Some other 'everyday maths' articles I've written:

A spreadsheet solution - the nearest point to the Red Arrows' flight path from my house
The Twelve Days of Christmas - summing triangle and square numbers
Why are manhole lids usually circular?
"BODMAS" puzzles - what's the fuss?