Tuesday, 17 October 2017

Quantitative and Qualitative Testing - Just tell me why!

"And so, you see, we achieved a 197% uplift in conversions with Recipe B!"
"Yes, but why?"
"Well, the page exit rate was down 14% and the click-through-rate to cart was up 12%."

"Yes, but WHY?"

If you've ever been on the receiving end of one of these conversations, you'll probably recognise it immediately.  You're presenting test results, where your new design has won, and you're sharing the good news with the boss.  Or, worse still, the test lost, and you're having to defend your choice of test recipe.  You're showing slide after slide of test metrics - all the KPIs you could think of, and all the ones in every big book you've read - and still you're just not getting to the heart of the matter.  WHY did your test lose?

No amount of numerical data will fully answer the "why" questions, and this is the significant drawback of quantitative testing.  What you need is qualitative testing.

Quantitative testing - think of "quantity" - numbers - will tell you how many, how often, how much, how expensive, or how large.  It can give you ratios, fractions and percentages.

Qualitative testing - think of "qualities" - will tell you what shape, what colour, good, bad, opinions, views and things that can't be counted.  It will tell you the answer to the question you're asking, and if you're asking why, you'll get the answer why.  It won't, however, tell you what the profitability of having a green button instead of a red one will be - it'll just tell you that people prefer green because respondents said it was more calming compared to the angry red one.

Neither is easier than the other to implement well, and neither is less important than the other.  In fact, both can easily be done badly.  Online testing and research may have placed the emphasis may be on A/B testing, and its rigid, reliable, mathematical nature, in contrast to qualitative testing where it's harder to provide concise, precise summaries, but a good research facility will require practitioners of both types of testing.

In fact, there are cases where one form of testing is more beneficial than the other.  If you're building a business case to get a new design fully developed and implemented, then A/B testing will tell you how much profit it will generate (which can then be offset against full development costs).  User testing won't give you a revenue figure like that.

Going back to my introductory conversation - quantitative testing will tell you why your new design has failed.  Why didn't people click the big green button?  Was it because they didn't see it, or because the wording was unhelpful, or because they didn't have enough information to progress?  A click-through-rate of 5% may be low, but "5%" isn't going to tell you why.  Even if you segment your data, you'll still not get a decent answer to the either-or question.  

Let's suppose that 85% of people prefer green apples to red.  
There's a difference between men and women:  95% of men prefer green apples; compared to just 75% of women.
Great.  Why?  In fact, in the 30-40 year old age group, nearly 98% of men prefer green apples; compared to just 76% of women in the age range.

See?  All this segmentation is getting us no closer to understanding the difference - is it colour; flavour or texture??

However, quantitative testing will get you the answer pretty quickly - you could just ask people directly.

You could liken it to quantitative testing being like the black and white outline of a picture, (or, if you're really good, a grey-scale picture) with qualitative being the colours that fit into the picture.  One will give you a clear outline, one will set the hues. You need both to see the full picture.

Thursday, 14 September 2017

Badly-Written Maths Questions [BODMAS]

I have ranted in the past (albeit briefly) about badly written maths questions. These are the kind of question that do the rounds on Facebook, where - due to the deliberately ambiguous way that the question is written - there are at least two different answers.

The idea of these questions isn't to test people's maths skills.  It's designed to 'go viral' by generating conflict and disagreement between the know-it-alls, the qualified mathematicians and those who can't recall or don't know how to handle maths questions when there isn't enough information to easily proceed.

You know the kind of thing:

What is 3 + 4 * 5 + 6 - 7?
Only 1 out of 10 will get this right!

Firstly:  this is NOT proper Maths.  It just isn't.  Don't worry if it's confusing - it's deliberately intended to be.

Secondly:  if you don't get it 'right', then you'll probably continue feeling that maths is irrelevant, complicated, meaningless and inaccessible.  Because that's probably what you thought before, and the long list of comments that say it's 22  or 34, all equally convinced that they're right and the other person is wrong.  There'll be a few comments about showing how they've worked it out, and then a few people will say BODMAS.


BODMAS is the agreed way in which we carry out calculations like my example.  Mathematicians don't like uncertainty or ambiguity, and will go to great lengths to make their meaning perfectly clear and precise.  All scientists are the same - they show great precision in language, whether that's words or numbers.

BODMAS states that a calculation should be carried out in a particular order:

Brackets - any terms in brackets (or parentheses) should be calculated first.
Orders - any numbers which are raised to powers (squared, cubed, square root) are done next, after any calculations in brackets.  (Previously called Operators)
Division - divisions are the next priority.  Any two numbers or terms which are next to each other have to be divided, after any brackets and operators, but before anything else.

Multiplications - after you've done all the divisions, you then do all the multiplications.
Additions - any terms which are to be added together are done after the multiplications.
Subractions - finally, any remaining amounts are to be subtracted.

So, to take my example:

3 + 4 * 5 + 6 - 7  =  ?

There are no brackets or orders (powers) in my calculation, so the first calculation I will do is the Multiplication.  4 * 5 = 20.

So now, my calculation looks like this:

3 + 20 + 6 - 7 = ?

There's no dividing in my expression, so I can move on to the additions:  3 + 20 + 6 = 29

Which leaves me with:
29 - 7 = 22

And the answer is therefore 22.

While it may be possible to read the question differently, this will give a mathematically inaccurate [wrong] answer.  It may seem natural to read the question from left to right, but this will give a different and wrong answer:

3 + 4 (=7)
*5 (=35)
+6 (=41)
-7 = 34

Wrong answer = 34.

If you think that it's unfair or unrealistic to have to follow such precision, let me present some examples from written English, that show how important it is to state things clearly and in the right order:

I'm glad I'm a man, and so is Lola
Is Lola a man?  Are you reading from left to right, or did you go back to the middle?
He fed her cat food.
Was he looking after her cat?  Or was he making a culinary error?
John saw the man on the mountain with a telescope.
Who has the telescope?

Or how about this one, which has recently started going around Facebook, and is (almost certainly deliberately) full of mathematical and grammatical problems.

1 rabbit saw 6 elephants while going to the river.
Every elephant saw 2 monkeys going towards the river.
Every monkey holds 1 parrot in their hands.
How many Animals are going towards the river ???
Does "to the river" count as "towards the river"?
"Every elephant saw 2 monkeys" - is that each elephant saw 2 monkeys, or they all saw the same 2 monkeys?
"How many animals?" - this depends on if you include birds in your definition of animals (some do, some don't).
This is the epitome of a trick question, and this kind of uncertainty is completely unacceptable in maths - but that's what drives the apparently viral threads on Facebook.  People will argue vehemently about one answer or the other - confusing everybody else and leading to the frustration that we see (it's much easier to explain things in a five minute conversation than it is with five paragraphs of comment text on social media).

Maths has enough of a bad reputation for being confusing, inaccessible and frustrating; it doesn't need people asking "What's 5 + 6 *7 -8? Only 1 in 5 know the real answer!" to make it any worse.

(The answer is 39)
(The answer to the Albert Einstein question (which is particularly devious)  is -13
3 - (6*3) + 2 = 3 (- 18 + 2) = 3 - 16 = -13

Tuesday, 12 September 2017

Doctor Who: Warriors of the Deep

Doctor Who: Warriors of the Deep, on DVD

I've recently acquired a number of classic Doctor Who episodes on DVD from my local Salvation Army charity shop, and have decided to watch and review them here - under the category of 'miscellaneous opinions'.

The first of these DVDs is "Warriors of the Deep", featuring Peter Davison as the Doctor, in a series that originally aired in January 1984.  To summarise the plot: the Doctor and his two companions, Tegan and Vislor, materialise on Sea Base 4, in the middle of a tense situation between the humans and the 'warriors of the deep'.  Sea Base 4 is deep under the sea, isolated from outside contact, and as we join the story we discover that they're  monitoring their enemies' movements.  The enemy has stealthily moved a probe to within observation range of Sea Base 4, but they've still be seen. Sea Base 4 wait, and then release a probe to investigate further.  This quiet move and counter-move is a recurring theme throughout the story, as the tension is slowly ratcheted up.

While the plot between Sea Base 4 and its alien opposition unfolds, we discover that the base's medical officer (Doctor Solar) and its controller have deliberately maneouvred an inexperienced officer into the role of the base's weapons officer.  As the pressure and stress of the outside situation increase, the rookie officer starts to crumble, and the controller and medical officer are able to manipulate the base commander into giving them a duplicate controller disk.  The purpose of this isn't immediately clear, but something is clearly awry.  Conversely, the underwater aliens have a clear purpose - to resurrect or awaken a large army of dormant warriors who've been stored in an underwater cavern for many years.

The story moves along steadily but not quickly - and with a constant sense of tension in the atmosphere.  The sets and costumes are understandably dated (filmed in 1984) but once you look past this, it's an engaging story.  The Sea Base has been conceived as part of a sterile,white and metallic future, with bright, clean lines throughout (similar to the Tardis of its day).  The main questions that the story explores are: are the aliens more dangerous than the duplicitous humans; and who are the aliens anyway?  

While moving along, the story gives a remarkably high proportion of time to the supporting cast - the Doctor and his companions barely get around 15% of the screen time in the first episode (by my estimation), and the second episode is the same.  I guess it's a reflection of the mini-series approach of the classic series, where  more time could be dedicated to building up the scenario and the various characters with their own stories and plans.  The story takes its time, but doesn't seem to dawdle, as each of the characters follows their own arc.

There are the overtones of the Cold War - mutual trust between the Doctor and the Sea Base officers, mutual distrust between the humans and the underwater aliens - as was common in 1980s' TV series.  It's genuinely difficult to decide whether to take sides with the aliens or the humans; neither side acts with complete transparency.  The aliens move first, and despite the Doctor's warning the humans fire first - to no avail.  The Silurians claim to be fighting a 'defensive war' (the Doctor claims there is no such thing) and are manipulating the humans into fighting a destructive war amongst themselves. It's a novel idea, and it certainly makes sense given the humans' duplicitous behaviour throughout this story.  The tension continues to increase as the Silurians begin their plot to initiate planetary war among the humans, and the Doctor and his companions develop a counter-plot to kill all the Silurians and Sea Devils. Who will blink?  I won't spoil the ending (having outlined the rest of the plot!) but it is entirely in keeping with the rest of the story.

Overall, I enjoyed this series - it was initially difficult to get past the dated sets and costumes (especially the aliens), and the main problems I had was how an underwater sea base could be so bright, airy and spacious (even when it's under attack and the weapons system is disabled); and how the pace of the story was much slower than modern Doctor Who.  Being spread over 100 minutes meant that there was more time spent on each aspect of the story - there were fewer quick action scenes, and significantly less running around (in contrast to modern Who, where chasing and running up and down corridors has become almost cliched).

Next:  The Sea Devils (I get the feeling I should have watched this one first)

Thursday, 31 August 2017

Playing Games by Rolling Dice

This is a very short study of how to vary board games by changing the way you use the rolls of two dice. When playing board games with my children, we've found that just adding the dice totals together doesn't always produce the most useful result. Classically (for example in Monopoly), players use the sum of two dice to determine how many spaces to move their counter.  This leads to a very simple distribution of results (die 1 is shown vertically, die 2 is shown horizontally; the sums complete the table). 

Sum 1 2 3 4 5 6
1 2 3 4 5 6 7
2 3 4 5 6 7 8
3 4 5 6 7 8 9
4 5 6 7 8 9 10
5 6 7 8 9 10 11
6 7 8 9 10 11 12

Average (mean):  7
Average (mode): 7

Now, for an alternative distribution, we can take the maximum value of the two dice (whichever is the higher of the two).  This only gives a range of 1-6, but it's skewed towards the higher end of the distribution (since there are 11 ways of scoring 6, and only one way of scoring 1).

Max 1 2 3 4 5 6
1 1 2 3 4 5 6
2 2 2 3 4 5 6
3 3 3 3 4 5 6
4 4 4 4 4 5 6
5 5 5 5 5 5 6
6 6 6 6 6 6 6

Mean 4.472222222
Mode 6

Conversely, if we take the minimum of the two values, then we have a distribution which is skewed to the lower end (there are 11 ways of scoring 1, and only one way of scoring 6).

Min 1 2 3 4 5 6
1 1 1 1 1 1 1
2 1 2 2 2 2 2
3 1 2 3 3 3 3
4 1 2 3 4 4 4
5 1 2 3 4 5 5
6 1 2 3 4 5 6

MEAN 2.527778


The straight sum gives an average of 7, with a symmetrical split
Taking the maximum (i.e. whichever of the two dice is the largest) gives a mode of 6, and a mean of 4.47.
Taking the minimum gives a mode of 1, and a mean of 2.52.

We found this useful in our games, where we introduce a special feature which allows you to either move your pieces forwards by one roll, or one of your opponents' pieces backwards by one roll.  It's not desirable to move pieces back further than forwards (there's the potential for people to make no forward progress, and extend the game excessively), so the skewed distribution of maximum or minimum is working well for us.

The distribution of totals for one die gives a mean average of 3.5
For two die, the mean is 7.
For three, I suspect the mean is 10.5, and in a future blog, I'll look at this in more detail, along with other ways of producing interesting distributions with just two dice.

Tuesday, 4 July 2017

2017 New Year's Resolutions, Reviewed

At the start of 2017, I made four New Year's Resolutions.  We're now half way through 2017, so this seems like a good point to review my progress on each of them.

1.  To give more than I receive
On the surface,I think this one has been easier than the others. It's been an exciting challenge, and throughout the year I (and we as a family) have  given away all sorts of items - but we've still received many things too.  I had my 40th birthday in February and was astonished by the generosity of my friends, which made giving more than receiving a real challenge.  I was also very pleased by my friends who made charitable donations on my behalf, and a survival shelter (UNHCR) and drinking water for 10 people (Oxfam) were donated for me.   This is giving and receiving simultaneously - I love it.

However, this has also been very challenging:  who am I giving to, and is it really giving?

Matthew 5:46-47
If you love those who love you, what reward will you get? Are not even the tax collectors doing that? And if you greet only your own people, what are you doing more than others? Do not even pagans do that? 

So, it's great to be able to post items on Facebook ("Free to good home...") but it's also self-filtering, since I'll only be giving to my friends ("those who love me") - so is that really rewarding? And it's hardly giving in secret if it's plastered all over social media.  And that's something else: it's difficult to say, "Oh yes, I'm doing really well at giving," without sounding like I'm boasting about it.  So I'll reiterate that it's been challenging to give to strangers, and to give without expecting reward, and I'll mention this verse, which has been a source of encouragement to me.

Proverbs 19:17
Whoever is generous to the poor lends to the LORD, and he will repay him for his deed.

And we will continue to make regular donations to our local Salvation Army shops. 

2.  To spend less time on trivial matters

Trivial matters are things that serve absolutely no practical purpose. Except that sometimes a bit of no practical purpose is a good thing.  Sometimes, after a busy day at work and having put the children to bed, trivial matters are a welcome break.  So no, I haven't completely cut out social media, TV, DVDs and the like, but I'm spending far less time on Facebook, almost zero time on Twitter and I am being more selective about what I watch on television.  And deleting the Facebook app from my phone was a good move - I'm no longer interrupted by the latest event being promoted by someone I follow, or by "Somebody else commented on the post you commented on a couple of days ago."  I am gently nudging myself to do something else while watching TV in the evenings... like writing articles for this blog, for example.

It's also made me think about what's actually trivial.  Building a Lego model with my children:  trivial or important?

3. To produce more than I consume 

No.  I am producing more than last year, and I am consuming less (and of better quality), but I don't think I'll ever tip the balance. After all, I have only one mouth but two ears.  However, there are a number of things I've 'produced' this year (and this is just a selection I can recall off the top of my head):

- A board game to play with two of my children (we titled it "Back to Base", and it's a huge three-player game played on a triangular board, currently on its fourth version)

- Pieces to play "Back to Base" (we each need five pieces, and they've become more elaborate over the months).  This has become almost a repair-not-replace, as I've kitbashed a number of figures from other games (for example, a figure of Christophe from Frozen - as found in a charity shop - became an astronaut with a camera).

- More blog posts than last year - that's easy to measure (this will be number 13 this year, compared to 14 for the whole of last year), and now I'm aiming to improve my quality as well as quantity.

- Various VHS to DVD conversions for friends (family weddings, for example)

A couple of pieces of music (in draft)
And I am consuming 
less - yes.  Less TV, for sure.  I'm reading instead - and mostly non-fiction.

4.  Repair, not replace

This hasn't been on my mind as much as the others (I had to look it up to remember what it was); I've repaired various toys for my children, and made various fixes around the house, but I haven't consciously repaired anything I'd otherwise have thrown away.

With one exception:  one of my pairs of jeans developed a small hole, and so I decided to patch it up.  It was only a small hole that wasn't immediately obvious, so only needed sewing back together and a small patch. I completed the repair with a small patch on the inside to hold my sewing together.  I then realised that the jeans were actually too small, so they went in the charity shop bag.  Repaired, not replaced, and then given away: two for the price of one! :-)

I will provide another update around Christmas time (when I shall be able to work on my giving!).

Thursday, 22 June 2017

The General Election (Inferences from Quantitative Data)

The Election

The UK has just had a general election: all the government representatives who sit in the House of Commons have all been selected by regional votes.  The UK is split into 650 areas, called constituencies, each of which has an elected Member of Parliament (MP). Each MP has been elected by voting in their constituency, and the candidate with the highest number of votes represents that constituency in the House of Commons.

There are two main political parties in the UK - the Conservative party (pursuing centre-right capitalist policies, and represented by a blue colour), and the Labour party (which pursues more socialist policies, and represented by as red colour).  I'll skip the political history, and move directly to the data:  the Conservative party achieved 318 MPs in the election; the Labour party achieved 262; the rest were spread between smaller parties. With 650 MPs in total, the Conservative party did not achieve a majority and have had to reach out to one of the smaller parties to reach the majority they require to obtain a working majority.

Anyway:  as the results for most of the constituencies had been announced, the news reporters started their job of interviewing famous politicians of the past and present.  They asked questions about what this meant for each political party; what this said about the political feeling in the country and so on.

And the Conservative politicians put a brave face on the loss of so many seats.  And the Labour politicians contained their delight at gaining so many seats and preventing a Conservative majority.

The pressing issue of the day is Brexit (the UK's departure from the European Union).  Some politicians said, "This tells us that the electorate don't want a 'hard' Brexit [i.e. to cut all ties completely with the EU], and that they want a softer approach." - views that they held personally, and which they thought they could infer from the election result.  O
thers said, "This shows a vote against austerity,"; "This vote shows dissatisfaction with immigration." and so on.

The problem is:  the question on election day is not, "Which of these policies do you like/dislike?" The question is, "Which of these people do you want to represent you in government?"   Anything beyond that is guesswork and supposition - whether that's educated, informed, biased, or speculative.

Website Data

There's a danger in reading too much into quantitative data, and especially bringing your own bias (intentionally or unintentionally) to bear on it.  Imagine on a website that 50% of people who reach your checkout don't complete their purchase.  Can you say why?

- They found out how much you charge for shipping, and balked at it.
- They discovered that you do a three-for-two deal and went back to find another item, which they found much later (or not at all)
- They got called away from their computer and didn't get chance to complete the purchase
- Their mobile phone battery ran out
- They had trouble entering their credit card number

You can view the data, you can look at the other pages they viewed during their visit.  You can even look at the items they had in their basket.  You may be able to write hypotheses about why visitors left, but you can't say for sure.  If you can design a test to study these questions, you may be able to improve your website's performance.  For example, can you devise a way to show visitors your shipping costs before they reach checkout?  Can you provide more contextual links to special offers such as three-for-two deals to make it easier for users to spend more money with you?  Is your credit card validation working correctly?  No amount of quantitative data will truly give you qualitative answers.

A word of warning:  it doesn't always work out as you'd expect.

The UK, in its national referendum in June 2016, voted to leave the EU.  The count was taken for each constituency, and then total number of votes was counted; the overall result was that "leave" won by 52% to 48%.  

However, this varied by region, and the highest leave percentage was in Stoke-on-Trent Central, where 69% of voters opted to leave.  This was identified by the United Kingdom Independence Party (UKIP) and their leader, Paul Nuttall, took the opportunity to stand as a candidate for election as an MP in the Stoke-on-Trent Central constituency in February 2017.  His working hypothesis was (I assume) that voters who wanted to leave the EU would also vote for him and his party, which puts forward policies such as zero-immigration, reduced or no funding for overseas aid, and so on - very UK-centric policies that you might imagine would be consistent with people who want to leave a multi-national group.  However, his hypothesis was disproved when the election results came in:

Labour Party - 7853
UKIP (Paul Nuttall) - 5233

Conservative Party - 5154
Liberal Democrat Party - 2083

He repeated his attempt in a different constituency in the General Election in June; he took 3,308 votes in Boston and Skegness - more than 10,000 fewer votes than the party's result in 2015.  Shortly afterwards, he stood down as the leader of UKIP.

So, beware: inferring too much from quantitative data - especially if you have a personal bias - can leave you high and dry, in politics and in website analysis.

Wednesday, 31 May 2017

Collatz Conjecture: 3n+5