Calculator Fun and Games 2: More Multiplications

My continued mining of the paperback Calculator Fun and Games by Ben Hamilton continues.  My previous post looked at multiplication calculations where the product of a multiplication contained the same digits as the two numbers being multiplied; for example: 2 * 8741 = 17482  leading to numerical anagrams.  In this post, I move forwards a few pages through the book to a similar set of questions, along with the question, "What do you notice?"

Calculators at the ready: here are some more multiplications:

Here goes:

8 * 473 = 3784

9 * 351 = 3159
15 * 93 = 1395
21 * 87 = 1827
27 * 81 = 2187
35 * 41 = 1435

In this case, they're all four-digit numerical anagrams.  I mentioned in previous post that my conversations with AI led me to find a four-digit numerical anagram of this kind - mine was the 27 * 81 = 2187.  So, are there any more?

Yes:  Copilot has found a few (possibly through brute force?)

21*60 = 1260
30*51 = 1530
80*86 = 6880

These are the two-digit * two digit pairs.

There are some single-digit * three-digit pairs that can be added to the list:

3* 501 = 1503
3* 510 = 1530
5* 251 = 1255
6* 201 = 1206
6* 210 = 1260
8 * 860 = 6880

Copilot has indicated that these are ALL the pairs; and it can also suggest a Python or BASIC loop that tries all the combinations of digits in turn (brute force).  It doesn't, however, provide much output on identifying patterns in the numbers.  It helpfully points out that a is a single-digit number, and the other number, b, is a three-digit number.  Their product, a *b is a four-digit number.

I'd call out 6880, which features twice (8*860 and  80*86).  However, that's only because it's possible to move the zero between either a or b.  The same applies to 3*510 (30*51) and 6*210 (60*21).

Otherwise, there's no obvious pattern - these numbers only produce numerical anagrams because of a certain coincidence that occurs in base 10.

It's still fun, though!

Other Calculator Fun and Games articles:

Snakes and Ladders (Collatz Conjecture)
Crafty Calculator Calculations (numerical anagrams, five digits)
More Multiplications (numerical anagrams, four digits)
Over and Out (reduce large numbers to zero in as few steps as possible)

Calculator Fun and Games: Crafty Calculator Calculations

I have written on a number of occasions about the book 'Calculator Fun and Games' by Ben Hamilton, which my parents gave to me as a Christmas present in the mid 1980s.  I'd always found maths interesting, but this book (and a calculator) opened up a world of puzzles and fun in maths.  I have revisited this book periodically, mining it for blog articles, and here I am again, going through it page by page looking at the underlying maths behind some of the games and puzzles it contains.  One such example is Snakes and Ladders (the Collatz Conjecture) which I covered in an article back in 2012.

Another of the puzzles is known simply as 'Crafty Calculator Calculations', and is a series of multiplications, with the question, "What do you notice about these numbers?"

Here are the multiplications.  I shall resist the temptation to throw them all into a spreadsheet, and in the spirit of the book will be using my calculator (a Casio fx-85MS which is about 20 years old).  After all, that's what they idea is - and the book does specify, "Try them on your calculator," so that's what I'm doing.

Section 1: The answers as written

2 * 8714 = 17428
2 * 8741 = 17482
3 * 4128 = 12384
3 * 4281 = 12843
3 * 7125 = 21375
3 * 7251 = 21753
6 * 7251 = 43506 (?)
8 * 4973 = 39784
8 * 6521 = 52168
9 * 7461 = 67149
14 * 926 = 12964
24 * 651 = 15624
42 * 678 = 28476
51 * 246 = 12546
78 * 624 = 48672
87 * 435 = 37845
75 * 231 = 17325
65 * 281 = 18265
65 * 983 = 63895
72 * 936 = 67392

What do I notice?  With the exception of a probable typo that I've highlighted, each of the answers is a numerical anagram of the two numbers being multiplied together.

Extension work 1: Fixing the typo

Firstly, 6 * 7251 = 43506, which is not a numerical anagram.  

Hypothesis:  the 7251 is a typo, having appeared in the previous line.  The six is probably correct, since it comes between a series of threes and before a pair of eights.

I've tried multiplying 7251 by 4, 5, 6 and 7, and none of them produce numerical anagrams, so the 7251 is definitely the typo.

Hypothesis: Fixing the typo is going to be difficult, unless I can find some patterns in the larger of the two numbers which are being multiplied.  Is there a pattern?

What do these numbers have in common?
8741   - prime
8714 - not prime
7461 - not prime
926 - not prime

So the answer is - not much.  It's more like a random distribution, although 1 and 7 appear quite frequently (because any number y multiplied by another number that ends in 1 will have the digit y at the end).

I can find (with relative ease) examples where I can get an anagram of four of the five digits:

5 * 6321 = 31605 (containing a zero instead of a two)
6 * 5321 = 31926 (containing a nine instead of a five)

But I haven't yet found one that will replace the error in the original question.

So the question is:  Is the initial set a complete set with no further triplets of numbers?

It's unlikely that this is a complete set - they're such an arbitrary bunch that it's highly possible there are many more.  But is there a way of making this a systematic search instead of just random guessing?

- the units of the larger number are either 1, 3, 4, 5, 6 or 8

- the smaller number and larger number have to multiply together to form a five-digit number.  3 * 2894 will only produce another four-digit number, for example.

The answer (through only trial and improvement) is that this is NOT a complete set of triplets.  For example:

5 * 2519 = 12595 is an example not included in the original list.

Four digits:  27 * 81 = 2187 (that's what you get for asking AI for help, and not specifying that you want a five-digit number).

Conclusions:

AI says this is an interesting exercise in combinatorics, but didn't actually add anything to the search (it would have taken longer to explain the question to it than it would to have found more myself). 

"There is a specific quantity of numbers that satisfy the requirement a *b = c where c is a numerical anagram of a and b, containing each of the digits of a and b once each, and where c is a five-digit number."

But can I fix the typo in the original series of questions?  No, and I have tried! 

Other articles in the Calculator Fun and Games series:

More Multiplications (numerical anagrams, four digits) - the sequel to this article!
Over and Out (reduce large numbers to zero as rapidly as possible)
Calculator Games: Front to Back
Snakes and Ladders (Collatz Conjecture)
Calculator Fun and Games: The Kaprekar Constant

Solving PEMDAS Problems

PEMDAS Problems

PEMDAS problems are a common challenge for anyone using math – especially when you're learning PEMDAS in school for the first time, and without much real-life context. PEDMAS stands for Parentheses, Exponents, Multiplication, Division, Addition, and Subtraction. It's a set of rules that tell you the order to follow when solving a math problem to get the right answer, and an answer which mathematicians can agree on internationally.  PEDMAS is the term in the US; BODMAS is the term used pretty much everywhere else.  I can't help wondering if this is going to end badly.

• Parentheses always come first. You must calculate everything inside the parentheses before moving on.
• Exponents (like squares, cubes, or square roots) are next.
• After that, it's the arithmetic functions: Divide, Multiply, Add, Subtract.  In BODMAS, Division always comes first.  For some strange reason I have not been able to fathom, in PEMDAS, you just do everything from left to right.  It's sure to cause confusion, and a quick look at social media will tell you that.  How can it not, if 3 *4 /3+2 is PEMDAS does 3*4 and then divide by three, and then add two?  If it's BODMAS then you do the division first, even though it's not on the left.  Let's leave that for a whole other discussion.

Why is PEMDAS Important?

Imagine you have this expression: 3 * 4 - 2

• Without PEMDAS, it's unclear whether you should multiply 3 by 4 first, or subtract 2 from 4.
• PEMDAS helps avoid confusion and makes sure everyone gets the same answer.  It's worth pointing out at this point that real mathematicians, who want to communicate unambiguously, always want to avoid confusion.  And by real mathematicians, I mean anybody except the fools who put stupid questions on social media just for the conflict.
  

Image from Hotpot

Let's Look at Some Examples:

1. (3 * 4) - 2

   • Parentheses first: 3 * 4 = 12
   • Subtraction: 12 - 2 = 10

2. 3 * (4 - 2)

   • Parentheses first: 4 - 2 = 2
   • Multiplication: 3 * 2 = 6

3. 4 * (6 - 2²)

   • Parentheses first:
     • Exponents: 2² = 4
     • Subtraction: 6 - 4 = 2
   • Multiplication: 4 * 2 = 8

Why Does PEMDAS Sometimes Cause Trouble?

• Tricky Online Problems: Some people intentionally write confusing problems (like 1 + 7 * 3) to see if others can solve them correctly.  There is a correct answer, but the ambiguity is a bigger problem than the math.
• Everyday Math: Even simple situations can involve PEDMAS. For example:
  • If each car has 4 wheels and is towing a caravan with 2 wheels, how many wheels are there in total for 4 cars?
    • Correct: 4 * (4 + 2) = 24 wheels 
    • Incorrect: (4 * 4) + 2 = 18 wheels.  Four wheels on four cars, and two for a caravan is not enough.

PEMDAS helps us avoid these kinds of mistakes and ensures we get the correct answer every time!

Image by Hotpot

Other online reading

The Math Equation Solver (will handle PEMDAS problems)

Other Calculator Fun and Games articles:

Snakes and Ladders (Collatz Conjecture)
Crafty Calculator Calculations (numerical anagrams with five digits)
More Multiplications (numerical anagrams, four digits)
Over and Out (reduce large numbers to zero as rapidly as possible)
Calculator Games: Front to Back
Calculator Games: Up, up and away with Ulam sequences
Calculator Games: The Kaprekar Constant

First times of 2024

You may think that as you get older, you do fewer and fewer things for the first time.  But actually, no, that's not the case.  And here's the list of things I can remember doing for the first time in 2024 - in no particular order.

First time keeping a close eye on how much chocolate I eat, and cutting beetroot out of my diet completely.  In 2023, I was diagnosed with kidney stones - not serious, but potentially painful.  They had been the cause of some discomfort for me in the second half of 2023, and I had various medical checks; the recommended treatment for them is to manage them through my diet.  Anything high in oxalates is out; anything with medium levels has to be taken in moderation.  So, I've not had any beetroot for the whole year, for the first time ever (this was a sad loss for me, I love pickled beetroot) and I've watched my chocolate intake.  I don't think I had any chocolate in January or February, then I lightened up a little.  It's not all bad news, so don't sell your shares in Cadbury's just yet!

First time seeing the Northern Lights.  Twice.  The first time I didn't actually think I'd seen them - they looked more like light pollution on the north horizon.  The second time, there was no mistaking them, even if they were still quite faint.

First time leaving the kids overnight with a babysitter, while spending a night with Naomi at a friend's wedding.  That was during the summer, and was a Friday night/Saturday morning.  We didn't go far, which made it an ideal first time for us to leave them altogether for most of Friday and most of Saturday morning.  They went to the cinema together on the Friday; we went to a wedding, meal and disco.  I think we had the best of it.

First time going to an airshow and giving somebody else my camera to take photos.  It worked out well - I got some mediocre video (a combination of the viewfinder on my camera, and the dreadful weather) while Liz took some amazing photos with the stormy skies.

First time using generative AI to produce a picture... and then produced dozens more.  I've also used generative AI (Chat GPT) to write BASIC programs for me.  I have marvelled at the ability of AI to turn text into images, and then got on and started using it without thinking too much about it!

First time laying loft boarding (successfully too).  Our loft is in two sections - the main part and a smaller section.  For some unknown reason, the smaller section (which has its own separate access) was never boarded over, but was very well stocked with fibreglass insulation.  In November, I boarded over most of the smaller section - enough to house our Christmas decorations when Christmas is finished.  That frees up more space in our main loft... for my train set! :-D

First time going on board a warship - the HMS Belfast in London.  We went on a family trip during October half term and spent three days in London; I took the boys round the interesting sights, while Lizzie and Naomi did the art galleries and Shakespeare's Globe.  We saw some locations we hadn't seen before, including HMS Belfast, which is highly recommended and free if you have Blue Peter badges (the boys do).  I went on a tour of a submarine when I was about six, but considering a submarine is all 'inside' with no windows, I can safely say the warship was far more interesting, with a strange alternating feeling of vertigo (on deck) and claustrophobia (inside).  

First time buying a car online, after the previous one started producing more smoke than one of the Red Arrows during their display.  We drove about four miles back home one Saturday afternoon, billowing huge clouds of bluish-white smoke, and deterring all other motorists within 300 yards.  Within 10 days we'd needed to order a replacement - online.  Same model, newer interior.  It's better than the old one too, with more gadgets inside.

First time playing a massive multiplayer online game - Fortnite - and subsequently the first time I bought my own gaming controller.  I've been a keyboard warrior since the mid 80s (almost before the internet was created) and then migrated to a mouse when I started playing real-time strategy games in the early 90s.  I had been borrowing my son's controller throughout January and most of February, and bought myself my own controller with my birthday money in mid February.  It's taken some minor damage to the cables (in early December) and still works perfectly, even if I'm not as good as my sons at playing!

First time learning, and coding in Python.  My programs are not complicated, and they all work on a local PC without any network access, but it's been very interesting and challenging.  I have considerable experience writing in BASIC and Visual Basic, and this was just the next step - learning a new syntax.  I wrote a program which will calculate how long people will have to wait in a queue at a supermarket, and then calculated the time savings from opening two (or more) checkouts at a supermarket compared to just one.  The results were very surprising, and I actually enjoyed analysing the output of the program.  More to follow next year.

In 2022, I became a Trustee for my church - overseeing the running of the administration of the church (legal compliance, health and safety and so on).  In 2024, I was asked and then was nominated to become the Treasurer of my local Scout group.  No, I don't know exactly what I'm doing, but I can operate a spreadsheet with ease, and I'm getting there.

First time going to a gym.  If you've met me, you can tell from my physique that I'm not a regular gym-goer.  Or ever been to a gym, even.  I was described during my teenage years as a bean-pole, and I've not put on much weight or muscle since then (except when I was at university, cycling six miles a day and becoming very trim indeed).  However, December at my local gym was 'bring a friend to the gym' month, so Lizzie brought me.  I've been three times this month, and I've still got the moves to cover 13 miles in an hour on an exercise bike.  It's not quite as scenic as being on a real bike (took Lizzie on a bike ride for the first time ever this year too) but there's less traffic and the weather is more amenable.

First time running on a treadmill - Isaac decided last December that he wanted one (he's a miniature runner), and so last New Year's Eve we bought him a cheap second-hand one.  I used it more than he did, but even so, it wasn't worth the space it took up.  It was fun while it lasted!

The Lists of Firsts

A first time for everything: 2018
2019 in reflection
First times in 2021 list
First times of 2022
First times in 2023
Things I did for the first time in 2024

BODMAS Problems

 BODMAS problems are a recurring challenge throughout the life of a mathematician - especially at school or college.  

If you use PEMDAS instead of BODMAS, you might find my article on PEMDAS Problems helpful instead.

BODMAS stands for Brackets, Operators, Division, Multiplication, Addition and Subtraction, and sets out the order in which you must work through a maths problem to get to the correct answer.  Brackets (called Parentheses in the US) always come first, and you have to calculate what's inside them before you do anything else.  Operators are 'powers' or 'indices' and include squares, cubes, square roots and cube roots.  The rest are the normal arithmetic calculations.

For example (3*4)-2 is not the same as 3*(4-2).  And without the brackets, you'd be looking at just 3*4-2 which is ambiguous.  It's not clear which way round you're supposed to do the calculations.  What do you do first?

Mathematicians don't make these rules up to be awkward or difficult.  Mathematicians hate confusion, ambiguity and uncertainty, and therefore they use BODMAS and brackets and all these rules so that when they talk to other mathematicians anywhere in the world, they are entirely clear what they're talking about.  They are even more specific and precise than scientists (in my experience) and will take great care to make sure that they are crystal clear about their calculations.

In my experience, mathematicians who understand BODMAS problems don't get involved in the BODMAS questions that go around on social media, where there are no brackets and a whole flame war kicks off between people who defend their answer to 1+5*3.  These are badly-written problems, written that way on purpose.

Let's take a look at some simple examples of BODMAS problems, and identify some of the possible pitfalls along the way:

a) Let's start with (3*4)-2

The brackets here clearly indicate that we should calculate 3*4 first.  3*4 = 12.  12 -2 = 10.

In fact, the brackets aren't required here - Multiplication always comes before Subtraction.

Let's change the position of the brackets:

3* (4-2)

Without the brackets, we'd do 3*4 first, but with the brackets, we must calculate 4-2 first.

4-2 = 2

3* 2 = 6

So there's a clear distinction between 10 and 6.  We must be careful with our BODMAS problems.

Another one:

b)  4 * (6-2²)

First is Brackets.  We must calculate the contents of the brackets, which is 6-2².
Within these brackets, we must do the Operator first.  Operators are squares, square roots, and any other powers.  In this question, the operator is the 2 squared.

2² = 4
6-4 = 2

So the contents of the Brackets comes to 2.

4*2 = 8

Let's compare this with the same calculation without the brackets, and use BODMAS to find out the value of 4 * 6-2²

BODMAS says that we do the Operators first (there are no Brackets in this question), and we know that 2² = 4

This gives us:  4 * 6 - 4 

Next, we do the multiplication, so 4 *6 = 24
And finally, the subtraction, 24 -4 = 20

So now we have a difference between
 4 * (6-2²) = 8
4 * 6 - 4 = 20 

Clearly we're going to have to be careful!

c) This time, let's do a calculation with algebra:

x²  + (2x * 6x) + (x -z)

Remembering to do our brackets first, we get:

x²  + 12x² + x - z

Operators: we can't do anything further with these at this stage; all that remains now are Addition and Subtraction.  Addition comes first:

x²  + 12x² + x - z = 13x²  + x - z

And we can't simplify this any further.

Why does BODMAS cause problems?

BODMAS expressions, like 3* (3+5+6) can occur even in simple everyday maths situations.

For example, how many wheels are there in total with on four four-wheeled cars if each car is carrying a caravan which has two wheels?

A car has four wheels; a caravan has two wheels.  That's six wheels.  And there are four cars and caravans, so that's 4 * 6 = 24 wheels altogether.

However, if you misunderstood or miscalculated, you might work like this:

There are four cars, each with four wheels.  That's sixteen wheels.  And two wheels for the caravan makes 18 wheels.

4 * (4+2) is very different from (4*4) + 2.  And that's why we have BODMAS - to help us avoid problems and misunderstandings, in cases from cars and caravans to planets and stars!

If you enjoyed this article, you may also be interested in some of the other articles I've written about mathematical subjects:

BODMAS on Social Media - What's The Fuss?
BODMAS and Badly Written Maths Questions
Snakes and Ladders (the Collatz Conjecture)
Crafty Calculator Calculations (numerical anagrams with five digits)

More Multiplications (numerical anagrams, four digits)
Over and Out (reduce large numbers to zero in as few steps as you can)
Calculator Games: Front to Back
Calculator Games: Up, up and away with Ulam sequences
Calcualtor Games: The Kaprekar Constant

Testing versus Implementing - why not just switch it on?

"Why can't we just make a change and see what happens? Why do we have to build an A/B test - it takes too long!  We have a roadmap, a pipeline and a backlog, and we haven't got time."

It's not always easy to articulate why testing is important - especially if your company is making small, iterative, data-backed changes to the site and your tests consistently win (or, worse still, go flat).  The IT team is testing carefully and cautiously, but the time taken to build the test and run it is slowing down everybody's pipelines.  You work with the IT team to build the test (which takes time), it runs (which takes even more time), you analyze the test (why?) and you show that their good idea was indeed a good idea.  Who knew?

Ask an AI what a global IT roadmap looks like...

However, if your IT team is building and deploying something to your website - a new way of identifying a user's delivery address; or a new way of helping users decide which sparkplugs or ink cartridges or running shoes they need - something new, innovative and very different, then I would strongly recommend that you test it with them, even if there is strong evidence for its effectiveness.  Yes, they have carried out user-testing and it's done well.  Yes, their panel loved it.  Even the Head of Global Synergies liked it, and she's a tough one to impress.  Their top designers have spent months in collaboration with the project manager, and their developers have gone through the agile process so many times that they're as flexible as ballet dancers.  They've barely reached the deadline for pre-Christmas implementation, and now is the time to implement it.  It is ready.  However, the Global Integration Leader has said that they must test before they launch, but that's okay as they have allocated just enough time for a pre-launch A/B test, then they'll go live as soon as the test is complete.

Sarah Harries, Head of Global Synergies

Everything hinges on the test launching on time, which it does.  Everybody in the IT team is very excited to see how users engage with the new sparkplug selection tool and - more importantly for everybody else - how much it adds to overall revenue.  (For more on this, remember that clicks aren't really KPIs). 

But the test results come back: you have to report that the test recipe is underperforming at a rate of 6.3% conversion drop.  Engagement looks healthy at 11.7%, but those users are dragging down overall performance.  The page exit rate is lower, but fewer users are going through checkout and completing a purchase.  Even after two full weeks, the data is looking negative.  

Can you really recommend implementing the new feature?  No; but that's not the end of the story.  It's your job to now unpick the data, and turn analysis into insights:  why didn't it win?!

The IT team, understandably, want to implement.  After all, they've spent months building this new selector and the pre-launch data was all positive.  The Head of Global Synergies is asking them why it isn't on the site yet.  Their timeline allowed three weeks for testing and you've spent three weeks testing.  Their unspoken assumption was that testing was a validation of the new design, not a step that might turn out to be a roadblock, and they had not anticipated any need for post-test changes.  It was challenging enough to fit in the test, and besides, the request was to test it.

It's time to interrogate the data.

Moreover, they have identified some positive data points:

*  Engagement is an impressive 11.7%.  Therefore, users love it.

*  The page exit rate is lower, so more people are moving forwards.  That's all that matters for this page:  get users to move forwards towards checkout.

*  The drop in conversion is coming from the pages in the checkout process.  That can't be related to the test, which is in the selector pages.  It must be a checkout problem.

They question the accuracy of the test data, which contradicts all their other data.

* The sample size is too small.

* The test ran for too long/did not run for long enough

* The test was switched off before it had a chance to recover its 6.3% drop in conversion

They suggest that the whole A/B testing methodology is inaccurate.

* A/B testing is outdated and unreliable.  

* The split between the two groups wasn't 50-50.  There are 2.2% more visitors in A than B.

Maybe they'll comment that the data wasn't analyzed or segmented correctly, and they make some points about this:

* The test data includes users buying other items with their sparkplugs.  These should be filtered out.

* The test data must have included users who didn't see the test experience.

* The data shows that users who browsed on mobile phones only performed at -5.8% on conversion, so they're doing better than desktop users.

Remember:  none of this is personal.  You are, despite your best efforts, criticising a project that they've spent weeks or even months polishing and producing.  Nobody until this point has criticised their work, and in fact everybody has said how good it is.  It's not your fault, your job is to present the data and to provide insights based on it.  As a testing professional, your job is to run and analyse tests, not to be swayed into showing the data in a particular way.

They ran the test at the request of the Global Integration Leader, and burnt three weeks  waiting for the test to complete.  The deadline for implementing the new sparkplug selector is Tuesday, and they can't stop the whole IT roadmap (which is dependent on this first deployment) just because one test showed some negative data.  They would have preferred not to test it at all, but it remains your responsibility to share the test data with other stakeholders in the business, marketing and merchandizing teams, who have a vested interest in the site's financial performance.  It's not easy, but it's still part of your role to present the unbiased, impartial data that makes up your test analysis, along with the data-driven recommendations for improvements.

It's not your responsibility to make the go/no-go decision, but it is up to you to ensure that the relevant stakeholders and decision-makers have the full data set in front of them when they make the decision.  They may choose to implement the new feature anyway, taking into account that it will need to be fixed with follow-up changes and tweaks once it's gone live.  It's a healthy compromise, providing that they can pull two developers and a designer away from the next item on their roadmap to do retrospective fixes on the new selector.  Alternatively, they may postpone the deployment and use your test data to address the conversion drops that you've shared.  How are the conversion drop and the engagement data connected?  Is the selector providing valid and accurate recommendations to users?  Does the data show that they enter their car colour and their driving style, but then go to the search function when they reach a question about their engine size?  Is the sequence of questions optimal?  Make sure that you can present these kinds of recommendations - it shows the value of testing, as your stakeholders would not be able to identify these insights from an immediate implementation.

So - why not just switch it on?  Here are four good reasons to share with your stakeholders:

* Test data will give you a comparison of whole-site behaviour - not just 'how many people engaged with the new feature?' but also 'what happens to those people who clicked?' and 'how do they compare with users who don't have the feature?'

* Testing will also tell you about  the financial impact of the new feature (good for return-on-investment calculations, which are tricky with seasonality and other factors to consider)

*  Testing has the key benefit that you can switch it off - at short notice, and at any time.  If the data shows that the test recipe is badly losing money then you identify this, and after a discussion with any key stakeholders, you can pull the plug within minutes.  And you can end the test at any time - you don't have to wait until the next IT deployment window to undeploy the new feature. 

* Testing will give you useful data quickly - within days you'll see how it's performing; within weeks you'll have a clear picture.

Similar posts I've written about online testing

Getting an online testing program off the ground
Building Momentum in Online testing
How many of your tests win?

Designing Personas for Design Prototypes

Part of my job is validating (i.e. testing and confirming) new designs for the website I work on.  We A/B test the current page against a new page, and confirm (or otherwise) that the new version is indeed better than what we have now.  It's often a last-stop measure before the new design is implemented globally, although it's not always a go/no-go decision.

The new design has gone through various other testing and validation first - a team of qualified user experience designers (UX)  and user interface designers (UI) will have decided how they want to improve the current experience.  They will have undertaken various trials with their designs, and will have built prototypes that will have been shown to user researchers; one of the key parts of the design process, somewhere near the beginning, is the development of user personas.

A persona in this context is a character that forms a 'typical user', who designers and product teams can keep in mind while they're discussing their new design.  They can point to Jane Doe and say, "Jane would like this," or, "Jane would probably click on this, because Jane is an expert user."

I sometimes play Chess in a similar way, when I play solo Chess or when I'm trying to analyze a game I'm playing.  I make a move, and then decide what my opponent would play.  I did this a lot when I was a beginner, learning to play (about 40 years ago) - if I move this piece, then he'll move that piece, and I'll move this piece, and I'll checkmate him in two moves!  This was exactly the thought process I would go through - making the best moves for me, and then guessing my opponent's next move.

It rarely worked out that way, though, when I played a real game.  Instead, my actual opponent would see my plans, make a clever move of his own and capture my key piece before I got chance to move it within range of his King.

Underestimating (or, to quote a phrase, misunderestimating) my opponent's thoughts and plans is a problem that's inherent with playing skill and strategy games like Chess.  In my head, my opponent can only play as well as I can. 

However, when I play solo, I can make as many moves as I like, but both sides can do whatever I like, and I can win because I constructed my opponent to follow the perfect sequence of moves to let me win.  And I can even fool myself into believing that I won because I had the better ideas and the best strategy.

And this is a common pitfall among Persona Designers (I've written a whole series on the pitfalls of A/B testing).  They impose too much of their own character onto their persona, and suddenly they don't have a persona, they have a puppet.

"Jane Doe is clever enough to scroll through the product specifications to find the compelling content that will answer all her questions."

"Joe Bloggs is a novice in buying jewellery for his wife, so he'll like all these pretty pictures of diamonds."

"John Doe is a novice buyer who wants a new phone and needs to read all this wonderful content that we've spent months writing and crafting."

This is something similar to the Texas Sharpshooter Fallacy (shooting bullets at the side of a barn, then painting the target around them to make the bullet holes look like they hit it).  That's all well and good, until you realize that the real customers who will spend real money purchasing items from our websites, have a very real target that's not determined by where we shoot our bullets.  We might even know the demographics of our customers, but even that doesn't mean we know what (or how) they think.  We certainly can't imbue our personas with characters and hold on to them as firmly as we do in the face of actual customer buying data that shows a different picture.  So what do we do?

"When the facts change, I change my mind. What do you do, sir?"
Paul Samuelson, Economist,1915-2009