The BBC Micro and sums of 2^n

Our first home computer was an Acorn Electron. Not as widely known as the Commodore 64 or the Sinclair Spectrum, the Electron is most easily described as a cut-down version of the BBC Micro (which was extremely common in schools).  The Electron ran BBC BASIC, which made it relatively user-friendly. There was a full BASIC manual in the box, along with a book containing some relatively simple games and programs, and using the two together it was relatively easy to enter and the customise the games.

In particular, it was possible to modify the games' graphics, and the character sprites.  Each component of the game (a gameplay character) was made either from a combination of simple ASCII characters >->  o<-< or from user-defined graphics.  And the way that you defined the sprites used some simple maths based on a grid and the powers of 2 (1, 2, 4, 8, 16, etc.).  The grid is 8x8 pixels, and each column has a value.  The sum of each row gives its overall value.

Here's an example:

In order to program one of the game sprites, you take each row one at a time. You identify which of the columns have been filled for that row, and then sum their values (based on the column heading).  The first row in the example above has the two central squares shaded, and these have the values 16 and 8; the sum is 24.  The next row has four squares shaded, and these are 32, 16, 8 and 4.  The next has the central six squares, where 64+32+16+8+4 = 126, and the fourth row has all eight squares, the sum is 255.

The heart shape has some new and some familiar combinations, but the same principle applies.

And this walking stick-man has an asymmetrical arrangement, but this is no problem as the same principles apply.

Admittedly I was 8-10 years old when I was programming these sprites, but it never occurred to me that each combination was represented ny a unique sum, and conversely, each total could only be shown by one combination of pixels.

Why? In base 10, each number can only be expressed as a unique sum of 10^0s, 10^1s, 10^2s... and so on.  We call these units, tens and hundreds, and each can hold a value between 0 and 9. So, 9 x 10^0 is 9, and if we want to show an additional unit, we have to add this as an increment to the next column, and have 1x10^1 + 0 x 10^0.  Each number can (and must) be represented by a unique combination of digits in the hundreds, tens and units columns.

The same applies in binary, base 2, which is the underlying basis of the pixel grid. Each pixel can be either 1 or 0, with 1 representing a pixel, and 0 representing a gap.  And every number can be represented by a unique combination of 1s and 0s - binary is just the same as decimal counting, its just the digits that are different!  This time, the values of each column aren't 10^0, 10^1, 10^2 and so on, but 2^0, 2^1, 2^2, 2^3, 2^4 and so on. 

Naturally, the Acorn Electron was also fluent in hexadecimal, but that's a whole other story.

How many of your tests win?

 As November heads towards December, and the end of the calendar year approaches, we start the season of Annual Reviews.  It's time to identify, classify and quantify our successes and failures opportunities from 2022, and to look forward to 2023.  For a testing program, this usually involves the number of tests we've run, and how many recipes were involved; how much money we made and how many of our tests were winners.

If I ask you, I don't imagine you'd tell me, but consider for a moment:  how many of your tests typically win?  How many won this year?  Was it 50%?  Was it 75%?  Was it 90%?  And how does this reflect on your team's performance?

50% or less

It's probably best to frame this as 'avoiding revenue loss'.  Your company tested a new idea, and you prevented them from implementing it, thereby saving your company from losing a (potentially quantifiable) sum of money.  You were, I guess, trying some new ideas, and hopefully pushed the envelope - in the wrong direction, but it was probably worth a try.

Around 75%

If 75% of your tests are winning, then you're in a good position and probably able to start picking and choosing the tests that are implemented by your company.  You'll have happy stakeholders who can see the clear incremental revenue that you're providing, and who can see that they're having good ideas.

90% or more

If you're in this apparently enviable position, you are quite probably running tests that you shouldn't be.  You're probably providing an insurance policy for some very solid changes to your website; you're running tests that have such strong analytical support, clear user research or customer feedback behind them that they're just straightforward changes that should be made.  Either that, or your stakeholders are very lucky, or have very good intuition about the website.  No, seriously ;-)

Your win rate will be determined by the level of risk or innovation that your company are prepared to put into their tests.  Are you testing small changes, well-backed by clear analytics?  Should you be?  Or are you testing off-the-wall, game-changing, future-state, cutting edge designs that could revolutionise the online experience? 

I've said before that your test recipes should be significantly different from the current state - different enough to be easy to distinguish from control, and to give you a meaningful delta.  That's not to say that small changes are 'bad', but if you get a winner, it will probably take longer to see it.

Another thought:  the win rate is determined by the quality of the test ideas, and how adventurous the ideas are, and therefore the win rate is a measure of the teams who are driving the test ideas.  If your testing team is focused on test ideas and has strengths in web analytics and customer experience metrics, then your team will probably have a high win rate.  Conversely, if your team is responsible for the execution of test ideas which are produced by other teams, then a measure of test quality will be on execution, test timing, and quantity of the tests you run.  You can't attribute the test win rate (high or low) to a team who develop tests; in fact, the quality of the code is a much better KPI.

What is the optimal test win rate?  I'm not sure that there is one, but it will certainly reflect the character of your test program more than its performance. 

Is there a better metric to look at?   I would suggest "learning rate":  how many of your tests taught you something? How many of them had a strong, clearly-stated hypothesis that was able to drive your analysis of your test (winner or loser) and lead you to learn something about your website, your visitors, or both?  Did you learn something that you couldn't have identified through web analytics and path analysis?  Or did you just say, "It won", or "It lost" and leave it there?  Was the test recipe so complicated, or contain so many changes, that isolating variables and learning something was almost completely impossible?

Whatever you choose, make sure (as we do with our test analysis) that the metric matches the purpose, because 'what gets measured gets done'.