Too Big Data?

Apparently, we are in the information age.  The Stone Age has passed, so has the Industrial Age and all that went with it.  Information is the new tool du jour, with vast quantities being produced, recorded, stored, analysed and picked apart, reconsistuted and reworked.  According to various internet sources (which are, as a type, notoriously unreliable), the current information age is unlike anything previously, with the potential to change the world (if it hasn't already).

But who's to say that any of this data is actually useful?  We may well be producing unprecedented volumes of data now, but that's only because anybody with an internet connection and a text editor can produce a blog (look at me).  Courtesy of this wonderful information age, anybody can produce a poorly-spelt, badly-punctuated and grammatically incorrect blog. 

Unfortunately, no storage system, whether it's a 5.25" floppy disk and drive, or a magneto-optical drive, or a CD-ROM or a USB memory pen or a web server, can determine the difference between quality data and meangingless drivel.  It's all stored, counted, analysed and so on.  All that we've done is provide anybody and everybody the opportunity to record the data that they had in their heads, and have it stored, and then displayed.  It's easy.  In fact, it's too easy.  I would venture that if Shakespeare had access to a blog, he would never have written to the high quality that he achieved with paper and quill.  The very act of getting ink onto paper (two substances that, despite our information age are still no closer to obsolescence) required time and thought, and his words were crafted.  Consider the time taken to create a cave painting.


Or how about the labour intensive process of hammering characters into stone tablets?  Now, I can sit here and hammer my fingers on an iPad with no real plan, producing sentence after sentence of data that will become stored, recorded and so on.  No wonder the latest craze is 'big data'.  Even if we separate the meaningful from the meaningless, the meaningful - and even the borderline cases - will require vast amounts of storage.  Do we really want to know what the girl next door had for breakfast?  Do her status updates on Facebook count as data?  Yes, they do, so no wonder we're producing more data than ever before... we're setting a pretty low bar on 'data', after all.  So, no wonder we've got big data - it's too big data if we aren't going to be discriminatory, or even selective.


As an aside, I do try to produce quality material in my blog (the web analytics, maths and science stuff especially; the film reviews less so, and the X-factor rants less so again).  I figure there's plenty of data out there, so I'm also trying to keep things fairly concise.


So, from this standpoint, I hope you'll forgive my cynicism when I hear that we are now producing more data in two days (or a year, or whatever) than was ever produced in the previous 4000 years.  We are also producing more waste, releasing more carbon dioxide, and more and more television channels than ever before.  Volume is not everything.  Quantity alone is a meaningless metric - as many in the web analytics area have pointed out before, traffic by itself is not a valuable KPI.  Which would you rather have, 10 tonnes of coal, or 10 grams of diamond?

Chess game: Sicilian Blunders

How not to play the Sicilian as White... .made a complete waste of my white-squared bishop, and got into all sorts of trouble with reckless pawn advances and finished off by not protecting my king.


Worse still, I isolated my king behind a doubled-pawn position and could not fight off black's direct attack.  The game ended very quickly after that.
Let's take a look at my biggest mistakes in this game (I'm sure they're meant to be called learning points).
By move 15, I've completely isolated my white-squared bishop.  I should surely have moved it back to c2 on move 13, to give it some hope of remaining in the game.


16. Be3 shows what a wasted move 9 Bd2 was.  I should have been more decisive earlier in the game.
18. d4 was a vast mistake.  I should have left the pawns blocked up in the middle.  As it was, I then decided to ditch my bishop (another mistake) and by move 23 my opponent has mobilised his pieces and is already hitting all the weaknesses in my pawns (and there are plenty to aim for).


24. Rc3 was a mistake.  Yes, it protects the pawns (although rooks should probably never have this duty at this point in the game), but it would have been better for me to play Rfd1 and provide my king with a way out.


From this point on, my pawns on f2 and f3 block my pieces from defending my king, and it's just a matter of time...!

Film review: The Green Hornet

THE GREEN HORNET (2011)

I can't honestly say why I put this on our Lovefilm list... I think it was a 'recent release' and, since I like superhero movies as a genre, I thought I'd put it on the list and give it a try.  Partly that, and partly that the list was getting a bit low and needing topping up.  Consequently, I had no idea of the Green Hornet's back story, and part way through the film, came to the conclusion that the whole thing was a parody of Batman (rich young bachelor with more money decides to fight crime with the aid of his trusty sidekick and some incredible gadgets).  It is only today, a couple of days after seeing the film, that I've just seen an episode of the 1960s Batman TV series and realised that the Green Hornet really is a genuine 'superhero' character.

In his appearance in the TV series, the Green Hornet appears to act as a supervillian, while acting secretly as a crime-fighter.  They've managed to carry this into the movie adaptation:  he's pretending to be a villian, while actually working to fight crime.  It's easier to see than to explain, but the Green Hornet decides to take over Los Angeles' criminal operations, with the aim of bringing them down in an illegal manner:  cue lots of shooting, explosions and so on, all done in true comic-book style.  Consequently, as his partner points out, they have the police AND the criminals chasing after them.


As complete novices in the criminal and crime-fighting worlds, the Green Hornet and Kato realise that they need help, advice and basically to be told what to do.  This comes in the form of Britt Reid's newly-hired personal assistant, Lenore Case, played by Cameron Diaz, an expert criminologist.  The would-be love triangle between Brett, Kato and Lenore is played for some very amusing scenes, and becomes a point of conflict between the would-be heroes.


Starting with the comical concept of pretending to be criminals, but working to fight crime, this film has some extremely amusing points, interspersed with some very funny scenes, and there were various points that had me laughing out loud.  There are plenty of gadgets - I'm sure many of these are based on the Green Hornet's history, so I apologise that I've no idea how relevant they are - there's the heroes' ineptitude played for laughs (in fact, a lengthy fight scene between Kato and Britt is filmed in madcap slapstick style - there's no missing what the directors were going for); and there's an extremely long car chase, involving a car getting stuck in a printing press and subsequently being driven around and office... minus its rear wheels.


There's the obligatory scene where Britt realises that his workaholic, distant father was actually a good man, working to expose a devious plot between criminals and politicians, and subsequently acts to restore his father's good name (and put the head back on his statue, but that's a whole other story), but it's deliberately played down as a serious emotional scene and is kept in line with the pacey comedy of the rest of the film.  To be honest, the whole film really does play as a parody of Batman, so I can't comment on how accurate it is to the original TV or radio series.


I'd like to discuss the final scene in the film, but I can't in too much detail (it would truly spoil the ending) but it involves Britt requiring treatment for a gunshot wound... except he's in too much pain to think rationally.  The ending is entirely in keeping with the story, and also opens the way for a possible sequel.


There are plenty of high-profile actors in supporting roles, which works well as I can't say I've ever seen the lead actors Seth Rogen (Britt Reid) and Jay Chou (Kato) in anything before.  James Franco (Harry Osborn in the Spiderman films) has a short role as a would-be crime boss, Edward James Olmos (Admiral Adama from Battlestar Galactica) features as the editor of Reid's newspaper, the Sentinel, and Tom Wilkinson (Batman Begins, Duplicity, The Full Monty) plays Reid's father.  As I mentioned, Cameron Diaz stars as Reid's personal assistant, and her experience playing comedy definitely helps here.  Everything is comic-book larger-than-life, but somehow it avoids being excessive and while completely unrealistic, manages to carry enough realism (just) to be very funny and engaging as a story.  I like.

X Factor Predictions Revisited and Updated

A few weeks ago, I listed a number of predictions about the X-Factor 2011, and here they are (so far) listed in bold with my comments.

*  At least one finalist to have estranged parent or sibling - I appreciate I'm late with this, given that immediately after the first episode, one of the judges discovered a brother she never knew she had.

This hasn't been uncovered yet, but give it time.  Let's not forget that a few weeks ago, Tulisa's long lost brother told us all about her childhood and upbringing.

*  Gary Barlow to have one of his Take That mates at the judges' house stage (and it won't be Robbie)

So that's score +1 for the Take That predicition, but -1 for suggesting it wouldn't be Robbie.  You win some, you lose some (a philosophy that might be of use to all those 'this is life or death for me' contestants).

*  One of finalists to have been bullied at school

We'll see...

*  There will be the formation of a boy group and girl group, made up of the boy dregs and girl dregs at the end of the boot camp stage.  "We want to put you together into a group [because we haven't got enough groups already]."

Yes.  It was an easy one, but it was worth mentioning.

*  These synthetic dregs-groups to go through to the live shows (you didn't think the judges would put them together and not let them go through, did you?).

And again, I was correct.  Too easy, really, but hey, some points are worth getting.

*  These synthetic groups to get eliminated in first two weeks - first the girls (who will dress inappropriately) and then the boys (who can't sing as a team)

I should qualify that I was expecting the public vote to start in week one, so give me another week here, folks!

*  Simon Cowell to make a guest appearance, to much fanfare and flashing lights

Still pending.

*  Last year's winner (whoever that was) to release album just in time for Christmas

Oh goodness me, was that really Matt Cardle hawking his new single last Sunday?  Really?  Imagine that.

*  Louis Walsh to pick a wildcard act (or just a wild act) which is no good, but which secures the votes of those who deliberately vote for the worst (Jedward, Wagner).

And this year, he's called Johnny.  It would have been Goldie too but she had the sense to leave.

*  There will be extensive media coverage of an apparent spat between two of the judges, probably the two ladies, but possibly the two blokes

Still waiting for this one, although the drama over "head judge" on The Xtra Factor was a parody in and of itself.  Nice one, ITV2.

*  One of the acts to suffer with a cough/cold/laryngitis/glandular fever part way through the TV shows

Still pending... just give them a few weeks.

*  Two of the acts to form a 'secret' relationship, again with much media coverage

I haven't worked out who this will be yet, but give me a week or so and I'll be able to suggest names.

Now for my additional predictions, having seen week one.

I wish I'd mentioned the excessive use of Orff's Carmina Burana classical piece every time something interesting (or dull) happens... like the judges walk on stage... or walk off the stage...  otherwise, I'd predict that they'd use it.

The synthetic girl group (which I've mentioned before) to wear excessively revealing clothes, and then to draw criticism for it, and then, in the same week, to get voted off.

Michael Jackson week.

Some disastrous cover versions of Take That and Westlife songs.

The judges to criticise each others' acts' "song selections" and "fashion sense" instead of the singing.

Movie Soundtrack Week.

One of the judges, probably Louis, to bend the rules on the allowed songs for "Movie Soundtrack Week" and pick a popular song that featured in an obscure movie.

One of the judges to say, "I think you could really be at risk this week," as a transparent ploy to get people to ring up and vote.  Seriously?  Do you think that the sales of the CDs and downloads comes close to the total phone revenue for the X-factor?  It's all about persuading, cajoling and manipulating people into voting.  I might start a whole other post on the manipulation of the public (and the public vote) by the judges' comments - "People really need to pick up the phone and vote for you this week" being a less subtle one, and "I think you're at risk" being slightly less obvious.

Deadlock.  Every week that it's possible, the judges will deploy deadlock instead of actually kicking off the weaker act.  Remember Jedward, hmm, and their excessively long stay on the show at the expense of acts who could sing better but didn't generate the same interest or phone votes?

I also wish I'd remembered that in the first few weeks, one of the judges usually makes a disastrous song choice and immediately dooms their act.  In 2007, it was Daniel DeBourg it was "Build Me Up Buttercup", this year it was Jonjo with "You Really Got Me."  Talk about a rabbit in the headlights (and in order to discuss rabbit in the headlights, I'd like to talk more about the ever-pale-faced Leon Jackson).

I predict that they'll release "The Charity Single" ... the money spinner now available with with 'extra' goodwill.

That's all for this time, so, keep watching (but not voting) until next time, when we'll probably discuss the allegations of phone-lines being rigged, of judges deliberately throwing their acts to the lions, and of not eliminating the right acts.

MVT: A Simplified Explanation of Complex Interactions

MVT WITH FRIDGE MAGNETS


My young daughter has developed a definite liking for Innocent Fruit Smoothies, which is great for the rest of us because she's guaranteed to get at least one of her five-a-day with every carton she drinks.  She and I also like the sets of magnets that come with special promotional packs; the current promotion is pictures of letters, but previously, it's been pictures of parts of different characters - heads, torsos and legs.  Looking at these yesterday, it occurred to me that mixing and matching the body parts was similar to optimising content in a multi-variate test, and also a good description of the difference between A/B and MVT.


In the same way as various parts of a web page can be changed, there are three parts of the characters that can be changed - the head, the torso and the legs, and there are a large number of different versions of each that can be used in the different areas.  


Here's the full collection that we currently have in our kitchen...

                           1                        2                               3                           4                                5

Now, consider building a web page with three different components - in a similar way to building a body with the three different magnets.  If we A/B/n test each of the five combinations above, then we might get the following results for each of the different components.  


Recipe 1:  350 points
Recipe 2:  475 points
Recipe 3:  420 points
Recipe 4:  430 points
Recipe 5:  320 points


And based on these scores, the winning recipe (or version, or whatever you'd like to call it) from our A/B tests is Recipe 2.  But then we'd go on to do separate A/B tests on the head, then we'd do the torso and, and then the legs.  These show that the best performing combination is Bigfoot Head, Scarecrow Torso, Astronaut Legs:

However, this only takes the results of the separate A/B tests in isolation.  Looking at the different options we have available, we can see just by looking that there's a better combination, which is this one:


This is the difference between MVT and A/B testing:  our A/B tests would not have realised that this combination would be a winning combination because they were only looking at each test by itself.  True MVT is not a series of simultaneous A/B tests, looking to improve each page component individually.  From a mathematical and scientific standpoint, the large number of combinations or recipes that are possible all need to be tested, making sure that each possible combination is included in the test.  However, this method of testing, called "full factorial", is really not feasible, and would take a very long time before the results could be confirmed, as the performance of each and every combination has to be tested.  Instead, there are various ways of testing a smaller group of the recipes, which will enable us to obtain results for each component, and to identify the best performer - even if we don't test it.  So, we'll be able to improve our testing method from simultaneous A/B testing (which has many flaws), to something which is approaching multi-variate testing.


As an example, here are some fictitious results of an MVT test series I've run, using the fridge magnets as my examples.  I've simplified the different options from the wide range I started with (just to keep things readable and understandable).  I've got the three positions - Head, Body and Legs - and I've got three different options.  


For the head, there's Egyptian, Bigfoot and Wrestler.

For the body or torso, there's Bigfoot, Scarecrow and Wrestler.

And for the feet, there's Wrestler, Robot and Astronaut.

 

Now, three different positions with three different options for each position gives us a total of 3^3 recipes, which is 27, and this would be a "full factorial" test, with the full range of recipes being tested.  However, by carrying out some MVT, it's possible to cut this down to just six tests, and here they are, with their corresponding "scores".



Test Head Torso  Feet  "Score"
1 Egyptian  Bigfoot  Wrestler  355
2 Wrestler  Bigfoot  Robot  379
3 Bigfoot  Wrestler  Wrestler  498
4 Wrestler  Wrestler  Astronaut 448
5 Egyptian  Scarecrow Astronaut 305
6 Bigfoot  Scarecrow Robot  420 


Note that each option for head, body and feet appears twice in each column, and that test 1 is the control version.  Without having to test all the versions, 


we can see from our results that Wrestler is clearly the better body - it featured in both of the highest scoring recipes.  Egyptian is also the weakest Head - it featured in the two lowest performing recipes.  


A good MVT software system will be able to determine how many tests are required to cover enough recipes and measure the effectiveness of each of its tests, and attribute these to the components of the recipe, so that it can provide the winning recipe.  Some MVT software providers, including Autonomy's Optimost software, provide an element contribution report after carrying out a round of MVT, which shows how each element affects the performance of any recipe it's included in.


For those who are interested, I used the following points system in producing my results - this is my approximate 'element contribution report'.
Head  Torso   Legs
Egyptian 75 Bigfoot  138 Wrestler  141
Bigfoot 150 Scarecrow 148 Robot  122
Wrestler 100 Wrestler  105 Astronaut 180


I deliberately adjusted the totals after summing, to highlight the effect of interactions; this was to promote the scores for an all-Wrestler version (in other words, I artificially scored any interactions higher - which would not necessarily happen without testing).  The total for each recipe was adjusted by a random setting, + or - up to 5%, to provide a small air of authenticity.  The problem with an element contribution report, however, is that it ignores any possible enhancements or interactions that we might get from specific combinations of elements - I had to adjust this manually afterwards.  By testing more actual recipes, it might be possible to start to uncover some of the interactions between the variables in the test.  It may not identify them, and the system may not attribute them correctly in its results; this would mean that it may not account for them fully when it determines the 'winning' recipe.  However, it's better than the isolated A/B tests that we were carrying out at the start of this post.  

Here are a few more fictitious results that show recipe results comparing the ‘actual’ test results, compared to their predicted results from the few recipes we tested above.


Test Head Torso Legs           Predicted “Score" Actual “Score”
7        Wrestler Wrestler Wrestler          449                          550
8       Bigfoot Bigfoot Astronaut                -                             502                          


Interestingly, the test results indicate that we should work on developing a test version of legs for Bigfoot.  Look at the results for tests 4 and 8.
Test  Head  Torso  Legs      Score
4 Wrestler Wrestler Astronaut 498
8 Bigfoot Bigfoot Astronaut 502


Tests 4 and 8 both have matching heads and torsoes, each with the astronaut head. In test 7, when we had a complete Wrestler, we obtained the maximum positive interaction, and achieved a bonus of 100 points.  Based on tests 4 and 8, where the scores were similar for a two-thirds body, it seems reasonable to assume that a complete Bigfoot will have a similar value as a complete Wrestler.  However, we don’t know the value of Bigfoot legs.  And worse still - or more importatnly, we don’t even know what Bigfoot legs look like, which is the tricky part.  So now we really begin iterative testing.  You didn’t really think that just because we’ve moved from A/B testing to MVT, that we’d completely optimise the page with just one round of MVT, did you?  8-)


And before you ask, yes, this is very over-simplified, and yes the figures are contrived.  As I've said before, MVT is not going to fix a website by itself - it will always require some thinking time to actually look at the results and analyse them, and then proceed through the analysis - recommendation - action - test cycle.


There are various "engines" available for building and then serving the MVT recipes I've shown above (I devised the recipes and then built my table of results long-hand, which was just about manageable for a 3x3x3 test).  One of the most popular engines, that a number of MVT providers use, is the Taguchi method of testing, which is used by some MVT service providers.  


The Taguchi method was designed in the 1940s and 1950s by a Japanese scientist and engineer called Genichi Taguchi.  He devised a radical new way of improving manufacturing quality, which was refined and perfected in a wide range of manufacturing applications, including the Japanese car and telecom industries.  This technique, the Taguchi Method of Process Improvement, can be applied to online testing, but it doesn't work quite as effectively as it does for manufacturing.  The online environment in the 21st century is very different from the manufacturing industry.  In particular, the Taguchi method doesn't properly consider the dependencies or synergies between the different areas – the 'interactions' – and assumes that each variable can be optimised independently from the others.  


A simple definition of the interactions between variables in a test like this is that the performance of one or more parts of the test depends on what else is being shown in the other parts, so that they can’t be optimised independently from each other.  I briefly mentioned this in my previous post, where I looked at the interaction between an image and the caption that went with it – but I'm hoping that this example with the fridge magnets is a little clearer.  


Another way of putting it is by saying, "Yes, A will beat B, unless we use D instead of C.  It depends."  If the success of A over B depends on using D or C, then there's an interaction there. 


Some so-called MVT service providers don't really carry out true multi-variate testing, instead they just carry out a range of simultaneous A/B tests, and don't look at the interactions between the different page components, and this leads to a sub-optimal solution.  Please don't misunderstand me, this will probably be an improvement on an untested version (unless the original had a strong positive interaction), but it's highly likely that it's not the best solution.  


Other, more sophisitcated providers have their own custom-built MVT engines which claim to be able to produce test recipes which will cover the full range of combinations (without having to test them all) and still be able to take interactions into consideration.  I can't comment on how effective they actually are (I've not used them, I've just read their whitepapers and their sales blurbs) but the key players, from what I've read, are


Vertster – followers and proponents of the Taguchi method of testing


Autonomy Optimost – mentioned above – do not use Taguchi, due to its limitations


Site Tuners – aware of the various methods for testing, and cover them all, have a strong awareness of the issues of interactions (and I borrowed their images in my previous post on interactions).


In conclusion, I think it’s reasonable to say that any testing is better than none, and considered, thoughtful testing is better than just testing.  It’s not just about the tools, it’s more about the brains and the process.  By doing any form of testing – and I should say that A/B testing is not the poor relation – you are on the right path to improving your website’s performance.

The Emerging Role of the Analyst

It was not that long ago that Internet Profiles Corporation (I/PRO) launched the first ever commercial log file analyser, LogAnalyzer back in June 1994, and the web analytics industry was born. Since then, technology has been advancing at a phenomenal pace. Finding the software and the staff to keep pace with it, and to keep measuring it, has become increasingly challenging. After all, software designers, web designers and JavaScript programmers rarely think about how usage of their technology can be measured when they develop their products; they’re far more interested in beating the competition and providing the best visitor experience (hopefully doing both at the same time).


This means that the role of the analyst has had to come a long way, and there have been various pitfalls to avoid along the way. For example, I/PRO went through mergers and takeovers along its way, and almost exactly 10 years after it was founded, I/PRO was declared insolvent. Hopefully most analysts will last longer than that (I/PRO was saved by another buyout, and its history has become lost in various name changes and mergers). One of the key pitfalls for current website analysts is to avoid being just a counter and a report.


Back in the mid-1990s, when website analysis was in its infancy, the analyst was probably just a counter and a reporter, and that was all that was required of the role - in itself, converting log files into meaningful data, and then information, was hard enough. The ‘analysis’ came from making sense of log files and translating them into counts and numbers, and was much more connected to computer analysis. To an analyst in the 1990s, qualifications in computer science would be more beneficial than those in marketing or mathematics, and the ability to assimilate large volumes of computer reports would be more use than intuition and curiosity.


However, with time, other players entered the website analytics software market, and the skill set required to be a website analyst started to shift, as the mathematical and log-file heavy lifting was moved from staff to software. Staff – i.e. people – were hired to translate the numbers into words and pictures (as business will always love a good graph) instead of converting the bits and bytes into numbers. More demands were placed on the staff, who became able to do the number-crunching instead of the log-file crunching.


We’ve seen a technological revolution, where the software providers have had to keep pace with business requirements, and business requirements have expanded to exploit the expanding capabilities of new web analytics software. Log file analysis was replaced with 1x1 transparent gifs, then JavaScript tags. We used to put counters at the bottom of our pages, and we would count hits and page views, now we count visits, visitors, and analyse click paths, and following integration with other data sources, we even measure revenue and sales.


Depending on your perspective, the role of the website analyst has now emerged from the shadows of a counter and reporter to a business-critical source of data and recommendations (in terms of business role); or from the computer science engineer to a data-savvy marketer and key decision influencer. We’re able to represent the website data in an understandable way – either by writing reports, e-mailing recommendations or presenting our analysis face-to-face.  We can up with ideas for testing, with supporting hypotheses.  The role of the analyst has emerged from a role of supporting existing projects (“This is what we’re going to do, see if it works”) to being a key consultant at the planning and delivery stages (“What do you think we should do here?”).  The questions have moved from closed questions to open questions, as web professionals have become more open-minded, and analysts have been able to get onto the front foot more often.


It reminds me of a scene in an episode of Star Trek Voyager (“Relativity”). One of the characters (“Seven”) consults the medical database and decides, after reviewing her syptoms, that she has a serious, degenerative illness, and consults the Doctor to explain her reasoning. The Doctor is a hologram, programmed to identify and cure sickness and disease in the crew, based on the data in the medical database. Seven and the Doctor then go through her symptoms, and the Doctor explains what he finds to be the problem, and gives her the necessary treatment, with an immediate improvement. He returns to his sickbay, and, as he leaves, says, “Remember, next time your human physiology fails, you don’t consult the database; just call me.” “You are the database.” “With two legs and a bedside manner!” As analysts, we’re often called on to represent our data, to make sense of it and to turn it into words. That, for me, is the emerging role of the analyst – from the number cruncher who follows along behind the web team, to a leading position in informing future decisions about a website.


This article has been written as part of the first ever web analytics Blogarama, with analysts sharing their views on 'the emerging role of the analyst'. 

Film titles: X of the Y

X of the Y film titles

I watch a lot of films.  Not vast numbers, and I'm not a film expert, or a film critic (despite writing articles full of criticisms of films - that's another story), but I've noticed an interesting or peculiar trend in the names of some films.

Return of the Jedi
Revenge of the Fallen
Revenge of the Sith
Dark of the Moon
Planet of the Apes
The Night of the Hunter (I haven't seen it, but apparently it's a thriller)
Pirates of the Caribbean

Even, historically, "Night of the Living Dead".  I haven't seen it - nor do I want to - but it's included in this rather strange list of films which has the title form "X of the Y".  I started thinking of the films I've seen, and then, in addition I've had a glance through various lists of top 100s or top 500s, to see if I could find many more.  Funnily enough, most of them are science fiction - the majority of "normal" films (as various people would call them) have "normal" titles.  So, my question is:  Why do they have the title format "X of the Y" and not "The Y's X".   Why not have, "The Apes' Planet" and "The Caribbean Pirates" or "The Sith's Revenge"?  I suppose there are various reasons, and I'm considering the following.  It could be:

*  a bad case of apostrophobia (people not knowing where to put the apostrophe), 
*  creation of uncertainty in the title (is there one Y, or more than one?)
*  a literal translation from a foreign language, 

Is it a bad case of apostrophobia?  No, it's not a real word, but it appears to me that an increasing number of people don't know how to use an apostrophe, and so they leave it out completely, or, in the case of film title writers, they rewrite their prose to avoid the need for one.  How do you punctuate the straight version of "Revenge of the Fallen"?  Is it "The Fallen's Revenge" or "The Fallens' Revenge" or do you go for "The Fallen Revenge" and completely lose the 's' and the apostrophe?  In each case, the meaning is slightly different - it's also explicit in each title that there is either one Fallen, or more than one, or in the last case that the Fallen has moved from being a noun (stating what it is) to an adjective (describing the type of revenge).  In this case it's also a little clumsy.  As far as the film goes, the accurate title would be first one (there's only one Fallen), and the second one looks clumsy.

The Star Wars films benefit from having Jedi and Sith which are single and plural terms - one Jedi, many Jedi, one Sith, many Sith -  so there's still a degree of uncertainty in how many Jedi and Sith are involved when you move to the straight form:  "The Jedi's Return", "The Sith's Revenge" and in fact "The Jedis' Return" would be incorrect.  However, "The Night of the Living Dead" is another where the Y is a plural term (I am assuming there's more than one of them, having successfully dodged the film so far), but "The Living Dead's Night" sounds equally menacing to me.

So perhaps it really is a case of developing uncertainty in the title, rather than explicitly stating that there's one or more of the Ys, which is my second suggestion.  The Revenge of the Sith - we really don't know how many there are - is it one, two, six or ten?  The Revenge of the Fallen has this benefit - as mentioned above, is it one character called the Fallen, or is there a group of fallen characters (which would be more likely)?

"The Dark of the Moon" is a good example of this: the straight version would be "The Moon's Dark" which sounds like the obvious statement that the Moon is dark.  The form used in the film title makes it clear that it's the Dark belonging to the Moon; this film title is unusual because dark isn't usually a noun, but an adjective - more mystery and uncertainty, which works well for the film.  

The other option - a literal translation from a foreign language - is unlikely, since most films aren't taken from e.g. French storylines, although it does fit.  For example, "le jardin des enfants" translates literally as The Garden of the Children, but would be written as the Children's Garden (another plural).  Or how about "La maison de l'homme grand" which is The House of the Large Man...  okay, it's a poor example, but it also shows the power of the plural group - compare that to the House of the Large Men.  No?  Perhaps not.

So, it seems we're going to be stuck with X of the Y - especially, as my list shows, in the science fiction genre, where the deliberate ambiguity that's created by the long form outweighs the simplicity of the Y's X form.  An extension of this is to sequels, where you could have Z of the X of the Y, for example, The Rise of the Planet of the Apes, which is worth double points.  

Are there any others?  Are there any more Z of the X of the Y in particular?

Probabilities and Free Toys, Part 2

Last time, I looked at solving a probability question:  for a set of 3, 4, 5 or n toys which are given away free inside a cereal packet, what's the probability of obtaining the full set of toys after buying the same number of packets (e.g. for 5 toys, getting all 5 after buying 5 packs).  

This time, having solved the easier question, let's take a look at the harder question:  with 10 toys (or a number larger than three or four), how many packs do I have to buy to be 50%, 70% or 90% sure of having the full set?


Once again, let's start with two toys (start small!):


After buying two packs, the probability of success is 0.5.  The two successful combinations are AB and BA, and the two unsuccessful are AA and BB (i.e. I get the same toy twice).


After buying three packs, the probability of success rises to 6/8, which is 0.75 or 75%.
There are eight total combinations (from AAA, AAB through to BBA and BBB), but only two are unsuccessful (AAA and BBB are unsuccessful), leaving six successful combinations.


After buying four packs, the probability of success rises to 7/8.  There are 16 total combinations, but the number of unsuccessful combinations remains at two, and the number of successful combinations rises to 14.


Generally, for the two-toy problem, the probability of getting a successful combination after t turns is equal to (2^t - 2) / 2^t  or, in words, the total number of combinations minus unsuccessful, divided by the total number of combinations.


This, however, is where it gets tricky.  With three toys and three packs, there are just six successful combinations (the exact permutations, in alphabetical order, are ABC, ACB, BAC, BCA, CAB and CBA) and 27 total combinations (which confirms what I proved above for the three toys and three packs case).  With three toys and four packs, the number of successes rises to 36, and the total number combinations is 81.  The success probability is 36/81 which is 4/9, coincidentally double the probability after three packs.  After five packs, the number of combinations rises to 243.


Looking at the number of combinations I've seen so far, for the three-toy problem, it's risen from 9 to 81 to 243, rising by a factor of 3 each time, and n=3, the number of toys squared. In fact the denominator, the total number of permutations, is simply n^t (number of toys raised to the power of the number of turns or packs).  This is also known as the formula for 'permutations with repitition'.


Now I need to identify the number of successful permutations - those that contain one of each of the toys.
For n=3 toys and t=3 turns, there are 6 successful permutations (as listed above).


For n=3 toys and t=4 turns, there are 36 successful permutations


Now, the question becomes - how many will there be after five turns?


Let's go back to four turns and look closely at each of the permutations we have.  We have the 36 successful ones, ABCA, ABCB, AABC and so on.  For these 36 successes, it doesn't matter what we pick next, we'll still have a successful combination; there are three options for each of these, so that's 36x3=108 successes for each of them.


There are also the 3 combinations AAAA, BBBB and CCCC which will not produce a success, irrespective of what we pick next, so that's 3x3=9 fails.


This leaves the rest, which logically must contain two different toys.  We've covered the ones which already contain three different toys, and we've looked at the ones which contain only one toy.  Since there were 81 combinations in total after four goes, there must be 81-(36+9) = 36 combinations which contain two different toys.  There is a probability of 1/3 of the next choice being the correct one, which equals 36 x 1/3 = 12.


So, after five turns, we have 108 + 12 = 120 successful permutations.  
Let's review:


After three turns: 6 successful permutations
After four turns: 36 successful permutations
After five turns:  120 successful permutations


Let's take a look at six turns, based on the process we used for five turns.


After five turns, we have 120 successes which will each yield three more successes, so 3x120 = 360
We also still have the combinations which have just one toy in - AAAAA, BBBBB and CCCCC, and these will each produce three more unsuccessful combinations, irrespective of the next choice.  3x3=9 unsuccessful combinations.


There are 3^5 total combinations after five turns, 243 in total, which means that we have 243-(120+9) = 114 other combinations which have two toys.  A third of these will become successful with the sixth turn, which is 114/3 = 38.


So, after six turns, we have 360 + 38 = 398 successful permutations.  I've deduced the formula for working out successful permutations in an iterative manner, but I don't have the computing power to determine the 10th, 15th or 115th term without knowing each one before.  Furthermore, this method won't easily expand to cover five, six, or ten toys. It's all about knowing how many successes you've had previously, and how many certainly won't become successful (because they have n-2 different toys and will require at least two more goes to become successful).


Three toys is an easy case - you either have a successful combination, a combination with only one toy repeated, or a two-toy combination with a 1/3 chance of becoming successful.  With four toys, you may already have one, two or three different toys, or be successful, and I don't quite see how to sort all that out.


So, next time, it's on to spreadsheet modelling.  I'm going to write a macro that simulates buying the cereal packets and examining the toys, and determining whether or not the combination is a success.  If maths fails, use sampling!

A Beginner's Social Media Strategy

I say 'for beginners' as I don't feel particularly qualified to discuss it in much detail (as you'll soon see), and this is more an explanation of my background and experience so far.


A few years ago, I set up my own website; you may have seen me refer to it previously. It was set up entirely as an exercise in website-building and tagging - it's all hard-coded HTML. It's tagged with Google Analytics, and I've been monitoring traffic to it since then, doing a little SEO and making changes (and hopefully improvements) to the content here and there. As analysts, we're usually charged with analysing, understanding and reporting stats, and then generating recommendations from them; we're not usually given a logon to a CMS and given free run of a website. By building my own website, I got to play both sides (and realised how time-consuming content generation and site maintenance can be). Better still, I've even been running A/B/C tests on it, and finding out how easy it is to set up (providing you've got multiple content ready to serve).


Anyway, with time, I moved from focussing on the website to a blog. Blogs are, for this JavaScript beginner, much easier than HTML websites, especially when I can't use server-side includes. So, even with a WYSIWYG HTML editor, blog posts are much easier than HTML pages. Once again I found out how to tag my blog by putting javascript includes in my posts and inserting GA tags in my posts, and I've monitored the traffic. I also discovered how many analysts there are out there who are reading this blog (it's not a huge number, but compared to the single digits I was experiencing before I started writing about web analytics, it's a significant uplift). The blog has an 'about me' page which includes my Facebook and Twitter details, so that people can follow me, and I post updates about my blog on Twitter or Facebook or both, and occasionally on the Yahoo Web Analytics forum.


All of which means that I've built up a cyclical path between my social media accounts (where you can find links to my blog) and my blog (which explains how to follow me on social media).


That's not a strategy, that's just a circle. Which brings me to one key question: What am I actually trying to achieve with all this online presence?


Am I trying to get Twitter followers? Am I doing this for my ego, or for PR, or something like that? Maybe, but probably not.  Am I trying to get Facebook friends? No - I've got enough friends (and I've met 99% of them in person) and I've successfully tracked down my best friends from primary school, high school and university. Am I trying to get more people to read my blog? Now then - that seems more likely. What I'm trying to do is to share what I know about various subjects (maths, chemistry, chess, web analytics) and hopefully build up an online reputation as a reliable source of useful, accurate information - to be regarded as a specialist in my field (and possibly even, one day, an expert).


People aren't going to get that level of knowledge about me from my Twitter feed (which, even with my best intentions, is very clouded up with links to miscellaneous stuff I find interesting). They are most certainly not going to get that from my Facebook updates, which are much more personal and include family updates, photos from days out and the like, and are very much about my views and opinions and general chatter. Hopefully, though, my peers and friends will read my blog, where I write my more considered opinions and views, and share what I hope will be useful insights into areas that specifically interest me - as I said, Chess, maths, chemistry and web analytics. The blog also has a Google Analytics goal set up - visitor views the About Me page - and this means that my social media strategy not only has an aim (to get people to read the blog) but a specific goal (find out more about me and my professional skills and experience).


I could go on and build KPIs about blog traffic levels and so on, and on to Twitter followers (excluding the spam accounts) but these will be secondary to getting people to view my profile page on my blog. I can THEN use analytics to tell me which blog post they read before reading my profile, and also where they came from... and then write more blogs on similar topics and post links on similar sources.


And that, in a nutshell, is my social media strategy. I can't say that it's scalable to a company level, but I think the main points (which are probably covered elsewhere) are:


What am I actually trying to achieve with all this online presence (answer in English words)?
What is my actual aim?
What does this look like as an online event (in page terms)? Make this an online 'goal' or 'event' in analytics package.
What type of visitor carries out a success event? Which social media site did they come from?
How do I get more of them? What sort of content do they look at?


I'm not sure if this is a social media strategy, or just a reiteration of a normal online strategy. Like I said, I'm a beginner on social media strategies (despite having a blog, Facebook and Twitter accounts for years) so I'm open to other suggestions!