Leveraging Data for Segment Identification and Personalised Experiences

In previous articles, I have explored the practical strategies for analysing user behaviour derived from website interactions and A/B testing methodologies. These techniques are particularly valuable when aiming to identify high-performing user segments and craft targeted digital experiences. However, this process often reveals unexpected insights, especially when initial testing does not yield favourable results.

Consider a scenario in which a newly introduced test design, despite its promising conceptual appeal, fails to deliver the desired impact. Quantitative results may be unequivocally negative, with every relevant key performance indicator (KPI) registering a decline. In such circumstances, the data suggest that the variation, such as Recipe B, is not simply underperforming; it has failed across the board. This presents two immediate paths forward: one may either accept the outcome, analyse the failure, learn from it, and iterate on future design efforts—or choose to segment the data to investigate underlying causes.

This latter approach can become a complex and treacherous endeavour, resembling a descent into a highly intricate and potentially misleading analytical rabbit hole.  Sorry, Alice!

For example:

• An initial segmentation comparing new versus returning visitors may reveal that returning users did not react as negatively as first-time users.

• A further segmentation might then show that returning users accessing the site via mobile devices performed slightly better than their desktop counterparts.

• Digging even deeper, one might uncover that returning mobile users interested in higher-priced products exhibited improved engagement or conversion metrics.

While these findings may appear promising, they come with a significant trade-off. After applying successive layers of filtering, the relevant audience is reduced from an initial 50% of total traffic to just 4.3%. This raises a critical question: is it worth allocating substantial resources to tailor a unique experience for such a narrow segment?

For certain brands, particularly high-end luxury retailers such as Rolls Royce or Beaverbrooks, personalised attention may be both viable and beneficial. However, for businesses operating in more commoditised sectors, such as discount pet supplies, the return on investment for micro-targeting may be far less compelling.

This same dilemma applies when designing personalisation campaigns. As I have highlighted previously, two major challenges confront marketers:

1. Acquiring and interpreting high-quality, granular data.

2. Producing and managing sufficient content to serve these differentiated segments.

Assuming these hurdles are overcome to a satisfactory degree, one must still exercise caution in the granularity of targeting. Excessive segmentation risks over-engineering the user experience, while minimal segmentation may fail to resonate at all.

Let us consider a common digital merchandising challenge: What content should be displayed in the homepage hero banner? What recommendations should populate the "We think you'll like this..." carousel? Should your virtual storefront attempt to predict and proactively suggest specific products based on past user behaviour?

For instance, one might propose that a returning visitor be presented with a Lego set—unpromoted, without a discount—on the basis of prior browsing behaviour. This raises the question: is your targeting sophistication high enough to ensure that this suggestion will genuinely resonate?

Many practitioners point to brands such as Netflix and Amazon as exemplars in personalised recommendation systems. Statements like "Because you watched Star Trek: Deep Space Nine" provide a transparent rationale for the curated content that follows. These systems succeed because they offer breadth—providing up to 42 scrollable options—ensuring that even if the primary recommendation does not engage, several others might.

This model can be effectively emulated in retail contexts. For instance, a virtual toy store could present a curated range of 42 Lego models and invite user exploration. Such an approach is far more engaging than presenting a single product, especially when stock limitations may lead to user frustration if the highlighted item is unavailable.

A broader tactic, such as "Would you like to explore our Lego collection?" may prompt greater engagement than asserting, "We believe you want this specific model." The former understands more about user independence and improves interaction metrics (which are often paramount KPIs in digital strategy, right?).

Compare the following messages:

• “Welcome to our toy shop! These are our favourite toys!”

• “We think you’re interested in construction toys.”

The first message represents a generic push strategy, commonly found in homepage banners that prioritise brand objectives over user intent ('we' are more important than 'you' - the long-standing tension between marketing and customer needs). The second, although still broad, reflects an attempt to connect with presumed customer interests. In this context, even minimal targeting is advantageous—and likely more effective than overly narrow personalisation.

Let us assume predictive modelling yields:

• A 71% likelihood a visitor is interested in Lego products.

• A 23% likelihood they are seeking Lego Technic.

• A 7% probability they want the Lego Technic Excavator

Rather than presenting the most specific item, a better approach would be to guide users toward the Lego Technic category and enable autonomous navigation from that point. This ensures relevancy while allowing for user discovery and choice.  Move users forwards one step down the funnel (or the rabbit hole), not two.  Maximise your chances of being correct, and let users navigate from there.

While the hypothetical website may sell any range of products, the Lego example is a globally recognisable, visually compelling use case. Naturally, any product category could be substituted in this framework. If I'd been analysing your browsing history, I might have tailored this piece to your specific interests.

Until next time!

Similar posts I've written about online testing

Getting an online testing program off the ground
Building Momentum in Online testing
How many of your tests win?

"Click For Free Money" on the Customer Journey

A few years ago, a colleague of mine highlighted a crucial insight: engagement on a button isn't always a reliable measure of success.  I know it's hardly an earth-shattering revelation, but it came at a time when KPIs were focusing on engagement, when he and I were looking at testing CTA wording.  Should we go for "Buy Now" or "Customise and Buy"?  Which is better - "View Details" or "Learn More"?  In short, we discovered that the best wording depended on the point in the purchase path, and we had to separate the analysis, because the button that achieved the highest click-through rate (CTR) wasn't always the one that led to the best overall conversion rate. 

In simpler terms, success isn't just about shuffling customers to the next page in our online purchase path. While persuasive wording can certainly encourage users to click and advance, we need to critically ask ourselves: does this genuinely help them progress beyond that initial stage, or are we just creating a false sense of momentum? This brings us to what I call the "Click for Free Money" fallacy. You can absolutely get people to click a button, but the subsequent page must deliver on the expectation set by that button's wording.  Click for Free Money will certainly get a lot of clicks, but there's no benefit to this approach, as people realise that the next page is actually just more information about our products, and maybe a coupon code to validate the 'free money' offer.  

The Promise and the Reality of the Click

Consider the user's mental model. When a button says "Add to Cart," the user fully expects to be taken to their cart fairly quickly. A brief, relevant detour to offer a highly personalized upsell might be acceptable, but anything more, or anything irrelevant, will create friction. Similarly, a "Customize" button should seamlessly lead to customization options, allowing the user to tailor their product without unnecessary distractions. And perhaps most critically, a "Checkout" button must initiate the checkout process. This is not the time for additional upsells, cross-sells, or any other interruptions that could derail a customer who is ready to complete their purchase. Each click builds an expectation, and failing to meet that expectation, even subtly, erodes trust and increases the likelihood of abandonment.  A lack of 'free money' will erode trust remarkably quickly, although it will probably be on the 'free money page, not the previous page.  Still, if you and your team are prepared to do anything to get users to move forwards from your page, you could deploy this tactic.  It depends on what your KPI actually is (and who's looking at the long-term journey).

The Detrimental Cost of Rushing Customers

Pushing customers forward too quickly isn't just inefficient; it can be detrimental to long-term customer relationships and conversion rates. If you don't continually reinforce the value proposition and reassure the user along their journey, they're highly likely to drop out at a later, more critical stage. For instance, if you enticed a user with a 10% discount while they were casually Browse your site, that discount needs to be prominently displayed and automatically applied on the cart page. Hiding it, or making them jump through hoops to redeem it, is a surefire way to lose a sale and disappoint a customer.

It's vital to shift our focus: moving customers towards a purchase isn't as important as guiding them through their journey to choose the best product for them. This is a fundamental difference between a transactional mindset and a customer-centric approach.

Think of it like a pushy sales assistant in a physical store. Imagine walking in, and immediately, an assistant shoves an item into your hands, puts an arm around your shoulder, and starts steering you towards the checkout. When you haltingly ask, "But is this my size?" they might dismissively respond, "Probably, yes." If you inquire, "Will it perform better than my previous widget?" they might curtly say, "Who cares? Cash or card?" Or if you try one more time, "But is it quieter and more efficient than the old model?" they might simply reply, "Yes. Is the shipping address the same as the billing address?"

While this sales assistant might technically "drag" the customer to the start of the checkout process, what was the true cost? A frustrated, confused, and potentially alienated customer who probably won't complete the purchase and certainly won't return (worse still, they'll tell their friends).

Preparing for the Next Step: The True Purpose of the Funnel

Often, all we accomplish by being overly aggressive or deceptive in the early stages of the funnel is simply shifting the exit point to a later stage in the user's journey. The customer still leaves, but now they're more annoyed because they've invested more time and effort. Instead, it's far more important and beneficial to leverage each step of the customer journey to authentically prepare users for what they'll encounter next. Each stage should provide value to the user, clarify information, and build confidence and trust.  In a situation where different parts of the journey belong to different developers, teams and managers, 

When a user is genuinely informed, reassured, and ready to take the next step—because they understand the value and feel confident in their choice—then and only then, should we make it effortless for them to move forward. This approach fosters trust, reduces abandonment rates, and ultimately leads to more satisfied customers and higher conversion rates in the long run.  We may not offer 'free money', but by subtly pushing users forwards in a path that they aren't fully prepared to take - by providing quicker paths or by removing content that is actually useful to users - we will merely persuade them to move forwards from our page into a situation where they're increasingly likely to leave from the next page.

But still, our next-step page flow metrics show a lovely funnel that shows we moved 10% more users to the next step.  Our analysis, then needs to show how many users move forwards to the next-next step, and then actually completed the full purchase journey.

Consider a simplified conversion funnel, from a landing page through a website to completing the checkout process.

Similar to the 'did you just code a distraction?' question - we see that users move forwards to the next step at a much higher rate.  950/998 = 95%, compared to 75% for control, and 65% for Recipe B.  See, you coded a winner, it just happened to be different than the one you expected!

However, when we look further down the funnel, we find that there's a massive 50% drop-off for Recipe C.  We failed to deliver free money, and people left.  Unsurprisingly, fewer visitors then reach the lower funnel states, and Recipe B is actually the winner (with Recipe C coming behind control).

And if you think that's far-fetched, then perhaps replace 'Free money' with something more believable... like, perhaps, "Add to Basket."  Do customers really get to add the item to basket when they click that button?  Or do they have to select a size, a colour, an upgrade, a guarantee, a warranty or something else first?

Or do you promise things with your CTAs that you aren't really delivering?  "Find out more" needs to show more product information about the product it's connected to.  Getting clicks is easy, but you need to keep an eye on the next step, not just the one you're testing.

Similar posts I've written about online testing

Getting an online testing program off the ground
Building Momentum in Online testing
How many of your tests win?

Why Too Many Cooks Spoil the A/B Testing Roadmap

How to build a lean, decisive testing program without sacrificing collaboration

When Consensus Kills Velocity

A/B testing is - in theory - designed to answer questions quickly: Which experience performs better? Which idea deserves further research and investment from a budget-conscious company and a stretched development team?

The power of testing lies in short cycles, sharp hypotheses, and decisive execution. Yet in many organizations, the A/B testing roadmap becomes a negotiation table. Every team wants to weigh in, every stakeholder demands review, and the result is predictable: delays, diluted experiments, and very little learning.  The time taken to run a test (i.e. from launch to confidence) starts to pale in comparison to the amount of time it takes to decide what should be in the test.

If your testing program needs a large, multi-functional committee to approve every hypothesis, pick every variant, and sign off every launch, you don’t have a testing program— you have a meeting calendar. It's a painful paradox: a method designed to reduce risk and speed up learning turns into a process that creates risk by slowing decisions and reduces learning by launching compromised tests that fail to answer meaningful questions.

The Ubuntu Concept—and Why Sequence Matters

Ubuntu, often summarized as “I am because we are,” is a powerful philosophy of interdependence, empathy, and shared success. In short, "If you want to go far, travel together; if you want to go fast, travel alone."  There can often be an emphasis on teamwork and consensus, but the truth is, with testing, the fewer people there are, the better.  Testing is such a powerful tool, and it takes so long to develop a test that everybody wants to get involved in deciding what goes in the test, and it has to be pixel perfect.  Tragically, testing's power becomes its downfall - tests take so long because everybody wants to have their input, but they take so long because everybody's input has to be taken into consideration.  There needs to be a time when a leader says, "Enough!"  When somebody takes responsibility and authority, and actually includes fewer people - not more - in the testing process.

The better application is Ubuntu-after-evidence. Before the test, a small, empowered team should define hypotheses, success metrics, and launch criteria with minimal friction. After the test, the broader community can engage—reviewing results, sharing learnings, and co-creating improvements that benefit everyone.

In other words, we preserve Ubuntu’s spirit of shared learning and collective improvement without letting endless pre-test consensus slow down the very mechanism that generates the learning.

Why Fewer People Make Testing Better

A lean testing team is not anti-collaboration. It is pro-clarity. In my experience, the most effective testing programs are lean and decisive because velocity matters above all. A/B testing’s value decays with time. Long approval and planning cycles reduce relevance and increase opportunity cost. Short cycles mean you learn faster, pivot earlier, and deploy compounding gains sooner.

Clear ownership prevents drift. When the roadmap belongs to everyone, it effectively belongs to no one. A focused owner or small squad can prioritize tests based on impact and feasibility, not politics. Committees tend to compromise, merging ideas into multi-variable tests that answer nothing clearly. Lean teams keep hypotheses tight, variants minimal, and interpretations crisp.

Everybody has the opinion that evidence should replace opinion, except where it contradicts what they want to do. A streamlined process moves debates downstream. Instead of arguing which idea is better, we test quickly and gather data, and then prove which idea was better.

Who Should Be in the Core Team?

The lean model works because it assigns clear roles and decision rights. At the center is the Product Manager, who owns the testing backlog and makes go/no-go decisions. They ensure that tests align with business objectives and that prioritization reflects impact rather than politics.

Alongside the PM, or combined in the same role, is the Experimentation or Data Lead, who designs hypotheses, defines success metrics, and ensures statistical rigor. This person calculates sample sizes, sets stopping rules, and interprets results so that every test produces actionable insights.  This role can is a specialized testing role, where a detailed and nuanced understanding of testing, significance, confidence, the testing pipeline, dev capabilities are vital.  They will work closely with the Product Manager when the process of testing has been documented and established.

The Engineering representative plays a critical role in implementing variants, managing feature flags, and ensuring telemetry accuracy. They safeguard performance and security standards, making sure experiments do not compromise the user experience.

The Business Lead works with the team to bring forward tactical business needs, which can feed directly into testing requirements.  Do you need to sell more high-price menswear this quarter?  Are you expanding into audiobooks as well as audio streaming services?  Where, from a business perspective, should we focus our short-term efforts?

A UX or Design specialist ensures that variants are user-friendly, accessible, and on-brand within pre-agreed guidelines. Their involvement prevents usability and accessibility issues and maintains consistency without introducing unnecessary complexity.  However, it is absolutely critical that the UX and design specialists support the velocity of the testing roadmap by supplying the assets that are needed for testing without going through their own endless and unnecessary cycles of 'discovery and framing' or 'research', which will single-handedly bring any test roadmap grinding to a complete stop.   

Legal and privacy advisors contribute upfront by defining evergreen guardrails for compliance and data handling. They are consulted during framework design, not as ad hoc gatekeepers for every test.

Finally, stakeholders remain informed and can submit ideas through a structured intake process, but they do not have veto power over launches. Their role is to provide context and consume learnings, not to slow down execution.  Stakeholders have visibility, not veto.

A Common Failure Pattern—and How to Fix It

Some organizations follow a familiar failure pattern. The quarterly roadmap is assembled in a multi-department forum. Hypotheses get neutered to accommodate every stakeholder’s request. Legal, Brand, Product, and Design each add gates. Engineering estimates are based on maximum-complexity variants. There are changes during development, there are changes during pre-launch quality checks.  Scenarios were not considered, and designers Tests are launched late with weak hypotheses and unclear success metrics.  Everybody wanted a slice of the testing cake, and now it's a half-baked mess. Results are inconclusive, and trust in testing erodes.

The fix is straightforward but requires discipline. Empower the core team described above with decision rights for hypothesis selection, prioritization, and launch. Set standard guardrails for brand, legal, and privacy constraints that are pre-agreed and do not require ad hoc approvals. Publish metrics definitions that are standardized and documented. Run short test sprints with limited dependencies. Finally, socialize results broadly through summaries and open Q&A sessions—Ubuntu applied post-evidence.

Conclusion: Ubuntu After Evidence

A/B testing is a learning engine. To keep it running, you must minimize friction at the point of decision and maximize inclusion at the point of reflection. Ubuntu teaches us that we rise together; testing teaches us that we rise faster when we let data lead. So don’t abandon Ubuntu—sequence it. Empower a small team to move quickly within clear guardrails. Then invite the broader community to interpret results, celebrate wins, and turn lessons into shared progress. Fewer people at the decision-making table means more learning for everyone.

Similar posts I've written about online testing

Getting an online testing program off the ground
Building Momentum in Online testing
How many of your tests win?

Do You Know How Well Your Test Will Perform?

 There are various ways of running tests - or more specifically, there are various ways of generating test hypotheses.  One that I've come across over the years, and increasingly so more recently, is the 'guess how well your test is going to perform' approach.  It's not called that, but it seems to me to be the most succinct description.  Apparently, we should already have a target improvement in mind before we even start the test.

"If we change the pictures on our site from cats to dogs, then we'll see a 3.5% increase in conversion."
"If we promote construction toys ahead of action figures, then we'll see a 4% lift in revenue."

If you know that's going to happen, why don't you do it anyway?

The main underlying challenge I have is that it's almost impossible to quantify the improvement you're going to get.  How do you know?

Well, let's attempt the calculation (with hypothetical numbers all the way through).

Let's say our latest campaign landing page has a bounce rate (user lands on page, then exits without visiting any other pages) of 75%.  10% engage with site search, 10% click on the menus at the top of the page, and 5% click on the content on the page (there are a few banners and a few links).

We've identified that most users aren't scrolling past the first set of banners and links, and we therefore hypothesise that if we make the banners smaller, and reduce the amount of padding around the links, that we can increase engagement with the content in the lower half of the page, and therefore improve the bounce rate.  We believe we can get 50% more links above the fold, and therefore increase the in-page engagement rate from 5% to 7.5%.  We will assume (and this is the fun bit) that this additional traffic converts at the same rate as the 5% we have so far, and therefore, we'll get a revenue lift of 50%.  This sounds like a lot, but given that the engagement rate is going up from a small number to a slightly larger number, it's unlikely to be a huge revenue lift in dollar terms (unless you're pouring in huge volumes of traffic - and watching it bounce at a rate of 75%).

Perhaps that was an over-simplification.  But if we knew that our test will give us a 5% lift (and we've still decided to test it), what happens when we launch the test?  Presumably, we'll stop it when it reaches the 5% lift, irrespective of the confidence level.  But what happens if it doesn't get to 5%?  What if it stubbornly sits at 4%?  Or maybe just 3%?  Did the test win, or did it lose?  In classical scientific terms, it lost, since we disproved our overly-specific hypothesis.  But from a business perspective, it still won, just not by as much as we had originally expected.  Would you go into a meeting with the marketing manager and say, "Sorry, Jim, our test only achieved a 3% revenue lift, so we've decided it was a failure."?

For me, it comes down to two arguments: 

If you can forecast your test result with a high degree of certainty, based on considerable evidence for your hypothesis, it's probably not worth testing and you should implement already.  Testing is best used for edge-cases with some degree of uncertainty. 

If, on the other hand, you have identified a customer problem with your site, and you can see that fixing it will give you a revenue lift - but you don't know how to fix it - then that's very good grounds for testing.  The hypothesis is not, "If we fix this problem, we'll get a 6% revenue lift," but, "If we fix this problem in this way then we'll get a revenue lift".  And that's where you need to encourage the website analysts and the customer feedback department (or the complaints department, or whoever advocates for customers within your company) to come together and find out where the problems are, and what they are, and how to address them.

That will undoubtedly bring good test ideas, and that's what you're looking for, even if you don't know how much revenue lift it will provide.

Other Web Analytics and Testing Articles I've Written

How not to segment test data (when your stakeholders want you to adapt your data)
Web Analytics - Gathering Requirements from Stakeholders
Analysis is Easy, Interpretation Less So (and why it's more valuable)
Telling a Story with Web Analytics Data
Reporting, Analysing, Testing and Forecasting
Pages with Zero Traffic

Team-Building Exercises and Games for Remote Workers

 The pandemic and lockdown period of 2020-2021 generated a quantum leap in the way in which we work.  I mean - I was working remotely for nine years before the pandemic, but for the vast majority of my colleagues, there was a significant change in working patterns.  My team went completely remote - all working from home - and in September 2021 I became the team's manager.  Since then, we've had some team members leave; I've recruited some new team members, and had the challenging task of building a team.

My team comprises eight project managers plus me, spread across four countries and multiple timezones, so you can imagine that meeting in person isn't going to happen any time soon.  However, that did not stop us from building a strong positive team ethos, and getting to know each other better.  

How?  By playing games.  I mean 'completing exercises' ;-). Here's a list, and a few additional points at the end.

1.  Whose Desk Is This?

Each team member takes a photograph of their desk, and uploads the picture to a shared PowerPoint Deck.  On a team call, we all review the photographs one by one, and try to determine whose desk this is.  It's fun, it's different, and it works as the team tries to deduce who is most likely to have a tidy desk, and untidy desk, toys on their desk... pictures on their wall... the widescreen monitor and so on.  The hilarity follows when the team realise they were completely wrong!

2.  Whose Fridge Is This?
Same principle, but more personal than the fridge.  We discovered that one team member keeps a thermos flask in the fridge (it's her husband's, and she goes mad about it), and that fridges are a lot more expensive in Brazil than in Europe.  We also quizzed each other about our eating habits, and what that stuff is in the glass bowl on the top shelf.  Again, entertaining and a better ice-breaker than 'tell us about yourself'.

3.  Whose Car Is This? and its near neighbour, Whose Front Door Is This?
You get the idea.  For Whose Car Is This, we went further by actually finding stock photos of our cars instead of just taking photos (although this was included as usual).  We were getting so good at looking around the photos (for example, clues for items on top of the fridge, the general scene of the kitchen) that we stepped it up by having the web-source photos as well.  Front doors are front doors, there's no disguising them.

For an added bonus, when you get close to the holiday season, you can have Whose Christmas Tree Is This?  We did, and everybody took part - Christmas tree, seasonal decorations, whichever.  

4.  Online Pictionary (draw the object).  

For this, I prepared a list of seasonal objects - snowflake, Santa Claus, elf, snowman, wise man, shepherd, and so on.  The team split themselves into two groups, and each person shared their screen and used Paint to draw the word.  I am sure there are better online versions of this game, but there was something charming and hilarious about watching everyone try and draw using MS Paint that worked well.  It worked for us in the pre-Christmas period, but could work at any time of year with a given theme.

5.  Two Truths and a Lie

This has become our team favourite, and it requires no extra work from the manager.  You just tell everyone to prepare two truths and a lie, and to either update a shared deck or mail their statements to you.  They don't even have to tell you which is the lie, and then you can play along too.  And, if you take notes and listen to what everyone says, you'll have some great source material for the next activity...

6.  Online Quizzes

We use Kahoot, which is simple, straightforward and requires no costs if you have 10 or fewer team members.  As the manager, you become the question master and ask a series of multiple choice or true/false questions.  You'll need to write the questions in advance, and these can cover any (or multiple) topics.  Some questions are work-related - "What is your analysis of this data?" "What does this data show?" "What is the most important thing to remember about XYZ situation?" and some aren't - "Which member of the team said they most wanted to spend an afternoon with Agatha Christie?" "Which member of the team said that they wanted to do a parachute jump?"

The great thing is that most of these games can be used with teams of any size and discipline (engineers, sales, purchasing and HR can all play these games). What else works well in generating and building team spirit?

A.  Regular team meetings.
I can't emphasise this enough.  Even if you don't have a team-building exercise planned for the week, and everybody's rushed off their feet, and there isn't much to add to all the emails, keep the call anyway.  Your team will appreciate face-time with you, and a chance to catch up with you.  You might not have much to say, but they may have questions for you, so don't cancel!

B.  Regular one-to-ones alongside the team meetings
I meet with each member of my team once a fortnight (or once every two weeks, or bi-weekly).  This gives each member chance to ask you questions that are specific and relevant to them, on any topic that they want to discuss, and which may not be suitable in a team forum.  Cancel these at your own risk!

Email Etiquette

I'm going to go completely off-topic in this post, and talk about something that I've started noticing more and more over recent months:  poor email etiquette.  Not poor spelling, or grammar, or style, but just a low standard of communication from people and businesses who send me emails.  Things like missing images, poor titles, wonky meta tags, and pre-header text (the part of an email that you see in your browser after the subject title).  This is all stuff that can be accepted, ignored or overlooked - it's fine.  But sometimes the content of the email - the writing style or lack of it - begins to speak more loudly than the text in it.
Way back in the annals of online history, internet etiquette ("netiquette") was a buzz-word that was bandied around chat rooms, HTML web pages, and the occasional online guide.

According to the BBC, netiquette means, "Respecting other users' views and displaying common courtesy when posting your views to online discussion groups." while Wikipedia defines it as, "is a set of social conventions that facilitate interaction over networks, ranging from Usenet and mailing lists to blogs and forums."  Which is fair enough.  In short, netiquette means "Play nicely!"

Email etiquette is something else - similar, but different.  Email is personal, while online posting is impersonal and has a much wider audience.  Email is, to all intents and purposes, the modern version of writing a letter, and we were all taught how to write a letter, right?  No?  Except that the speed of email means that much of the thought and care that goes into writing a letter (or even word-processing one) has also started to disappear.  Here, then are my suggestions for good email etiquette.

 - Check your typing.  You might be banging out a 30-second email, but it's still worth taking an extra five seconds to check that everything is spelt correctly.  "It is not time to launch the product" and "It is now time to launch the product" will both beat a spell-checker, but only one of them is what you meant to say.  


- Use the active tense instead of the passive.  Saying "I understand," or "I agree" just reads better and conveys more information than "Understood." or "Agreed."  You're not a robot, and you don't have to lose your personality to communicate effectively via email.

- Write in complete sentences.  Just because you're typing as fast as you think doesn't mean that your recipients will read the incomplete sentences you've written and correctly extrapolate them back to your original thoughts.  The speed of email delivery does not require speedier responses.  Take your time.  If you start dropping I, you, me, then, that, if  and other important nouns and pronouns from your sentences, and replacing them with full stops, then you're going to confuse a lot of people.  This ties in with the previous point - just because the passive tense is shorter than the active doesn't mean that it will be easier to understand.  You will also irritate those who are having to increase their effort in order to understand you.

"Take your time..."

- Don't use red text, unless you know what you're doing.  Red text says "This is an error", which is fine if you're highlighting an error, but will otherwise frustrate and irritate your readers.  Full capitals is still regarded as shouting (although have you ever noticed that comic book characters shout in almost all their speech bubbles?), which is okay if you want to shout, but not recommended if you want to improve the readability of your message.

- Shorter sentences are better than long ones.  Obviously, your sentences still need to be complete, but this suggestion applies especially if your readers don't read English as their first language.  Break up your longer sentences into shorter ones.  Keep the language concise.  Split your sentences instead of carrying on with an "and...". You're not writing a novel, you're writing a message, so you can probably lose subordinate clauses, unnecessary adverbs and parenthetical statements.  Keep it concise, keep it precise.  This also applies to reports, analyses and recommendations.  Stick to the point, and state it clearly.

"Keep it concise, keep it precise."

- Cool fingers on a calm keyboard.  If you have to reply to an email which has annoyed, irritated or frustrated you, then go away and think about your reply for a few minutes.  Keep calm instead of flying off the handle and hammering your keyboard.  Pick out the key points that need to be addressed, and handle them in a cool, calm and factual manner.  "Yes, my idea is better than yours, and no, I don't agree with your statements, because..." is going to work better in the long term than lots of red text and block capitals.  

 - Remember that sarcasm and irony will be almost completely lost by the time your message reaches its recipient(s).  If you're aiming to be sarcastic or ironic, then you'd better be very good at it, or dose it with plenty of smileys or emoticons to help get the message across.  Make use of extra punctuation, go for italics and capital letters, and try not to be too subtle.  If in doubt, or if you're communicating with somebody who doesn't know you very well, then avoid sarcasm completely.  Sometimes, this can even apply over the phone, too.  Subtlety can be totally lost over a phone conversation, so work out what you want to say, and say it clearly.  Obviously!

- Please and thank you go a long, long way.  If you want to avoid sounding heavy handed and rude, then use basic manners.  If you're making a request, then say please.  If you're acknowledging somebody's work, then say thank you.  You'll be amazed at how this improves working relationships with everybody around you - a little appreciation goes a long way.  I know this is hardly earth-shattering, nor specific to email, but it's worth repeating.

- When you've finished, stop.  Don't start wandering around the discussion, bringing up new subjects or changing topic.  Start another email instead.

FOR EXAMPLE

A potential worst case?  You could start (and potentially end) an email with "Disagree."

Don't Make Me Think = Don't Make Me Read?

We all know (because we've all been told) that online selling must follow the eternal (pre 1990) mantra that 'less is more'.  Less clutter, less text and less mess means less confusion equals more clarity equals more sales.  Yet in the offline world, shop shelves are stacked with 23 different brands of toothpaste, 32 different types of shampoo, and aisles and aisles of goods.  And if you think the bricks-and-mortar model is dead, and you'd prefer a modern comparison:  why are Amazon's warehouse shelves so full of so many different types of identical products?  And why do they have so many web pages?

Ignoring that argument, we know less is more because if there's "more" then people will have to think about the product they're buying. And we know thinking is bad because there's a book about online usability called "Don't make me think," and in this era of online publishing and blogging, you must surely be a respected authority on a subject if you can get a book published.  We may not have read your book, but we've heard of it, and we know the title.  [Steve Krug is a respected authority, and was even before he wrote his book].

But could you imagine if this less-is-more attitude was taken a step further in the offline world?

Imagine going to a bricks-and-mortar store, reaching the checkout with your selection, and then making eye contact with the checkout assistant. He (or she) looks away, grabs your shopping off you; scans the barcodes and runs up the total, then looks at you. You look back. He points to the total displayed on the cash register. You reach for your credit card. He waves his hand towards the card machine.


You put your card in the machine. He does whatever it is they do on the cash register to make the magical words “Enter your PIN” appear. There’s brief delay, the words “Transaction approved” appear in black on green on the dot-matrix screen, the assistant hands you your receipt and without any further delay turns to face the next customer.

Delightful, isn’t it? After all, you bought your items and paid for it successfully, didn’t you?  And did you have to think?  Perhaps you would have preferred to use one of those self-service checkouts, complete with 'unexpected item in the bagging area'.

This is the epitome of decluttered pages, where removing anything and everything that isn’t part of the main purchase experience is the absolute aim and surely conversion improvements will follow.  As online analysts, we've trained our managers and superiors to think that less is more, that clutter is bad and that we should just get people to press the button and buy the stuff. However, I believe that kind of thinking is out-of-date and needs to be revisited.

Our visitors are intelligent people.  They are not all high-speed 'I'd buy this even quicker if you got out of my way' purchasers.  Some of them actually want to read about your product - its unique selling point, its benefits, why it's not the cheapest product you offer.  They want to view the specifications (Is it waterproof?  Is it dishwasher safe?) and they won't buy the product unless you reassure them that it's the one they want.  We can reduce our product specifications to icons (yes, it's ultra secure, and it's got central locking and built-in satnav) but if your icon isn't either well-known or intuitive, then you made things worse by removing the words "with a built-in satnav" and replacing them with a picture of ... well... a wifi point?  Have we have started to think (subconsciously) that we should just show glossy pictures of our products and that this would be enough?
 

Is text bad?  And is more text worse?  I don't think so - after all, you're reading this, and despite my best efforts, it's pretty text-heavy with only a couple of images thrown in to break up all these words.  I'm making some pretty heavy demands on you (to read this much text, paragraphed but largely unformatted), but you've stuck with me this far.  I know online selling is different from online reading, but if you've managed to read this much, then it's fair to assume you can read some sales copy on a web page for a product that you are genuinely interested in.

So, is less more?  I am calling for a more balanced approach towards website content. We need to understand that it isn't simply about "less" and "more".  It's about the right content in the right place, at the right time to support a visitor's intentions (which may be 'to buy the product' or may be 'to find the best one for me'.)  And having said this much, I'll stop.

Online Optimisation is a Game of Chess

It occurred to me recently that there are some similarities between playing a game of Chess and optimising a website through testing (apart from the considerable differences between them).

 Many years ago, when I started this blog, I envisaged it as a place where I would type up my chess games, with commentary, lessons and points for improvement.  A quick glance through my archive of blog posts will quickly show that the blog really hasn't matched its original plan.  It evolved and changed, in particular in 2011, when I shared my first few posts about web analytics and I realised that I could attract significantly more readers by blogging and sharing about web analytics than I have ever managed with Chess.  I figure that Chess is a much more mature subject, with many more people who are considerably more experienced than me; whereas web analytics (and especially my main area of interest, testing) is a much younger field, and there's scope for sharing ideas and experiences which are still novel and interesting.  However, there are some similarities between the two - here are a few thoughts.


Strategies and Tactics

Black prepares a tactic.

Plans in Chess can be broadly separated into strategies (long-term aims) and tactics (short term two-or-three move plans).  The long-term aim is to checkmate your opponent's king, and throughout the game that will become the more prominent goal.  In the short term, as you progress through the opening moves, you'll identify potential opportunities to capture your opponent's pieces, to put your pieces on good squares and to restrict your opponent's chances of beating you.  Some of these are short term, some are long term.  For example, it may be possible to win one of your opponent's bishops with a cunning trap (if your opponent is not vigilant).  This will make it easier to achieve your long-term goal of winning the game by checkmating your opponent.

Testing offers the same range of opportunities:  are you looking for quick wins (who isn't?) or are you planning to redesign an entire page, or even an entire site?  Will you be implementing wins as soon as you have confirmed results, or are you going to iterate and try to do even better?  Are you compiling wins to launch in one big bang, or are you testing, implementing and then repeating?  How far ahead are you planning?  Neither approach is necessarily better, as long as you're planning, and everybody is agreed on the plan!


Aims and Goals

White threatens checkmate.

As I've just mentioned, in Chess there's a clear goal:  checkmate your opponent's king, by threatening to capture his king and leaving him with no way to escape.  The aim for your online testing program may not be so clear cut, and if it isn't, then it might be time to find one single aim for it.  You might call it the mission statement for your optimisation program, but whatever you call it, it's important that everybody who's involved in your program understands the purpose of the testing team.  The same applies to each test, as I've mentioned before - each test should have a clear aim, that everybody understands and that can be measured in some clearly-defined way.

Values and KPIs

Each piece in Chess has a certain nominal value; the values aren't precise, but they provide a meaningful comparison of the strength and ability of each piece.  For example, the rook is worth five pawns, the knight and bishop are worth three pawns each, and the queen is worth nine.  This enables players to quickly evaluate a position in a game and say which player is winning, or if the position is roughly equal.  It's slightly more complicated than that, as you have to take into consideration the position of the pieces and so on, but a quick comparison of the total material value that each player has will give a good idea of who's winning.

Rooks are worth five pawns; bishops are worth three pawns and queens are worth nine.

This also means that it's possible to determine if a plan or a strategy is likely to win and is worth pursuing.  It may be possible to trap your opponent's rook (worth five pawns), but if doing so will mean losing two knights (each worth three pawns) and a pawn, then the trap is not really beneficial to you.  If, on the other hand, you could trap your opponent's king (winning the game) at the cost of two rooks and a knight, then that's definitely worth doing.
 
The key performance indicator for a game of Chess is your opponent's king, and if you can measure how close you are to capturing (or checkmating) your opponent's king, then you can see how close you are to winning the game.  You also need to keep your own king safe, but that's where the analogy breaks down :-)

White wins despite less material.

In online testing, your plan, strategy and tests all need to have KPIs.  Once you've established your long-term aim, you can set KPIs against each test which are connected to achieving the long-term aim.  If you want to improve the return on investment (ROI) of your online marketing, you could look at the landing page... improve the bounce rate and the exit rate, and encourage more people to move further into your site and view your products.  Alternatively, you could look at improving conversion of visitors from cart (basket) to checkout.  Or perhaps the flow of visitors through your checkout process.  Providing you can tie each of your tests and your tactics to the overall strategy: "Improve ROI for online marketing" then you can measure whether or not it's succeeding.

Classifying your KPIs in order of importance is also important - as we saw in Chess, if you can win your opponent's pieces but lose several of yours in the process, then it's probably not a good idea.  In testing, what would you do if your test recipe had a worse bounce rate but higher overall conversion (a situation that's not impossible)?  Which is the more important metric - conversion, or bounce rate?  Would your answer be the same if it was improved conversion but lower revenue (people not spending as much per order)?  Are you going to capture your opponent's knight but lose your queen?

Win, lose or draw?

Queen takes King: checkmate!

In Chess, there are clear rules that determine the outcome of a game.  Either one player wins (so the other loses), or it's a draw, and there are various ways of drawing:  including by agreement (both players decide neither can win); by stalemate (one player cannot make any legal moves) or a drawn position where it's clear that neither player has enough material to checkmate the other.


I hate losing at Chess.  However, the truth be told, I'm not much above average as a Chess player, and I'm the weakest player at my club (this has not deterred me, and I still play for fun).  This means that I get plenty of chances to analyse my losses, and see where I could improve.  Do I make the same mistakes in future games?  Not usually, no.  

However, I'm not satisfied to stay as the weakest player in my club - I just see this as an opportunity to do some giant-slaying in my future matches.  I read books, I visit Chess websites, I practice against other people and against computers, but I especially review my own games.  Occasionally, I win.  And do I analyse my winning games?  Absolutely - I may have already seen during the game that my opponent missed a chance to beat me, but did I also miss a chance to win more easily?

In online optimisation, the rules for calling a test a win, lose or draw are still up for debate, and they vary between companies.  And so they should.  For example, how long should you run your test? Each company will have its own testing program with its own tactics and strategies, and its own requirements.  Do you want to have 99.9% confidence that a test will win, or do you want some directional test data to support something you already believed based on other data sources?  How quickly do you want to run the next test, or implement the winner?  Providing that the rules for calling the win, lose or draw are agreed in advance, I might even suggest that they could vary between tests. This is, of course, totally different from Chess, where the centuries-old rules of the game clearly state the requirements for a win or a draw.  Otherwise though, I think it's fair to say that KPIs, metrics and strategy have their approximate equivalents in pieces, pawns and plans - and that thinking and planning are definitely the way forward!

Chess cartoons taken from the 1971 printing of  Chess for Children, originally published 1960.

What is a "growth hacker"?

Okay, I admit it: I'm confused.  I kept up to date with "experts", "gurus", "rock stars" and "ninjas", but I've reached the limit of my understanding. Why are we (as the online analytics community) now using the term 'hacker'?  When modems were dial-up, and going online meant connecting your computer to your telephone headset, hackers were bad people who illegally broke into (or 'hacked') networks

Nowadays, though, hackers are everywhere, and one of the main culprits (especially online) are the "growth hackers".  I'm just going to borrow from Wikipedia to set some context for what these new hackers actually are:

• Hacker (term), is a term used in computing that can describe several types of persons
  • Hacker (computer security) someone who seeks and exploits weaknesses in a computer system or computer network
  • Hacker (hobbyist), who makes innovative customizations or combinations of retail electronic and computer equipment
  • Hacker (programmer subculture), who combines excellence, playfulness, cleverness and exploration in performed activities

Hacker image credit: Pinsoft Studios
So we have, "excellence, playfulness, cleverness and exploration" in performed activities?  Really?  That's what a hacker is nowadays?  Can't we just be good at what we do?  We have to be playful and creative while we're at it?  Perhaps I'm an optimisation hacker and I never realised.

My research into growth hacking indicates that the term was first coined in 2010 by Sean Ellis.  Why he chose 'hacking', I'm not sure (especially given its previous connotations), but here's how he described growth hacking:

"A growth hacker is a person whose true north is growth.  Everything they do is scrutinized by its potential impact on scalable growth. ... I’ve met great growth hackers with engineering backgrounds and others with sales backgrounds; the common characteristic seems to be an ability to take responsibility for growth and an entrepreneurial drive.  The right growth hacker will have a burning desire to connect your target market with your must have solution ...  The problem is that not all people are cut out to be growth hackers."

So:  a growth hacker is a marketer whose key performance indicator is growth.  So why 'hacker'?  Perhaps it's about cracking the code for growth and finding a short cut to success?  Perhaps it's about carving a way through a jungle filled with bad ideas for growth, with an instinctive true north and a sharp blade to cut through all the erroneous ideas?

Image credit: Recode.net

It's an imaginitive either way.  And now, five years after Mr Ellis's post, it seems we have growth hackers everywhere (ironic, considering the URL for the original blog post is /where-are-all-the-growth-hackers/  - they now have multiple websites and Twitter accounts ;-)

So - my curiousity has been satisfied: a [good] growth hacker is a marketer who will help rapidly accelerate growth for a small or start-up company by rapidly analysing what's working for its audience and focusing on those strategies with agility and velocity.  Why are they so popular now?  Because - I'm guessing - after the global financial issues of 2008-2009, there is now much more interest and emphasis in start-ups and the entrepreneurial spirit - and every start-up needs a growth hacker to crack the code to accelerated growth rates.

When to Quit Iterative Testing: Snakes and Ladders

I have blogged a few times about iterative testing, the process of using one test result to design a better test and then repeating the cycle of reviewing test data and improving the next test.  But there are instances when it's time to abandon iterative testing, and play analytical snakes and ladders instead.  Surely not?  Well, there are some situations where iterative testing is not the best tool (or not a suitable tool) to use in online optimisation, and it's time to look at other options. 

Three situations where iterative testing is totally unsuitable:

1.  You have optimised an area of the page so well that you're now seeing the law of diminshing returns - your online testing is showing smaller and smaller gains with each test and you're reaching the top of the ladder.
2.  The business teams have identified another part of the page or site that is a higher priority than the area you're testing on.
3.  The design teams want to test something game-changing, which is completely new and innovative.

This is no bad thing.

After all, iterative testing is not the be-all-and-end-all of online optimization.  There are other avenues that you need to explore, and I've mentioned previously the difference between iterative testing and creative testing.  I've also commented that fresh ideas from outside the testing program (typically from site managers who have sales targets to hit) are extremely valuable.  All you need to work out is how to integrate these new ideas into your overall testing strategy.  Perhaps your testing strategy is entirely focused on future-state (it's unlikely, but not impossible). Sometimes, it seems, iterative testing is less about science and hypotheses, and more like a game of snakes and ladders.

Three reasons I've identified for stopping iterative testing.

1.  It's quite possible that you reach the optimal size, colour or design for a component of the page.  You've followed your analysis step by step, as you would follow a trail of clues or footsteps, and it's led you to the top of a ladder (or a dead end) and you really can't imagine any way in which the page component could be any better.  You've tested banners, and you know that a picture of a man performs better than a woman, that text should be green, the call to action button should be orange and that the best wording is "Find out more."  But perhaps you've only tested having people in your banner - you've never tried having just your product, and it's time to abandon iterative testing and leap into the unknown.  It's time to try a different ladder, even if it means sliding down a few snakes first.

2.  The business want to change focus.  They have sales performance data, or sales targets, which focus on a particular part of the catalogue:  men's running shoes; ladies' evening shoes, or high-performance digital cameras.  Business requests can change far more quickly than test strategies, and you may find yourself playing catch-up if there's a new priority for the business.  Don't forget that it's the sales team who have to maintain the site, meet the targets and maximise their performance on a daily basis, and they will be looking for you to support their team as much as plan for future state.  Where possible, transfer the lessons and general principles you've learned from previous tests to give yourself a head start in this new direction - it would be tragic if you have to slide down the snake and start right at the bottom of a new ladder.

3.  On occasions, the UX and design teams will want to try something futuristic, that exploits the capabilities of new technology (such as Scene 7 integration, AJAX, a new API, XHTML... whatever).  If the executive in charge of online sales, design or marketing has identified or sponsored a brand new online technology that will probably revolutionise your site's performance, and he or she wants to test it, then it'll probably get fast-tracked through the testing process.  However, it's still essential to carry out due diligence in the testing process, to make sure you have a proper hypothesis and not a HIPPOthesis.  When you test the new functionality, you'll want to be able to demonstrate whether or not it's helped your website, and how and why.  You'll need to have a good hypothesis and the right KPIs in place.  Most importantly - if it doesn't do well, then everybody will want to know why, and they'll be looking to you for the answers.  If you're tracking the wrong metrics, you won't be able to answer the difficult questions.

As an example, Nike have an online sports shoe customisation option - you can choose the colour and design for your sports shoes, using an online palette and so on.  I'm guessing that it went through various forms of testing (possibly even A/B testing) and that it was approved before launch.  But which metrics would they have monitored?  Number of visitors who tried it?  Number of shoes configured?  Or possibly the most important one - how many shoes were purchased?  Is it reasonable to assume that because it's worked for Nike, that it will work for you, when you're looking to encourage users to select car trim colours, wheel style, interior material and so on?  Or are you creating something that's adding to a user's workload and making it less likely that they will actually complete the purchase?

So, be aware:  there are times when you're climbing the ladder of iterative testing that it may be more profitable to stop climbing, and try something completely different - even if it means landing on a snake!