Header tag

Showing posts with label testing. Show all posts
Showing posts with label testing. Show all posts

Wednesday, 7 October 2026

Battleships and A/B Testing

When you picture a strategic board game, the first thing that often comes to mind is the timeless pen‑and‑paper showdown known as Battleships.  Two players plot secret fleets on a hidden grid, then take turns calling out coordinates in hopes of sinking their opponent’s fleet. Oddly enough, the same logic that underpins this nostalgic pastime also powers the modern marketer’s most valuable tool: A/B testing.  But you knew I was going to say that in this blog, right? ;-)


There are some obvious and some obscure similarities between a round of Battleships and the iterative process of optimizing complex elements of a website, such as a navigation menu or a landing page, or a series of products on a category page. I've observed that it can take a blend of hypothesis, random probing (or sometimes just gut feel) and data‑driven refinement can turn a handful of menu items into a high‑performing traffic‑masterpiece.  You aren't going to hit the perfect combination first time, but you aren't going to have to work out all the combinations (some of them just don't make sense) to get to the optimum answer fairly quickly.

 

The Playing Field: The Grid and The Page Layout 

 

Both scenarios start with an unknown distribution. In Battleships you lack any clue where the opponent's fleet sits; on a fresh website (or on a site component that you've not tested before) you have limited insight into which menu label will attract the most clicks.  Or, you may have extensive data on engagement with your menu options, but you don't know what the most effective sequence might be.  

The first step in both games is to gather information, and the most straightforward way to do that is to make a series of probes. 

  


Random or “Best‑Guess” First Moves 

 

A seasoned Battleship player knows that a completely random first shot is often a good baseline. It gives an unbiased sample of the board and avoids early bias toward a particular quadrant. Likewise, when devising an A/B test on a navigation menu, we have to start with a baseline version (Recipe A, the current state).  This is where we start, based on the original agreed sequence for the menu (or the one that was agreed or imposed most recently).   Your first test recipe - your first guess, or challenger - will be the best guess from existing UX guidelines, past experience, engagement data, clickstream analysis or competitor analysis. Please, please, let it have some form of data backing: it's not essential, but it will certainly help.  At this stage, it's perfectly fine (in fact, it's better) if you have multiple hypotheses on the go... everybody devises a recipe, and you select the best for testing.  The number of recipes you can test will depend on your traffic, but at this stage you want to maximize your options.

I mean, wouldn't Battleships be quicker (and more exciting) if you got to pick three or four co-ordinates per turn, instead of just one?

 

Does the the randomness matter? 

 

At this point, you're focusing more on exploration, and less on exploitation.  Random probes let you explore the unknown space before focusing on promising areas.  More importantly, you'll avoid premature convergence: if you jump straight to an “obvious” menu order, you may lock yourself into a sub‑optimal layout before you ever see the data.  One things that's very important at this stage is to test more than one new idea - in Battleships, this would be like shouting out four or five guesses each turn.  A scattershot approach in your testing will yield the fastest results - you're not necessarily aiming to find a winner immediately, this is about testing a range of different hypotheses.  Will colour be helpful? Which item should go first in the list?  Which should go in the middle?  


The First Hit: Discovering a “Ship” 

 

When a player announces “B5: hit!” they’ve uncovered part of a ship. This moment gives them very specific information: a ship occupies that cell and, by the rules, must extend either horizontally or vertically. 

 

In A/B testing, the equivalent is a statistically significant lift in a key metric, for example, a 6 % increase in overall click‑through rate (CTR) for the new menu order. That lift tells you three things: 

 

1. Presence – The altered ordering resonates with users. 

2. Direction – The change (e.g., moving “Pricing” closer to the left) is beneficial. 

3. Potential shape – The magnitude of uplift hints at how many “cells” (menu items) may still be optimized. 

 

Just as a Battleship player deduces the ship’s orientation after the first round of hits (if they're lucky, or clever), the marketer can infer which type of change matters most: position, wording, visual emphasis, or grouping. 

 

Targeted Follow‑ups: “Scanning” Around a Hit 

 

In the game, once you have a hit you start probing the adjacent squares (B4, B6, A5, C5) to determine the ship’s shape. Each subsequent coordinate is a targeted test rather than a blind guess.  You shift from random guessing to focused fire, and A/B testing follows the same pattern: 

 

Recipe B in the first test showed an initial improvement (e.g., swapping “Products” and “Solutions”). 

In the second test, you can start tweaking other items in the list.  Remember that the number of possible permutations is far more than you would expect (and with just a few items in your list, the total will exceed a 10x10 Battleships board) so you're not going to hit on the perfect winner immediately (unless you are spectacularly lucky).
 

These follow‑up experiments are nested or sequential, concentrating resources on the most promising hypotheses. The data you collect after each iteration narrows the “search space,” just as each new coordinate reduces the number of possible ship layouts, and continues to point you in the right direction.  This is where multiple recipes are important in each test - you'll need to triangulate your results, not just make one-off guesses.  Multiple recipes with one or two differences between them will help you understand what matters, what helps and what makes things worse!

 

 

Mapping the “Board”: Heatmaps, Click‑maps, and Probability Grids 

 

Advanced Battleship players may use a probability heat‑map that shows where ships are *most likely* to be hidden, based on prior hits and the rules of ship lengths.  This takes a simple game and makes it remarkably complicated, but if you're up for a challenge, it's worth a try.

 

Similarly, modern website analytics provide heatmaps and click‑maps that visualize where visitors tend to gravitate. Tools such as Hotjar, Crazy Egg, or built‑in Google Analytics “Behavior Flow” generate a probability surface over the navigation bar: 

 

When you overlay this visual data onto your A/B test results, you get a dual feedback loop: the statistical lift from the test and the qualitative ink‑blot from the heatmap. Together they guide you toward the next set of coordinates (menu tweaks) with a higher chance of “sinking” the under‑performing items. 

 

 

Defining Victory: From “Sinking a Ship” to “Winning the Funnel” 

 

In Battleships, you win by sinking all opponent ships. In navigation optimization, the win condition is more nuanced, and long-time readers will immediately think of my articles on KPIs - do you have enough KPIs; has everybody agreed on the KPIs, and do you have too many? 

 

Primary goal: Increase CTR for a strategic link (e.g., “Contact”).  Watch out that you're not over-indexing on Click Through Rate, though, because of the "click for free money" challenge.

Secondary goals: Reduce bounce rate, shorten time‑to‑conversion, improve overall conversion. 

 

When a series of tests successively “sink” the low‑performing placements and elevate the high‑value ones, you achieve a balanced funnel: the menu now routes users efficiently, minimizing friction. 

 

 The Role of Iteration: The Game Never Truly Ends 

 

Even after every ship is sunk, seasoned players often start a new round, perhaps with a different grid size or ship configuration. Likewise, once a navigation menu reaches a satisfactory conversion rate, the digital landscape continues to shift: new products launch, branding evolves, and user expectations change.  Marketing introduce a new product; a new menu item is added as new features, products or services are promoted.

 

Continuous A/B testing becomes a maintenance routine, akin to a never‑ending Battleships campaign. Each new feature or redesign is a fresh fleet to place, and every subsequent test is another salvo.  Sometimes, it can feel like you're hitting moving targets, which certainly makes things more challenging, and you need to think about the principles you've learned, not just the specifics.

  

Takeaways 

 

* Uncertainty is a feature, not a flaw
Both Battleships and A/B testing thrive on systematically reducing unknowns. 


* Start simple, then target
A modest random or best
‑guess change gives you a baseline, after which data‑driven refinements pay off. 


* Visual probability aids decision
‑making
Heatmaps for navigation function the same way as ship‑probability grids in the classic game. 


* Iterative loops lead to mastery
Each successful test recipe brings you closer to an optimal menu, just as each hit brings you closer to sinking a ship. 

* Never stop playing
The digital environment evolves; continuous testing keeps you ahead, much like an endless series of Battleship rounds. 

 

Final Thought 

 

Next time you stare at a spreadsheet of A/B test results, imagine yourself peering over a Battleships board. Each data point is a coordinate, each statistically significant lift a “hit,” and each subsequent experiment a strategic scan for the rest of the vessel. By embracing the game‑like mindset of exploration, hypothesis, and disciplined iteration, you’ll transform a simple website feature into a high‑performing conduit, efficiently guiding visitors precisely where they want to go, and where you'd like them to go, too.

 

(Happy testing, and may your next “shot” be a clean hit!)

Similar posts I've written about online testing

Wednesday, 6 May 2026

Leveraging Data for Segment Identification and Personalised Experiences

In previous articles, I have explored the practical strategies for analysing user behaviour derived from website interactions and A/B testing methodologies. These techniques are particularly valuable when aiming to identify high-performing user segments and craft targeted digital experiences. However, this process often reveals unexpected insights, especially when initial testing does not yield favourable results.

Consider a scenario in which a newly introduced test design, despite its promising conceptual appeal, fails to deliver the desired impact. Quantitative results may be unequivocally negative, with every relevant key performance indicator (KPI) registering a decline. In such circumstances, the data suggest that the variation, such as Recipe B, is not simply underperforming; it has failed across the board. This presents two immediate paths forward: one may either accept the outcome, analyse the failure, learn from it, and iterate on future design efforts—or choose to segment the data to investigate underlying causes.

This latter approach can become a complex and treacherous endeavour, resembling a descent into a highly intricate and potentially misleading analytical rabbit hole.  Sorry, Alice!

For example:

  • An initial segmentation comparing new versus returning visitors may reveal that returning users did not react as negatively as first-time users.

  • A further segmentation might then show that returning users accessing the site via mobile devices performed slightly better than their desktop counterparts.

  • Digging even deeper, one might uncover that returning mobile users interested in higher-priced products exhibited improved engagement or conversion metrics.

While these findings may appear promising, they come with a significant trade-off. After applying successive layers of filtering, the relevant audience is reduced from an initial 50% of total traffic to just 4.3%. This raises a critical question: is it worth allocating substantial resources to tailor a unique experience for such a narrow segment?

For certain brands, particularly high-end luxury retailers such as Rolls Royce or Beaverbrooks, personalised attention may be both viable and beneficial. However, for businesses operating in more commoditised sectors, such as discount pet supplies, the return on investment for micro-targeting may be far less compelling.

This same dilemma applies when designing personalisation campaigns. As I have highlighted previously, two major challenges confront marketers:

  1. Acquiring and interpreting high-quality, granular data.

  2. Producing and managing sufficient content to serve these differentiated segments.

Assuming these hurdles are overcome to a satisfactory degree, one must still exercise caution in the granularity of targeting. Excessive segmentation risks over-engineering the user experience, while minimal segmentation may fail to resonate at all.

Let us consider a common digital merchandising challenge: What content should be displayed in the homepage hero banner? What recommendations should populate the "We think you'll like this..." carousel? Should your virtual storefront attempt to predict and proactively suggest specific products based on past user behaviour?

For instance, one might propose that a returning visitor be presented with a Lego set—unpromoted, without a discount—on the basis of prior browsing behaviour. This raises the question: is your targeting sophistication high enough to ensure that this suggestion will genuinely resonate?

Many practitioners point to brands such as Netflix and Amazon as exemplars in personalised recommendation systems. Statements like "Because you watched Star Trek: Deep Space Nine" provide a transparent rationale for the curated content that follows. These systems succeed because they offer breadth—providing up to 42 scrollable options—ensuring that even if the primary recommendation does not engage, several others might.

This model can be effectively emulated in retail contexts. For instance, a virtual toy store could present a curated range of 42 Lego models and invite user exploration. Such an approach is far more engaging than presenting a single product, especially when stock limitations may lead to user frustration if the highlighted item is unavailable.

A broader tactic, such as "Would you like to explore our Lego collection?" may prompt greater engagement than asserting, "We believe you want this specific model." The former understands more about user independence and improves interaction metrics (which are often paramount KPIs in digital strategy, right?).

Compare the following messages:

  • “Welcome to our toy shop! These are our favourite toys!”

  • “We think you’re interested in construction toys.”

The first message represents a generic push strategy, commonly found in homepage banners that prioritise brand objectives over user intent ('we' are more important than 'you' - the long-standing tension between marketing and customer needs). The second, although still broad, reflects an attempt to connect with presumed customer interests. In this context, even minimal targeting is advantageous—and likely more effective than overly narrow personalisation.

Let us assume predictive modelling yields:

  • A 71% likelihood a visitor is interested in Lego products.

  • A 23% likelihood they are seeking Lego Technic.

  • A 7% probability they want the Lego Technic Excavator

Rather than presenting the most specific item, a better approach would be to guide users toward the Lego Technic category and enable autonomous navigation from that point. This ensures relevancy while allowing for user discovery and choice.  Move users forwards one step down the funnel (or the rabbit hole), not two.  Maximise your chances of being correct, and let users navigate from there.

While the hypothetical website may sell any range of products, the Lego example is a globally recognisable, visually compelling use case. Naturally, any product category could be substituted in this framework. If I'd been analysing your browsing history, I might have tailored this piece to your specific interests.

Until next time!

Similar posts I've written about online testing

Friday, 24 April 2026

"Click For Free Money" on the Customer Journey

A few years ago, a colleague of mine highlighted a crucial insight: engagement on a button isn't always a reliable measure of success.  I know it's hardly an earth-shattering revelation, but it came at a time when KPIs were focusing on engagement, when he and I were looking at testing CTA wording.  Should we go for "Buy Now" or "Customise and Buy"?  Which is better - "View Details" or "Learn More"?  In short, we discovered that the best wording depended on the point in the purchase path, and we had to separate the analysis, because the button that achieved the highest click-through rate (CTR) wasn't always the one that led to the best overall conversion rate. 

In simpler terms, success isn't just about shuffling customers to the next page in our online purchase path. While persuasive wording can certainly encourage users to click and advance, we need to critically ask ourselves: does this genuinely help them progress beyond that initial stage, or are we just creating a false sense of momentum? This brings us to what I call the "Click for Free Money" fallacy. You can absolutely get people to click a button, but the subsequent page must deliver on the expectation set by that button's wording.  Click for Free Money will certainly get a lot of clicks, but there's no benefit to this approach, as people realise that the next page is actually just more information about our products, and maybe a coupon code to validate the 'free money' offer.  


The Promise and the Reality of the Click

Consider the user's mental model. When a button says "Add to Cart," the user fully expects to be taken to their cart fairly quickly. A brief, relevant detour to offer a highly personalized upsell might be acceptable, but anything more, or anything irrelevant, will create friction. Similarly, a "Customize" button should seamlessly lead to customization options, allowing the user to tailor their product without unnecessary distractions. And perhaps most critically, a "Checkout" button must initiate the checkout process. This is not the time for additional upsells, cross-sells, or any other interruptions that could derail a customer who is ready to complete their purchase. Each click builds an expectation, and failing to meet that expectation, even subtly, erodes trust and increases the likelihood of abandonment.  A lack of 'free money' will erode trust remarkably quickly, although it will probably be on the 'free money page, not the previous page.  Still, if you and your team are prepared to do anything to get users to move forwards from your page, you could deploy this tactic.  It depends on what your KPI actually is (and who's looking at the long-term journey).

The Detrimental Cost of Rushing Customers

Pushing customers forward too quickly isn't just inefficient; it can be detrimental to long-term customer relationships and conversion rates. If you don't continually reinforce the value proposition and reassure the user along their journey, they're highly likely to drop out at a later, more critical stage. For instance, if you enticed a user with a 10% discount while they were casually Browse your site, that discount needs to be prominently displayed and automatically applied on the cart page. Hiding it, or making them jump through hoops to redeem it, is a surefire way to lose a sale and disappoint a customer.

It's vital to shift our focus: moving customers towards a purchase isn't as important as guiding them through their journey to choose the best product for them. This is a fundamental difference between a transactional mindset and a customer-centric approach.

Think of it like a pushy sales assistant in a physical store. Imagine walking in, and immediately, an assistant shoves an item into your hands, puts an arm around your shoulder, and starts steering you towards the checkout. When you haltingly ask, "But is this my size?" they might dismissively respond, "Probably, yes." If you inquire, "Will it perform better than my previous widget?" they might curtly say, "Who cares? Cash or card?" Or if you try one more time, "But is it quieter and more efficient than the old model?" they might simply reply, "Yes. Is the shipping address the same as the billing address?"

While this sales assistant might technically "drag" the customer to the start of the checkout process, what was the true cost? A frustrated, confused, and potentially alienated customer who probably won't complete the purchase and certainly won't return (worse still, they'll tell their friends).

Preparing for the Next Step: The True Purpose of the Funnel

Often, all we accomplish by being overly aggressive or deceptive in the early stages of the funnel is simply shifting the exit point to a later stage in the user's journey. The customer still leaves, but now they're more annoyed because they've invested more time and effort. Instead, it's far more important and beneficial to leverage each step of the customer journey to authentically prepare users for what they'll encounter next. Each stage should provide value to the user, clarify information, and build confidence and trust.  In a situation where different parts of the journey belong to different developers, teams and managers, 

When a user is genuinely informed, reassured, and ready to take the next step—because they understand the value and feel confident in their choice—then and only then, should we make it effortless for them to move forward. This approach fosters trust, reduces abandonment rates, and ultimately leads to more satisfied customers and higher conversion rates in the long run.  We may not offer 'free money', but by subtly pushing users forwards in a path that they aren't fully prepared to take - by providing quicker paths or by removing content that is actually useful to users - we will merely persuade them to move forwards from our page into a situation where they're increasingly likely to leave from the next page.

But still, our next-step page flow metrics show a lovely funnel that shows we moved 10% more users to the next step.  Our analysis, then needs to show how many users move forwards to the next-next step, and then actually completed the full purchase journey.

Consider a simplified conversion funnel, from a landing page through a website to completing the checkout process.


Similar to the 'did you just code a distraction?' question - we see that users move forwards to the next step at a much higher rate.  950/998 = 95%, compared to 75% for control, and 65% for Recipe B.  See, you coded a winner, it just happened to be different than the one you expected!

However, when we look further down the funnel, we find that there's a massive 50% drop-off for Recipe C.  We failed to deliver free money, and people left.  Unsurprisingly, fewer visitors then reach the lower funnel states, and Recipe B is actually the winner (with Recipe C coming behind control).

And if you think that's far-fetched, then perhaps replace 'Free money' with something more believable... like, perhaps, "Add to Basket."  Do customers really get to add the item to basket when they click that button?  Or do they have to select a size, a colour, an upgrade, a guarantee, a warranty or something else first?

Or do you promise things with your CTAs that you aren't really delivering?  "Find out more" needs to show more product information about the product it's connected to.  Getting clicks is easy, but you need to keep an eye on the next step, not just the one you're testing.

Similar posts I've written about online testing




Wednesday, 11 February 2026

Why Too Many Cooks Spoil the A/B Testing Roadmap


Why Too Many Cooks Spoil the A/B Testing Roadmap

How to build a lean, decisive testing program without sacrificing collaboration

When Consensus Kills Velocity

A/B testing is - in theory - designed to answer questions quickly: Which experience performs better? Which idea deserves further research and investment from a budget-conscious company and a stretched development team?

The power of testing lies in short cycles, sharp hypotheses, and decisive execution. Yet in many organizations, the A/B testing roadmap becomes a negotiation table. Every team wants to weigh in, every stakeholder demands review, and the result is predictable: delays, diluted experiments, and very little learning.  The time taken to run a test (i.e. from launch to confidence) starts to pale in comparison to the amount of time it takes to decide what should be in the test.

If your testing program needs a large, multi-functional committee to approve every hypothesis, pick every variant, and sign off every launch, you don’t have a testing program— you have a meeting calendar. It's a painful paradox: a method designed to reduce risk and speed up learning turns into a process that creates risk by slowing decisions and reduces learning by launching compromised tests that fail to answer meaningful questions.

The Ubuntu Concept—and Why Sequence Matters

Ubuntu, often summarized as “I am because we are,” is a powerful philosophy of interdependence, empathy, and shared success. In short, "If you want to go far, travel together; if you want to go fast, travel alone."  There can often be an emphasis on teamwork and consensus, but the truth is, with testing, the fewer people there are, the better.  Testing is such a powerful tool, and it takes so long to develop a test that everybody wants to get involved in deciding what goes in the test, and it has to be pixel perfect.  Tragically, testing's power becomes its downfall - tests take so long because everybody wants to have their input, but they take so long because everybody's input has to be taken into consideration.  There needs to be a time when a leader says, "Enough!"  When somebody takes responsibility and authority, and actually includes fewer people - not more - in the testing process.

The better application is Ubuntu-after-evidence. Before the test, a small, empowered team should define hypotheses, success metrics, and launch criteria with minimal friction. After the test, the broader community can engage—reviewing results, sharing learnings, and co-creating improvements that benefit everyone.

In other words, we preserve Ubuntu’s spirit of shared learning and collective improvement without letting endless pre-test consensus slow down the very mechanism that generates the learning.

Why Fewer People Make Testing Better

A lean testing team is not anti-collaboration. It is pro-clarity. In my experience, the most effective testing programs are lean and decisive because velocity is what matters most. A/B testing’s value decays with time. Long approval and planning cycles reduce relevance and increase opportunity cost. Short cycles mean you learn faster, pivot earlier, and deploy compounding gains sooner.

Clear ownership prevents drift. When the roadmap belongs to everyone, it effectively belongs to no one, and everyone has the potential to veto a test immediately before launch (after all the effort of aligning everybody, agreeing designs and actually writing the test code). A focused owner or small squad can prioritize tests based on impact and feasibility, not politics. Committees tend to compromise, merging ideas into multi-variable tests that answer nothing clearly. Lean teams keep hypotheses tight, variants minimal, and interpretations crisp.

Everybody has the opinion that evidence should replace opinion, except where it contradicts what they want to do.  If an objection sounds like, "We don't want to run that test in case it contradicts what we've already planned to build", then it probably is. A streamlined process moves debates downstream. Instead of arguing which idea is better, we test quickly and gather data, and then prove which idea was better.

Who Should Be in the Core Team?

The lean model works because it assigns clear roles and decision rights. At the center is the Product Manager, who owns the testing backlog and makes go/no-go decisions. They ensure that tests align with business objectives and that prioritization reflects impact rather than politics.

Alongside the PM, or combined in the same role, is the Experimentation or Data Lead, who designs hypotheses, defines success metrics, and ensures statistical rigor. This person calculates sample sizes, sets stopping rules, and interprets results so that every test produces actionable insights.  This role can is a specialized testing role, where a detailed and nuanced understanding of testing, significance, confidence, the testing pipeline, dev capabilities are vital.  They will work closely with the Product Manager when the process of testing has been documented and established.

The Engineering representative plays a critical role in implementing variants, managing feature flags, and ensuring telemetry accuracy. They safeguard performance and security standards, making sure experiments do not compromise the user experience.

The Business Lead works with the team to bring forward tactical business needs, which can feed directly into testing requirements.  Do you need to sell more high-price menswear this quarter?  Are you expanding into audiobooks as well as audio streaming services?  Where, from a business perspective, should we focus our short-term efforts?

A UX or Design specialist ensures that variants are user-friendly, accessible, and on-brand within pre-agreed guidelines. Their involvement prevents usability and accessibility issues and maintains consistency without introducing unnecessary complexity.  However, it is absolutely critical that the UX and design specialists support the velocity of the testing roadmap by supplying the assets that are needed for testing without going through their own endless and unnecessary cycles of 'discovery and framing' or 'research', which will single-handedly bring any test roadmap grinding to a complete stop.   

Legal and privacy advisors contribute upfront by defining evergreen guardrails for compliance and data handling. They are consulted during framework design, not as ad hoc gatekeepers for every test.

Finally, stakeholders remain informed and can submit ideas through a structured intake process, but they do not have veto power over launches. Their role is to provide context and consume learnings, not to slow down execution.  Stakeholders have visibility, not veto.

A Common Failure Pattern—and How to Fix It

Some organizations follow a familiar failure pattern. The quarterly roadmap is assembled in a multi-department forum. Hypotheses get neutered to accommodate every stakeholder’s request. Legal, Brand, Product, and Design each add gates. Engineering estimates are based on maximum-complexity variants. There are changes during development, there are changes during pre-launch quality checks.  Scenarios were not considered, and designers Tests are launched late with weak hypotheses and unclear success metrics.  Everybody wanted a slice of the testing cake, and now it's a half-baked mess. Results are inconclusive, and trust in testing erodes.

The fix is straightforward but requires discipline. Empower the core team described above with decision rights for hypothesis selection, prioritization, and launch. Set standard guardrails for brand, legal, and privacy constraints that are pre-agreed and do not require ad hoc approvals. Publish metrics definitions that are standardized and documented. Run short test sprints with limited dependencies. Finally, socialize results broadly through summaries and open Q&A sessions—Ubuntu applied post-evidence.

Conclusion: Ubuntu After Evidence

A/B testing is a learning engine. To keep it running, you must minimize friction at the point of decision and maximize inclusion at the point of reflection. Ubuntu teaches us that we rise together; testing teaches us that we rise faster when we let data lead. So don’t abandon Ubuntu—sequence it. Empower a small team to move quickly within clear guardrails. Then invite the broader community to interpret results, celebrate wins, and turn lessons into shared progress. Fewer people at the decision-making table means more learning for everyone.

Similar posts I've written about online testing


Wednesday, 3 September 2025

Did You Just Code a Distraction?

In the fast-paced world of web development and digital marketing, creating new features to enhance user experience is not just a common practice, it's pretty much par for the course.  Everybody has ideas about making the website better, and it usually involves some sort of magical feature that will help users find the exact product they want in a mere matter of seconds - a shopping genie, or something involving AI.   The new feature for your website is almost always visually appealing, interactive, and the designers are confident it will boost user engagement with their favourite persona. Before rolling it out, you wisely decide to run an A/B test to measure its effectiveness.

So you code the test, and you run it for the usual length of time.  You follow all the advice on LinkedIn about statistical significance (we can all describe it and we all have our own ways of calculating it, thank you) and getting a decent sample size. The test results are in, and they’re a mixed bag. On one hand, the new feature is a hit in terms of engagement. It receives twice as many clicks compared to the other clickable elements on the page, such as those lovely banners, the promotional links and the pretty pictures. However, your  deeper dive into the data reveals a concerning trend. While the new feature attracts a lot of attention and engagement, the conversion rate for users who interact with it is only around 2.5%. In contrast, the conversion rate for users who engage with the existing content on the page is significantly higher, at around 4.1%. 

This is key.  It really is insufficient to look only at engagement data (click through rate) as a success metric.  Yes, it is important, but it is not enough.  After all, if you want to create a banner with a high click rate, then you could simply write "Buy one get one free", or better still, "Buy one get two free - click here."  It's essential that you set expectations with your banners, calls to action and features - what's to stop you from writing "Click here for free money!"?  If your priority in testing is to generate clicks, then you'll degenerate into coding the on-site versions of clickbait, and that's a terrible waste of a potential lead.  You might think 'click here for free money' is sharp practice, but how close do you want to get to that line?  Where does your desire to generate interest in your new feature overtake the overall strategy for your website?  Is irritating and misleading your customers really part of that strategy?

So, what went wrong with your test? The short answer is that you coded a pretty distraction. Here’s a breakdown of why this happens and how to address it:

Misalignment with User Intent

The new feature, despite being engaging, may not align with the primary intent of your users. If it diverts their attention away from the main conversion paths, it can reduce overall effectiveness. Users might be intrigued by the new feature but not find it relevant to their immediate needs.  You misunderstood your persona's motivation; it might be time to write your persona with this additional information.  I've said before that designers will sometimes create personas that would find this feature useful - it's like putting the cart before the horse.  Write the persona and think what that user actually wants; don't create the feature and try to produce a persona to fit.

Cognitive Load

Introducing a new element can increase the cognitive load on users.  Or, in plain English - you've given them something else to read, understand, think about and add to their decision-making process.  Why would we do that?

Users now have to process and understand this new feature, which can be mentally taxing. If the feature doesn’t provide immediate value or clarity, users might get distracted and abandon their original task.  They used up their time, effort and patience while interacting with your new feature, and gave up on their primary purpose (which was to buy something from you).

Disruption of User Flow

A well-designed website guides users smoothly towards conversion goals. A new feature that stands out too much can disrupt this flow, causing users to deviate from their intended path. This disruption can lead to lower conversion rates, as users get sidetracked.  How do I get back to my intended path?  This new feature has proved to be the next big shiny thing, and while it's attracting user engagement, it's confusing them and preventing them from getting to where they wanted to.  

The Solutions

To avoid coding distractions, consider the following strategies:

User-Centric Design

Not my favourite phrase, since it leads to design without A/B testing and designers designing for their favourite personas.  Ensure that any new feature is designed with the user’s needs and goals in mind. Conduct user research to understand what your audience values and how they navigate your site, and then align your new features and your development roadmap with these insights.  This will enhance, rather than disrupt, the user experience, and reduce the amount time of wasted on development the next shiny bauble - it looks nice and impresses senior management, but is not good for users.  

Incremental Testing

Instead of launching a fully-fledged feature, start with a minimal viable version and test its impact incrementally. This approach allows you to gather feedback and make necessary adjustments before a full rollout.  Use test data in conjunction with user research to gain a full picture of what you thought was going to happen, and what actually happened.

Clear Value Proposition

Make sure the new feature has a clear and compelling value proposition. Users should immediately understand its purpose and how it benefits them. If you can't explain the new feature and why it's helpful, then you've probably coded a puzzle.  This clarity can help integrate the feature seamlessly into the user journey.  If the test 'fails', then you'll learn that the value proposition you promoted was not what users wanted to read, and you can try something different.

Monitor and Iterate

Continuously monitor the performance of new features and be ready to iterate based on user feedback and data. If a feature is not performing as expected, don’t hesitate to tweak or even remove it to maintain a smooth user experience.  It's time to swallow your pride and start again.  If you change direction now, you'll have less distance to travel than if you wait six months before unimplementing the eye-catching blunder you've launched.

Conclusion

In the quest to innovate and improve user engagement, it’s crucial to strike a balance between novelty and functionality. While new features can attract attention, they must also support the primary goals of your website. By focusing on user-centric design, incremental testing, and clear value propositions, you can avoid coding distractions and create features that truly enhance the user experience.

Other Web Analytics and Testing Articles I've Written

How not to segment test data (even if your stakeholders ask you to)

Designing Personas for Design Prototypes
Web Analytics - Gathering Requirements from Stakeholders
Analysis is Easy, Interpretation Less So
Telling a Story with Web Analytics Data
Reporting, Analysing, Testing and Forecasting
Pages with Zero Traffic

Wednesday, 27 August 2025

Do You Know How Well Your Test Will Perform?

 There are various ways of running tests - or more specifically, there are various ways of generating test hypotheses.  One that I've come across over the years, and increasingly so more recently, is the 'guess how well your test is going to perform' approach.  It's not called that, but it seems to me to be the most succinct description.  Apparently, we should already have a target improvement in mind before we even start the test.



"If we change the pictures on our site from cats to dogs, then we'll see a 3.5% increase in conversion."
"If we promote construction toys ahead of action figures, then we'll see a 4% lift in revenue."

If you know that's going to happen, why don't you do it anyway?

The main underlying challenge I have is that it's almost impossible to quantify the improvement you're going to get.  How do you know?

Well, let's attempt the calculation (with hypothetical numbers all the way through).

Let's say our latest campaign landing page has a bounce rate (user lands on page, then exits without visiting any other pages) of 75%.  10% engage with site search, 10% click on the menus at the top of the page, and 5% click on the content on the page (there are a few banners and a few links).

We've identified that most users aren't scrolling past the first set of banners and links, and we therefore hypothesise that if we make the banners smaller, and reduce the amount of padding around the links, that we can increase engagement with the content in the lower half of the page, and therefore improve the bounce rate.  We believe we can get 50% more links above the fold, and therefore increase the in-page engagement rate from 5% to 7.5%.  We will assume (and this is the fun bit) that this additional traffic converts at the same rate as the 5% we have so far, and therefore, we'll get a revenue lift of 50%.  This sounds like a lot, but given that the engagement rate is going up from a small number to a slightly larger number, it's unlikely to be a huge revenue lift in dollar terms (unless you're pouring in huge volumes of traffic - and watching it bounce at a rate of 75%).

Perhaps that was an over-simplification.  But if we knew that our test will give us a 5% lift (and we've still decided to test it), what happens when we launch the test?  Presumably, we'll stop it when it reaches the 5% lift, irrespective of the confidence level.  But what happens if it doesn't get to 5%?  What if it stubbornly sits at 4%?  Or maybe just 3%?  Did the test win, or did it lose?  In classical scientific terms, it lost, since we disproved our overly-specific hypothesis.  But from a business perspective, it still won, just not by as much as we had originally expected.  Would you go into a meeting with the marketing manager and say, "Sorry, Jim, our test only achieved a 3% revenue lift, so we've decided it was a failure."?

For me, it comes down to two arguments: 

If you can forecast your test result with a high degree of certainty, based on considerable evidence for your hypothesis, it's probably not worth testing and you should implement already.  Testing is best used for edge-cases with some degree of uncertainty. 

If, on the other hand, you have identified a customer problem with your site, and you can see that fixing it will give you a revenue lift - but you don't know how to fix it - then that's very good grounds for testing.  The hypothesis is not, "If we fix this problem, we'll get a 6% revenue lift," but, "If we fix this problem in this way then we'll get a revenue lift".  And that's where you need to encourage the website analysts and the customer feedback department (or the complaints department, or whoever advocates for customers within your company) to come together and find out where the problems are, and what they are, and how to address them.

That will undoubtedly bring good test ideas, and that's what you're looking for, even if you don't know how much revenue lift it will provide.

Other Web Analytics and Testing Articles I've Written

How not to segment test data (when your stakeholders want you to adapt your data)
Web Analytics - Gathering Requirements from Stakeholders
Analysis is Easy, Interpretation Less So (and why it's more valuable)
Telling a Story with Web Analytics Data
Reporting, Analysing, Testing and Forecasting
Pages with Zero Traffic


Sunday, 24 November 2024

Testing versus Implementing - why not just switch it on?

"Why can't we just make a change and see what happens? Why do we have to build an A/B test - it takes too long!  We have a roadmap, a pipeline and a backlog, and we haven't got time."

It's not always easy to articulate why testing is important - especially if your company is making small, iterative, data-backed changes to the site and your tests consistently win (or, worse still, go flat).  The IT team is testing carefully and cautiously, but the time taken to build the test and run it is slowing down everybody's pipelines.  You work with the IT team to build the test (which takes time), it runs (which takes even more time), you analyze the test (why?) and you show that their good idea was indeed a good idea.  Who knew?


Ask an AI what a global IT roadmap looks like...

However, if your IT team is building and deploying something to your website - a new way of identifying a user's delivery address; or a new way of helping users decide which sparkplugs or ink cartridges or running shoes they need - something new, innovative and very different, then I would strongly recommend that you test it with them, even if there is strong evidence for its effectiveness.  Yes, they have carried out user-testing and it's done well.  Yes, their panel loved it.  Even the Head of Global Synergies liked it, and she's a tough one to impress.  Their top designers have spent months in collaboration with the project manager, and their developers have gone through the agile process so many times that they're as flexible as ballet dancers.  They've barely reached the deadline for pre-Christmas implementation, and now is the time to implement it.  It is ready.  However, the Global Integration Leader has said that they must test before they launch, but that's okay as they have allocated just enough time for a pre-launch A/B test, then they'll go live as soon as the test is complete.


Sarah Harries, Head of Global Synergies

Everything hinges on the test launching on time, which it does.  Everybody in the IT team is very excited to see how users engage with the new sparkplug selection tool and - more importantly for everybody else - how much it adds to overall revenue.  (For more on this, remember that clicks aren't really KPIs). 

But the test results come back: you have to report that the test recipe is underperforming at a rate of 6.3% conversion drop.  Engagement looks healthy at 11.7%, but those users are dragging down overall performance.  The page exit rate is lower, but fewer users are going through checkout and completing a purchase.  Even after two full weeks, the data is looking negative.  

Can you really recommend implementing the new feature?  No; but that's not the end of the story.  It's your job to now unpick the data, and turn analysis into insights:  why didn't it win?!

The IT team, understandably, want to implement.  After all, they've spent months building this new selector and the pre-launch data was all positive.  The Head of Global Synergies is asking them why it isn't on the site yet.  Their timeline allowed three weeks for testing and you've spent three weeks testing.  Their unspoken assumption was that testing was a validation of the new design, not a step that might turn out to be a roadblock, and they had not anticipated any need for post-test changes.  It was challenging enough to fit in the test, and besides, the request was to test it.

It's time to interrogate the data.

Moreover, they have identified some positive data points:

*  Engagement is an impressive 11.7%.  Therefore, users love it.
*  The page exit rate is lower, so more people are moving forwards.  That's all that matters for this page:  get users to move forwards towards checkout.
*  The drop in conversion is coming from the pages in the checkout process.  That can't be related to the test, which is in the selector pages.  It must be a checkout problem.

They question the accuracy of your test data, which contradicts all their other data.

* The sample size is too small.
* The test was switched off before it had a chance to recover its 6.3% drop in conversion

They suggest that the whole A/B testing methodology is inaccurate.

* A/B testing is outdated and unreliable.  
* The split between the two groups wasn't 50-50.  There are 2.2% more visitors in A than B.

Maybe they'll comment that the data wasn't analyzed or segmented correctly, and they make some points about this:

* The test data includes users buying other items with their sparkplugs.  These should be filtered out.
* The test data must have included users who didn't see the test experience.
* The data shows that users who browsed on mobile phones only performed at -5.8% on conversion, so they're doing better than desktop users.

Remember:  none of this is personal.  If they're sinking down to questioning the validity of your testing method, then you're probably doing a good job.  A/B testing packages don't come with bulletproof helmets to protect the messengers.  A question that is well worth asking is: would they question the validity of A/B testing if your results were flat or positive?  Hmmm?

In this situation, you are, despite your best efforts, constructively criticising a project that they've spent weeks or even months polishing and producing.  Nobody until this point has even dared to criticise their work, and in fact everybody has said how good it is (and the alternative was to go back and redraw or rewrite code, and nobody wants that).  It's not your fault, your job is to present the data and to provide insights based on it.  As a testing professional, your job is to run and analyse tests, not to be swayed into showing the data in a particular way.

They ran the test at the request of the Global Integration Leader, and burnt three weeks  waiting for the test to complete (and maybe another week or so for the analysis).  The deadline for implementing the new sparkplug selector is Tuesday, and they can't stop the whole IT roadmap (which is dependent on this first deployment) just because one test showed some negative data.  They would have preferred not to test it at all, but it remains your responsibility to share the test data with other stakeholders in the business, marketing and merchandizing teams, who have a vested interest in the site's financial performance.  It's not easy, but it's still part of your role to present the unbiased, impartial data that makes up your test analysis, along with the data-driven recommendations for improvements.

It's not your responsibility to make the go/no-go decision, but it is up to you to ensure that the relevant stakeholders and decision-makers have the full data set in front of them when they make the decision.  They may choose to implement the new feature anyway, taking into account that it will need to be fixed with follow-up changes and tweaks once it's gone live.  It's a healthy compromise, providing that they can pull two developers and a designer away from the next item on their roadmap to do retrospective fixes on the new selector.  
Alternatively, they may postpone the deployment and use your test data to address the conversion drops that you've shared.  How are the conversion drop and the engagement data connected?  Is the selector providing valid and accurate recommendations to users?  Does the data show that they enter their car colour and their driving style, but then go to the search function when they reach a question about their engine size?  Is the sequence of questions optimal?  Make sure that you can present these kinds of recommendations - it shows the value of testing, as your stakeholders would not be able to identify these insights from an immediate implementation.

So - why not just switch it on?  Here are four good reasons to share with your stakeholders:

* Test data will give you a comparison of whole-site behaviour - not just 'how many people engaged with the new feature?' but also 'what happens to those people who clicked?' and 'how do they compare with users who don't have the feature?'
* Testing will also tell you about  the financial impact of the new feature (good for return-on-investment calculations, which are tricky with seasonality and other factors to consider)
*  Testing has the key benefit that you can switch it off - at short notice, and at any time.  If the data shows that the test recipe is badly losing money then you identify this, and after a discussion with any key stakeholders, you can pull the plug within minutes.  And you can end the test at any time - you don't have to wait until the next IT deployment window to undeploy the new feature. 
* Testing will give you useful data quickly - within days you'll see how it's performing; within weeks you'll have a clear picture.