"Click For Free Money" on the Customer Journey

A few years ago, a colleague of mine highlighted a crucial insight: engagement on a button isn't always a reliable measure of success.  I know it's hardly an earth-shattering revelation, but it came at a time when KPIs were focusing on engagement, when he and I were looking at testing CTA wording.  Should we go for "Buy Now" or "Customise and Buy"?  Which is better - "View Details" or "Learn More"?  In short, we discovered that the best wording depended on the point in the purchase path, and we had to separate the analysis, because the button that achieved the highest click-through rate (CTR) wasn't always the one that led to the best overall conversion rate. 

In simpler terms, success isn't just about shuffling customers to the next page in our online purchase path. While persuasive wording can certainly encourage users to click and advance, we need to critically ask ourselves: does this genuinely help them progress beyond that initial stage, or are we just creating a false sense of momentum? This brings us to what I call the "Click for Free Money" fallacy. You can absolutely get people to click a button, but the subsequent page must deliver on the expectation set by that button's wording.  Click for Free Money will certainly get a lot of clicks, but there's no benefit to this approach, as people realise that the next page is actually just more information about our products, and maybe a coupon code to validate the 'free money' offer.  

The Promise and the Reality of the Click

Consider the user's mental model. When a button says "Add to Cart," the user fully expects to be taken to their cart fairly quickly. A brief, relevant detour to offer a highly personalized upsell might be acceptable, but anything more, or anything irrelevant, will create friction. Similarly, a "Customize" button should seamlessly lead to customization options, allowing the user to tailor their product without unnecessary distractions. And perhaps most critically, a "Checkout" button must initiate the checkout process. This is not the time for additional upsells, cross-sells, or any other interruptions that could derail a customer who is ready to complete their purchase. Each click builds an expectation, and failing to meet that expectation, even subtly, erodes trust and increases the likelihood of abandonment.  A lack of 'free money' will erode trust remarkably quickly, although it will probably be on the 'free money page, not the previous page.  Still, if you and your team are prepared to do anything to get users to move forwards from your page, you could deploy this tactic.  It depends on what your KPI actually is (and who's looking at the long-term journey).

The Detrimental Cost of Rushing Customers

Pushing customers forward too quickly isn't just inefficient; it can be detrimental to long-term customer relationships and conversion rates. If you don't continually reinforce the value proposition and reassure the user along their journey, they're highly likely to drop out at a later, more critical stage. For instance, if you enticed a user with a 10% discount while they were casually Browse your site, that discount needs to be prominently displayed and automatically applied on the cart page. Hiding it, or making them jump through hoops to redeem it, is a surefire way to lose a sale and disappoint a customer.

It's vital to shift our focus: moving customers towards a purchase isn't as important as guiding them through their journey to choose the best product for them. This is a fundamental difference between a transactional mindset and a customer-centric approach.

Think of it like a pushy sales assistant in a physical store. Imagine walking in, and immediately, an assistant shoves an item into your hands, puts an arm around your shoulder, and starts steering you towards the checkout. When you haltingly ask, "But is this my size?" they might dismissively respond, "Probably, yes." If you inquire, "Will it perform better than my previous widget?" they might curtly say, "Who cares? Cash or card?" Or if you try one more time, "But is it quieter and more efficient than the old model?" they might simply reply, "Yes. Is the shipping address the same as the billing address?"

While this sales assistant might technically "drag" the customer to the start of the checkout process, what was the true cost? A frustrated, confused, and potentially alienated customer who probably won't complete the purchase and certainly won't return (worse still, they'll tell their friends).

Preparing for the Next Step: The True Purpose of the Funnel

Often, all we accomplish by being overly aggressive or deceptive in the early stages of the funnel is simply shifting the exit point to a later stage in the user's journey. The customer still leaves, but now they're more annoyed because they've invested more time and effort. Instead, it's far more important and beneficial to leverage each step of the customer journey to authentically prepare users for what they'll encounter next. Each stage should provide value to the user, clarify information, and build confidence and trust.  In a situation where different parts of the journey belong to different developers, teams and managers, 

When a user is genuinely informed, reassured, and ready to take the next step—because they understand the value and feel confident in their choice—then and only then, should we make it effortless for them to move forward. This approach fosters trust, reduces abandonment rates, and ultimately leads to more satisfied customers and higher conversion rates in the long run.  We may not offer 'free money', but by subtly pushing users forwards in a path that they aren't fully prepared to take - by providing quicker paths or by removing content that is actually useful to users - we will merely persuade them to move forwards from our page into a situation where they're increasingly likely to leave from the next page.

But still, our next-step page flow metrics show a lovely funnel that shows we moved 10% more users to the next step.  Our analysis, then needs to show how many users move forwards to the next-next step, and then actually completed the full purchase journey.

Consider a simplified conversion funnel, from a landing page through a website to completing the checkout process.

Similar to the 'did you just code a distraction?' question - we see that users move forwards to the next step at a much higher rate.  950/998 = 95%, compared to 75% for control, and 65% for Recipe B.  See, you coded a winner, it just happened to be different than the one you expected!

However, when we look further down the funnel, we find that there's a massive 50% drop-off for Recipe C.  We failed to deliver free money, and people left.  Unsurprisingly, fewer visitors then reach the lower funnel states, and Recipe B is actually the winner (with Recipe C coming behind control).

And if you think that's far-fetched, then perhaps replace 'Free money' with something more believable... like, perhaps, "Add to Basket."  Do customers really get to add the item to basket when they click that button?  Or do they have to select a size, a colour, an upgrade, a guarantee, a warranty or something else first?

Or do you promise things with your CTAs that you aren't really delivering?  "Find out more" needs to show more product information about the product it's connected to.  Getting clicks is easy, but you need to keep an eye on the next step, not just the one you're testing.

Similar posts I've written about online testing

Getting an online testing program off the ground
Building Momentum in Online testing
How many of your tests win?

Why Too Many Cooks Spoil the A/B Testing Roadmap

How to build a lean, decisive testing program without sacrificing collaboration

When Consensus Kills Velocity

A/B testing is - in theory - designed to answer questions quickly: Which experience performs better? Which idea deserves further research and investment from a budget-conscious company and a stretched development team?

The power of testing lies in short cycles, sharp hypotheses, and decisive execution. Yet in many organizations, the A/B testing roadmap becomes a negotiation table. Every team wants to weigh in, every stakeholder demands review, and the result is predictable: delays, diluted experiments, and very little learning.  The time taken to run a test (i.e. from launch to confidence) starts to pale in comparison to the amount of time it takes to decide what should be in the test.

If your testing program needs a large, multi-functional committee to approve every hypothesis, pick every variant, and sign off every launch, you don’t have a testing program— you have a meeting calendar. It's a painful paradox: a method designed to reduce risk and speed up learning turns into a process that creates risk by slowing decisions and reduces learning by launching compromised tests that fail to answer meaningful questions.

The Ubuntu Concept—and Why Sequence Matters

Ubuntu, often summarized as “I am because we are,” is a powerful philosophy of interdependence, empathy, and shared success. In short, "If you want to go far, travel together; if you want to go fast, travel alone."  There can often be an emphasis on teamwork and consensus, but the truth is, with testing, the fewer people there are, the better.  Testing is such a powerful tool, and it takes so long to develop a test that everybody wants to get involved in deciding what goes in the test, and it has to be pixel perfect.  Tragically, testing's power becomes its downfall - tests take so long because everybody wants to have their input, but they take so long because everybody's input has to be taken into consideration.  There needs to be a time when a leader says, "Enough!"  When somebody takes responsibility and authority, and actually includes fewer people - not more - in the testing process.

The better application is Ubuntu-after-evidence. Before the test, a small, empowered team should define hypotheses, success metrics, and launch criteria with minimal friction. After the test, the broader community can engage—reviewing results, sharing learnings, and co-creating improvements that benefit everyone.

In other words, we preserve Ubuntu’s spirit of shared learning and collective improvement without letting endless pre-test consensus slow down the very mechanism that generates the learning.

Why Fewer People Make Testing Better

A lean testing team is not anti-collaboration. It is pro-clarity. In my experience, the most effective testing programs are lean and decisive because velocity matters above all. A/B testing’s value decays with time. Long approval and planning cycles reduce relevance and increase opportunity cost. Short cycles mean you learn faster, pivot earlier, and deploy compounding gains sooner.

Clear ownership prevents drift. When the roadmap belongs to everyone, it effectively belongs to no one. A focused owner or small squad can prioritize tests based on impact and feasibility, not politics. Committees tend to compromise, merging ideas into multi-variable tests that answer nothing clearly. Lean teams keep hypotheses tight, variants minimal, and interpretations crisp.

Everybody has the opinion that evidence should replace opinion, except where it contradicts what they want to do. A streamlined process moves debates downstream. Instead of arguing which idea is better, we test quickly and gather data, and then prove which idea was better.

Who Should Be in the Core Team?

The lean model works because it assigns clear roles and decision rights. At the center is the Product Manager, who owns the testing backlog and makes go/no-go decisions. They ensure that tests align with business objectives and that prioritization reflects impact rather than politics.

Alongside the PM, or combined in the same role, is the Experimentation or Data Lead, who designs hypotheses, defines success metrics, and ensures statistical rigor. This person calculates sample sizes, sets stopping rules, and interprets results so that every test produces actionable insights.  This role can is a specialized testing role, where a detailed and nuanced understanding of testing, significance, confidence, the testing pipeline, dev capabilities are vital.  They will work closely with the Product Manager when the process of testing has been documented and established.

The Engineering representative plays a critical role in implementing variants, managing feature flags, and ensuring telemetry accuracy. They safeguard performance and security standards, making sure experiments do not compromise the user experience.

The Business Lead works with the team to bring forward tactical business needs, which can feed directly into testing requirements.  Do you need to sell more high-price menswear this quarter?  Are you expanding into audiobooks as well as audio streaming services?  Where, from a business perspective, should we focus our short-term efforts?

A UX or Design specialist ensures that variants are user-friendly, accessible, and on-brand within pre-agreed guidelines. Their involvement prevents usability and accessibility issues and maintains consistency without introducing unnecessary complexity.  However, it is absolutely critical that the UX and design specialists support the velocity of the testing roadmap by supplying the assets that are needed for testing without going through their own endless and unnecessary cycles of 'discovery and framing' or 'research', which will single-handedly bring any test roadmap grinding to a complete stop.   

Legal and privacy advisors contribute upfront by defining evergreen guardrails for compliance and data handling. They are consulted during framework design, not as ad hoc gatekeepers for every test.

Finally, stakeholders remain informed and can submit ideas through a structured intake process, but they do not have veto power over launches. Their role is to provide context and consume learnings, not to slow down execution.  Stakeholders have visibility, not veto.

A Common Failure Pattern—and How to Fix It

Some organizations follow a familiar failure pattern. The quarterly roadmap is assembled in a multi-department forum. Hypotheses get neutered to accommodate every stakeholder’s request. Legal, Brand, Product, and Design each add gates. Engineering estimates are based on maximum-complexity variants. There are changes during development, there are changes during pre-launch quality checks.  Scenarios were not considered, and designers Tests are launched late with weak hypotheses and unclear success metrics.  Everybody wanted a slice of the testing cake, and now it's a half-baked mess. Results are inconclusive, and trust in testing erodes.

The fix is straightforward but requires discipline. Empower the core team described above with decision rights for hypothesis selection, prioritization, and launch. Set standard guardrails for brand, legal, and privacy constraints that are pre-agreed and do not require ad hoc approvals. Publish metrics definitions that are standardized and documented. Run short test sprints with limited dependencies. Finally, socialize results broadly through summaries and open Q&A sessions—Ubuntu applied post-evidence.

Conclusion: Ubuntu After Evidence

A/B testing is a learning engine. To keep it running, you must minimize friction at the point of decision and maximize inclusion at the point of reflection. Ubuntu teaches us that we rise together; testing teaches us that we rise faster when we let data lead. So don’t abandon Ubuntu—sequence it. Empower a small team to move quickly within clear guardrails. Then invite the broader community to interpret results, celebrate wins, and turn lessons into shared progress. Fewer people at the decision-making table means more learning for everyone.

Similar posts I've written about online testing

Getting an online testing program off the ground
Building Momentum in Online testing
How many of your tests win?

First times of 2025

 Continuing the slowly-growing thread of 'things I did for the first time this year', here's my list for 2025.

1.  I stopped trying to studiously write a blog article (or publish one) every month.  I discovered halfway through the year that the search engines weren't even finding my articles, and hence I was getting no traffic to them.  Instead of just launching new articles out into space each month, I started work on improving the links between the articles (all the Star Trek reviews now contain links to each other; so do the Star Wars; most of the analytics, the maths and so on).  I also posted backlinks to my site from other relevant sites, and shared more of my articles on LinkedIn.  Has traffic gone up?  Not really!  It's a slow process, and I haven't been diligent or studious in my approach.

2. I fixed a broken top-down toilet seat.  One of those integrated ones where there is zero access to the underneath of the seat.  It's all done top-down, and it took a YouTube video to make it happen.  And it was a success.

3.  After last year's success laying down loft boards for our smaller loft, I increased the coverage of loft boards in our main loft from about 50% to more like 70%.  It's not complete, as I am checking if there are any risks in boarding over the recessed lighting we have in our bathroom - I don't want anything to overheat :-)  This has given me plenty of room for my slowly expanding model railway, which now has two complete loops and series of sidings - with more planned.

4.  I learned to ask for help.

In March, I decided it was time to set up my airband radio with an external aerial.  There's zero reception inside the house, especially with the laptops, PCs, tablets, and other electronic toys all producing noise across the radio band.  So I set up a small outside aerial - I bought it off ebay and stuck it on our gatepost.  It produced a slight improvement in reception, but it was clearly time to get something better.  So I asked my dad about an outdoor aerial, and bought the brackets and so on for it.  We rigged it up, about three metres off the ground, at a slightly off-vertical angle (not deliberate).  My next door neighbour offered to set up something a little more meaningful, and, in exchange for some of his favourite lager, rigged up the same aerial about five metres off the ground.  I can now hear some of the radio traffic from aircraft on their way towards Heathrow.

We also revamped some of our garden this year, including getting rid of some old railway sleepers we'd used to landscape a raised flower bed.  They were wet, rotten, and sagging.  And still very heavy, and very dense.  We hired a skip, but it wasn't quite big enough to hold the sleepers, so I set about them with a hand-saw, a small electric saw and a lot of hard work.  My other neighbour turned up with his electric chainsaw... even he had some difficulty getting through the dense wood, but he persevered and trimmed the sleepers down to size.  I have a small offcut from the most resistant sleeper sitting on my desk, as a reminder to me to ask for help.

5. I planted some conkers.  They haven't even taken root - in fact they're more rot than root, but I'm not too concerned.  Next autumn, I'll read up on the best way to plant conkers and try again (for example, if a conker floats, it's not going to germinate at all.  Who knew?).

6.  During the autumn, I went to the cinema on three consecutive nights.  Twice, nearly.  The local Cineworld showed the Tobey Maguire Spiderman trilogy over a weekend in September, and the Christopher Nolan Batman films on three consecutive nights in October.  We didn't see Spiderman 2 - it was Lizzie's idea and she had something else on the Saturday night - but we saw all the other films.  I haven't been to the cinema quite as much as this year, and certainly not to see films I'd already seen before!

7. I started using AI to generate music. I like it.

8. I took one of our cats to the out-of-hours emergency vets. Tango suffered a broken leg back in April (we think he was hit by a car), and we had to take him for an immediate check before taking him to our normal vets the following day. We now have pet insurance.

9. I attended an outdoor Remembrance Sunday event. In the past, I've gone to church, but this time I went with Ben (who's joined the Army Cadets this year) for the parade through Hanley. It was freezing.

It looks like a short list, but it doesn't include the things I'm doing on a larger scale (my team at work now covers nearly 20 people, which is quite a jump from 10), or the things I've started but not yet finished, but it's been an interesting year, nonetheless.  

The Lists of Firsts

A first time for everything: 2018
2019 in reflection
First times in 2021 list
First times of 2022
First times in 2023
Things I did for the first time in 2024

Plus-Plus Hexel Two-Dimensional Tessellations

Plus-Plus, according to the Plus-Plus website, allows you to "Unleash the power of your imagination with a unique design in a single shape.  Plus-Plus offers unlimited opportunities for creativity. From a single repeated element, the possibilities are endless as you bring your ideas to life in both 2D and 3D forms."

Plus-Plus (or sometimes just PlusPlus) is a toy made up of small plastic pieces similar to jigsaw puzzle pieces, which are flexible enough to interlock in either two or three dimensions, and strong enough to stand up and make small towers or other 3D models.

My interest in Plus-Plus is purely geometrical.  I was most interested to discover if it's possible to produce a repeating pattern of PlusPlus pieces (which I think the manufacturers refer to as 'hexels') in a 2D surface with perfect tesselation?  

As an aside, I guess they call them hexels because they approximate a hexagonal shape?  I looked at them and decided they were made up of nine mini-squares all combined together, but a nonel sounds daft.  Either way, they are shaped like two plus symbols that have been overlapped ++

After some logical puzzling with a large pile of hexels, I was able to confirm that it's possible to make a tessellating pattern with them.  I don't know if anybody else has named the pattern, so I'm going to go with HV, for Horizontal Vertical.  Each hexel is connected to another in a straight line, with the orientation for the adjacent hexel being rotated 90 degrees (in the same plane) to the first.  The repeating pattern is Horizontal, Vertical, Horizontal, Vertical, and so on.  The picture shows how it looks in a line, and then connects multiple lines to cover a 2D surface.

Interestingly, HV is not the only way to connect a series of hexels: they can be connected HHH, with a slight diagonal offset (green lines below), or connected VVV (orange lines below).  I want to mention at this point that the two are not the same:  it's possible to have multiple HH lines tessellate by lining them up together (there's no interlocking between adjacent lines, so the structure is physically weak, but it still tessellates). 

However, HH and VV cannot tessellate together.  The two patterns are incompatible.  Rotating a HH line will not make it fit into a VV line. Another way of thinking of it (through rotation) is that one line steps 'upwards' while another steps 'downwards' - see below.

Something I found particularly interesting is that it's possible to combine HH with VH, and also to combine VV with VH.  In the diagram below, the grey hexels are in VH configuration.  The white hexels are in HH, with the lines of HH running from top left to bottom right (it's a short line, and the interface between the patterns runs perpendicular to it). The blue hexels fit into both patterns, and show the interface or the border between the two.

And here is proof that you can combine VH, HH and VV:  the grey is VH as before; the white is HH, while the orange is VV.  The blues show the interface between VH and HH; the yellow shows the interface between VV and VH.  The bands of white, grey and orange can be of any fixed width, but they don't get narrower or wider.

This reminded me of the studying I did at Cambridge University, looking at phases of iron-carbon alloys.  Each 'grain' of a particular alloy would have a certain area (or volume, in 3D) where it was made entirely of a particular phase of iron/carbon, and the composition would be consistent throughout the grain.  There would be a grain boundary where it touched another grain of the same composition, but where the grain had been oriented differently.

Diagram from http://www.gowelding.com/met/carbon.htm 

I have not yet considered 3D shapes - these are clearly possible, and I shall be looking at if it's possible to produce regular polyhedra such as cubes and cuboids, and what limitations there are.

Other Geometry Articles I've Written:

Calculating the tetrahedral bond angle
The angle of elevation of a geostationary satellite

Port Vale 0 Arsenal 2 Match Report

Port Vale vs Arsenal, 24 September 2025

There are only two teams in the football league which aren’t named after the places where they’re located; and yesterday they met for only the second time in 30 years.  Port Vale (Stoke on Trent) and Arsenal (London) are currently 61 league places apart, and to be fair it showed: Arsenal just weren’t as good as they should have been on paper.  The wage bill for one of Arsenal’s players is more than the entire Port Vale team’s wages; after last night’s performance, I wonder if Arteta is getting his value for money?

It was a disappointing performance from the London side, who seemed to have trouble doing anything with their overwhelming levels of possession.  After grabbing an early goal, they struggled to do anything positive or meaningful with the ball, despite the professional encouragement from their fans, and the consistent support of the officials.  The Arsenal goalkeeper, Kepa Arrizabalaga, passed a pre-game fitness check, which was more than could be said for the linesman monitoring the Port Vale defensive line.  He continued to show signs of an ongoing shoulder injury, which prevented him from lifting his flag anywhere above horizontal for the entire first half, while the Arsenal forwards frequently found themselves receiving the ball with nobody but the goalkeeper to beat.  You think I’m kidding?  There were zero Arsenal offsides in the whole game.

The linesmen, whose shoulder shows signs of improving

The referee too seemed in awe of the Premiership team’s visit to Vale Park, admiring their ‘strength on the ball’ and ‘technique’ which frequently left the Vale players getting a close up of the turf as they were ‘tackled’ off the ball.  Strangely, this was not a symmetrical arrangement; whenever an Arsenal player was dispossessed, this was seen as a sign of rough treatment and was typically identified as an illegal challenge. 

The referee reminds the Port Vale strikers to be kind to the Arsenal goalkeeper

He's not offside, ok?

One thing I must confess, though, is that Arsenal didn’t employ cynical and overly defensive strategies after obtaining their opening goal.  They kept moving the ball with good technique – they didn’t anything particularly productive with it (they achieved over four times the number of passes that Vale did), and most of their second-half corners were passed all the way back to the halfway line – and wore Vale down with efficiency and energy.  They saved their timewasting for more subtle tactics, and one of the most egregious was with the substitutions.  I’m not expert at football, but watching one of the Arsenal players (and an England player too) dawdle his way off the pitch when he was substituted was more gamesmanship than sportsmanship.  Maybe he wasn’t looking forward to the long drive home?  Maybe he just wanted to stay and play a bit more?

It's a long walk to the touchline.  So long, that even I could get my camera out, focus it, and take a photo.

He wasn’t the only one in no hurry to leave the field of play, as other substitutions took longer than was probably fair. There were delays taking throw-ins, there were in-team discussions about whose turn it was to take this particular corner, and maybe you’d like to try it this time?

Here's an Arsenal corner.  Coming soon.

Port Vale, for their part, showed considerable effort but also looked to be in awe of their visitors.  During the second half, the Vale front line got a hold of the ball a couple of times – at one point in a very promising position facing goal on the edge of the penalty area, only to flounder at the last minute.  In fact, the data shows that Vale didn’t even achieve a single shot on target.  It was going to be one of those nights.

While being a significant disappointment for the Arsenal, who only scraped an early goal and a late one, the game was entertaining for sure.  One of the funniest parts of the match came in the second half, when, after a Vale substitution, Arsenal were expected to restart play with a throw-in.  There was a breakdown in communication between the referee and the Arsenal player: the referee pointed persistently to where the throw-in should be taken (approximately 5 metres ahead of the halfway line, on the Arsenal left), while the Arsenal player was standing around 10-15 metres further forward of that point.  There then followed a confused discussion between the Arsenal player, trying to take the throw in, and the referee, vehemently pointing 10 metres further back.  This happened in front of where we were sitting, and with the crowd around us (did I mention this was virtually a sell-out?) we did our best to point out the miscommunication.

Taking a throw-in.  It's supposed to be from where the ball went out.  Who knew?

Even though miscommunication is frequent in football matches, and it was noted among the fans.  Arsenal brought almost 3,000 fans to Vale Park, and they stood, sang and shouted with a high degree of organization and professionalism.  There was genuinely no unpleasantness between the two sets of fans, none of the jeering or rude gesturing I have observed at other grounds, and everybody got on with shouting for their team.  At least I think that’s what we were doing – in some cases I struggled to turn the chanted syllables into phrases, or even specific words, and on a couple of occasions I managed it, then regretted it.  Football fans can certainly employ some colourful metaphors.

Speaking of organization and professionalism: the Arsenal players certainly showed this, at a completely different level to the Vale players.  At one point in the first half, Vale gained possession (legally and everything), and in order to hold possession, passed it back from the midfield to the defenders, where it was carefully passed along the line.  But not for very long: with alarming efficiency, the Arsenal players deployed a 10-man press, with the defenders moving up to the halfway line and the forward players squeezing possession.  Vale almost crumbled in the face of this threat, and did well to keep the ball away from their goal:  Arsenal in possession were interesting; Arsenal chasing possession were terrifying.

The size of Arsenal’s squad was clear to see, with players at the match wearing numbers like 41, 49 and 56.  This was probably the Arsenal B-team.  I hope so, for Arteta's sake.  On the other hand, the Port Vale shirts didn't even show a sponsor.

Football is an 11-a-side sport, with shirts numbered 1-56.

The stats tell the story fairly well:  Arsenal dominated all the main numbers, and have to be disappointed with the output from their efforts.  A lucky early goal, and one at the end made it for them. The game was billed as a David vs Goliath clash—except David forgot his slingshot and Goliath turned up wearing Crocs.  Arsenal, sitting proudly at 2nd in the Premier League, took on Port Vale, languishing at 19th in League One, a full 61 league places below. The result? A narrow and nervy 2–0 win for the North London giants. Yes, really.  They say that you can only beat the team in front of you, and that’s all that Arsenal managed, when a much more impressive scoreline was expected.  Sad times for all.

Possession
Arsenal 81%
Port Vale 19%

Passes
Arsenal 789  (731 completed, 93%)
Port Vale 183 (115 completed, 63%)

Shots
Arsenal 11  (7 inside box , 4 outside)
Port Vale 3 (2 inside box, 1 outside)

Shots on Target
Arsenal 4
Port Vale 0

Corners
Arsenal 6
Port Vale 1

Offsides
Arsenal 0
Port Vale 2

Did You Just Code a Distraction?

In the fast-paced world of web development and digital marketing, creating new features to enhance user experience is not just a common practice, it's pretty much par for the course.  Everybody has ideas about making the website better, and it usually involves some sort of magical feature that will help users find the exact product they want in a mere matter of seconds - a shopping genie, or something involving AI.   The new feature for your website is almost always visually appealing, interactive, and the designers are confident it will boost user engagement with their favourite persona. Before rolling it out, you wisely decide to run an A/B test to measure its effectiveness.

So you code the test, and you run it for the usual length of time.  You follow all the advice on LinkedIn about statistical significance (we can all describe it and we all have our own ways of calculating it, thank you) and getting a decent sample size. The test results are in, and they’re a mixed bag. On one hand, the new feature is a hit in terms of engagement. It receives twice as many clicks compared to the other clickable elements on the page, such as those lovely banners, the promotional links and the pretty pictures. However, your  deeper dive into the data reveals a concerning trend. While the new feature attracts a lot of attention and engagement, the conversion rate for users who interact with it is only around 2.5%. In contrast, the conversion rate for users who engage with the existing content on the page is significantly higher, at around 4.1%. 

This is key.  It really is insufficient to look only at engagement data (click through rate) as a success metric.  Yes, it is important, but it is not enough.  After all, if you want to create a banner with a high click rate, then you could simply write "Buy one get one free", or better still, "Buy one get two free - click here."  It's essential that you set expectations with your banners, calls to action and features - what's to stop you from writing "Click here for free money!"?  If your priority in testing is to generate clicks, then you'll degenerate into coding the on-site versions of clickbait, and that's a terrible waste of a potential lead.

So, what went wrong with your test? The short answer is that you coded a pretty distraction. Here’s a breakdown of why this happens and how to address it:

Misalignment with User Intent

The new feature, despite being engaging, may not align with the primary intent of your users. If it diverts their attention away from the main conversion paths, it can reduce overall effectiveness. Users might be intrigued by the new feature but not find it relevant to their immediate needs.  You misunderstood your persona's motivation; it might be time to write your persona with this additional information.

Cognitive Load

Introducing a new element can increase the cognitive load on users. They have to process and understand this new feature, which can be mentally taxing. If the feature doesn’t provide immediate value or clarity, users might get distracted and abandon their original task.  They used up their time, effort and patience while interacting with your new feature, and gave up on their primary purpose (which was to buy something from you).

Disruption of User Flow

A well-designed website guides users smoothly towards conversion goals. A new feature that stands out too much can disrupt this flow, causing users to deviate from their intended path. This disruption can lead to lower conversion rates, as users get sidetracked.  How do I get back to my intended path?  This new feature has proved to be the next big shiny thing, and while it's attracting user engagement, it's confusing them and preventing them from getting to where they wanted to.  

The Solutions

To avoid coding distractions, consider the following strategies:

User-Centric Design

Not my favourite phrase, since it leads to design without A/B testing and designers designing for their favourite personas.  Ensure that any new feature is designed with the user’s needs and goals in mind. Conduct user research to understand what your audience values and how they navigate your site, and then align your new features and your development roadmap with these insights.  This will enhance, rather than disrupt, the user experience, and reduce the amount time of wasted on development the next shiny bauble - it looks nice and impresses senior management, but is not good for users.  

Incremental Testing

Instead of launching a fully-fledged feature, start with a minimal viable version and test its impact incrementally. This approach allows you to gather feedback and make necessary adjustments before a full rollout.  Use test data in conjunction with user research to gain a full picture of what you thought was going to happen, and what actually happened.

Clear Value Proposition

Make sure the new feature has a clear and compelling value proposition. Users should immediately understand its purpose and how it benefits them. This clarity can help integrate the feature seamlessly into the user journey.  If the test 'fails', then you'll learn that the value proposition you promoted was not what users wanted to read, and you can try something different.

Monitor and Iterate

Continuously monitor the performance of new features and be ready to iterate based on user feedback and data. If a feature is not performing as expected, don’t hesitate to tweak or even remove it to maintain a smooth user experience.  It's time to swallow your pride and start again.  If you change direction now, you'll have less distance to travel than if you wait six months before unimplementing the eye-catching blunder you've launched.

Conclusion

In the quest to innovate and improve user engagement, it’s crucial to strike a balance between novelty and functionality. While new features can attract attention, they must also support the primary goals of your website. By focusing on user-centric design, incremental testing, and clear value propositions, you can avoid coding distractions and create features that truly enhance the user experience.

Other Web Analytics and Testing Articles I've Written

How not to segment test data (even if your stakeholders ask you to)

Designing Personas for Design Prototypes
Web Analytics - Gathering Requirements from Stakeholders
Analysis is Easy, Interpretation Less So
Telling a Story with Web Analytics Data
Reporting, Analysing, Testing and Forecasting
Pages with Zero Traffic

Do You Know How Well Your Test Will Perform?

 There are various ways of running tests - or more specifically, there are various ways of generating test hypotheses.  One that I've come across over the years, and increasingly so more recently, is the 'guess how well your test is going to perform' approach.  It's not called that, but it seems to me to be the most succinct description.  Apparently, we should already have a target improvement in mind before we even start the test.

"If we change the pictures on our site from cats to dogs, then we'll see a 3.5% increase in conversion."
"If we promote construction toys ahead of action figures, then we'll see a 4% lift in revenue."

If you know that's going to happen, why don't you do it anyway?

The main underlying challenge I have is that it's almost impossible to quantify the improvement you're going to get.  How do you know?

Well, let's attempt the calculation (with hypothetical numbers all the way through).

Let's say our latest campaign landing page has a bounce rate (user lands on page, then exits without visiting any other pages) of 75%.  10% engage with site search, 10% click on the menus at the top of the page, and 5% click on the content on the page (there are a few banners and a few links).

We've identified that most users aren't scrolling past the first set of banners and links, and we therefore hypothesise that if we make the banners smaller, and reduce the amount of padding around the links, that we can increase engagement with the content in the lower half of the page, and therefore improve the bounce rate.  We believe we can get 50% more links above the fold, and therefore increase the in-page engagement rate from 5% to 7.5%.  We will assume (and this is the fun bit) that this additional traffic converts at the same rate as the 5% we have so far, and therefore, we'll get a revenue lift of 50%.  This sounds like a lot, but given that the engagement rate is going up from a small number to a slightly larger number, it's unlikely to be a huge revenue lift in dollar terms (unless you're pouring in huge volumes of traffic - and watching it bounce at a rate of 75%).

Perhaps that was an over-simplification.  But if we knew that our test will give us a 5% lift (and we've still decided to test it), what happens when we launch the test?  Presumably, we'll stop it when it reaches the 5% lift, irrespective of the confidence level.  But what happens if it doesn't get to 5%?  What if it stubbornly sits at 4%?  Or maybe just 3%?  Did the test win, or did it lose?  In classical scientific terms, it lost, since we disproved our overly-specific hypothesis.  But from a business perspective, it still won, just not by as much as we had originally expected.  Would you go into a meeting with the marketing manager and say, "Sorry, Jim, our test only achieved a 3% revenue lift, so we've decided it was a failure."?

For me, it comes down to two arguments: 

If you can forecast your test result with a high degree of certainty, based on considerable evidence for your hypothesis, it's probably not worth testing and you should implement already.  Testing is best used for edge-cases with some degree of uncertainty. 

If, on the other hand, you have identified a customer problem with your site, and you can see that fixing it will give you a revenue lift - but you don't know how to fix it - then that's very good grounds for testing.  The hypothesis is not, "If we fix this problem, we'll get a 6% revenue lift," but, "If we fix this problem in this way then we'll get a revenue lift".  And that's where you need to encourage the website analysts and the customer feedback department (or the complaints department, or whoever advocates for customers within your company) to come together and find out where the problems are, and what they are, and how to address them.

That will undoubtedly bring good test ideas, and that's what you're looking for, even if you don't know how much revenue lift it will provide.

Other Web Analytics and Testing Articles I've Written

How not to segment test data (when your stakeholders want you to adapt your data)
Web Analytics - Gathering Requirements from Stakeholders
Analysis is Easy, Interpretation Less So (and why it's more valuable)
Telling a Story with Web Analytics Data
Reporting, Analysing, Testing and Forecasting
Pages with Zero Traffic