The first discussion huddle I led at the Digital Analytics Hub in 2014 looked at why average order value changes in checkout tests, and was an interesting discussion. With such a specific title, it was not surprising that we wandered around the wider topics of checkout testing and online optimisation, and we covered a range of issues, tips, troubles and pitfalls of online testing.
But first: the original question - why does average order value (AOV) change during a checkout test? After all, users have completed their purchase selection, they've added all their desired items to the cart, and they're now going through the process of paying for their order. Assuming we aren't offering upsells at this late stage, and we aren't encouraging users to continue shopping, or offering discounts, then we are only looking at whether users complete their purchase or not. Surely any effect on order value should be just noise?
For example, if we change the wording for a call to action from 'Continue' to 'Proceed' or 'Go to payment details', then would we really expect average order value to go up or down? Perhaps not. But, in the light of checkout test results that show AOV differences, we need to revisit our assumptions.
After all, it's an oversimplification to say that all users are affected equally, irrespective of how much they're intending to spend. More analysis is needed to look at conversion by basket value (cart value) to see how our test recipe has affected different users based on their cart value. If conversion is affected equally across all price bands, then we won't see a change in AOV. However, how likely is that?
Other alternatives: perhaps there's no real pattern in conversion changes: low-price-band, mid-price-band, high-price-band and ultra-high-price-band users show a mix of increases and decreases. Any overall AOV change is just noise, and the statistical significance of the change is low.
But let's suppose that the higher price-band users don't like the test recipe, and for whatever reason, they decide to abandon. The AOV for the test recipe will go down - the spread of orders for the test recipe is skewed to the lower price bands. Why could this be? We discussed various test scenarios:
- maybe the test recipe missed a security logo? Maybe the security logo was moved to make way for a new design addition - a call to action, or a CTA for online chat - a small change but one that has had significant consequences.
- maybe the test recipe was too pushy, and users with high ticket items felt unnecessarily pressured or rushed? Maybe we made the checkout process feel like express checkout, and we inadvertantly moved users to the final page too quickly. For low-ticket items, this isn't a problem - users want to move through with minimum fuss and feel as if they're making rapid progress. Conversely, users who are spending a larger amount want to feel reassured by a steady checkout process which allows the user to take time on each page without feeling rushed?
- sometimes we deliberately look to influence average order value - to get users to spend more, add another item to their order (perhaps it's batteries, or a bag, or the matching ear-rings, or a warranty). No surprises there then, that average order value is influenced; sometimes it may go down, because users felt we were being too pushy.
Here's how those changes might look as conversion rates per price band, with four different scenarios:
Scenario 1: Conversion (vertical axis) is improved uniformly across all price bands (low - very high), so we see a conversion lift and average order value is unchanged.
Scenario 2: Conversion is decreased uniformly across all price bands; we see a conversion drop with no change in order value.
Scenario 3: Conversion is decreased for low and medium price bands, but improved for high and very-high price bands. Assuming equal order volumes in the baseline, this means that conversion is flat (the average is unchanged) but average order value goes up.
Scenario 4: Conversion is improved selectively for the lowest price band, but decreases for the higher price bands. Again, assuming there are similar order volumes (in the baseline) for each price band, this means that conversion is flat, but that average order value goes down.
There are various combinations that show conversion up/down with AOV up/down, but this is the mathematical and logical reason for the change.
Explaining why this has happened, on the other hand, is a whole different story! :-)
But first: the original question - why does average order value (AOV) change during a checkout test? After all, users have completed their purchase selection, they've added all their desired items to the cart, and they're now going through the process of paying for their order. Assuming we aren't offering upsells at this late stage, and we aren't encouraging users to continue shopping, or offering discounts, then we are only looking at whether users complete their purchase or not. Surely any effect on order value should be just noise?
For example, if we change the wording for a call to action from 'Continue' to 'Proceed' or 'Go to payment details', then would we really expect average order value to go up or down? Perhaps not. But, in the light of checkout test results that show AOV differences, we need to revisit our assumptions.
After all, it's an oversimplification to say that all users are affected equally, irrespective of how much they're intending to spend. More analysis is needed to look at conversion by basket value (cart value) to see how our test recipe has affected different users based on their cart value. If conversion is affected equally across all price bands, then we won't see a change in AOV. However, how likely is that?
Other alternatives: perhaps there's no real pattern in conversion changes: low-price-band, mid-price-band, high-price-band and ultra-high-price-band users show a mix of increases and decreases. Any overall AOV change is just noise, and the statistical significance of the change is low.
But let's suppose that the higher price-band users don't like the test recipe, and for whatever reason, they decide to abandon. The AOV for the test recipe will go down - the spread of orders for the test recipe is skewed to the lower price bands. Why could this be? We discussed various test scenarios:
- maybe the test recipe missed a security logo? Maybe the security logo was moved to make way for a new design addition - a call to action, or a CTA for online chat - a small change but one that has had significant consequences.
- maybe the test recipe was too pushy, and users with high ticket items felt unnecessarily pressured or rushed? Maybe we made the checkout process feel like express checkout, and we inadvertantly moved users to the final page too quickly. For low-ticket items, this isn't a problem - users want to move through with minimum fuss and feel as if they're making rapid progress. Conversely, users who are spending a larger amount want to feel reassured by a steady checkout process which allows the user to take time on each page without feeling rushed?
- sometimes we deliberately look to influence average order value - to get users to spend more, add another item to their order (perhaps it's batteries, or a bag, or the matching ear-rings, or a warranty). No surprises there then, that average order value is influenced; sometimes it may go down, because users felt we were being too pushy.
Here's how those changes might look as conversion rates per price band, with four different scenarios:
Scenario 2: Conversion is decreased uniformly across all price bands; we see a conversion drop with no change in order value.
Scenario 3: Conversion is decreased for low and medium price bands, but improved for high and very-high price bands. Assuming equal order volumes in the baseline, this means that conversion is flat (the average is unchanged) but average order value goes up.
Scenario 4: Conversion is improved selectively for the lowest price band, but decreases for the higher price bands. Again, assuming there are similar order volumes (in the baseline) for each price band, this means that conversion is flat, but that average order value goes down.
There are various combinations that show conversion up/down with AOV up/down, but this is the mathematical and logical reason for the change.
Explaining why this has happened, on the other hand, is a whole different story! :-)
