How not to Segment Test Data

 Segmenting Test Data Intelligently

Sometimes, a simple 'did it win?' will provide your testing stakeholders with the answer they need. Yes, conversion was up by 5% and we sold more products than usual, so the test recipe was clearly the winner.  However, I have noticed that this simple summary is rarely enough to draw a test analysis to a close.  There are questions about 'did more people click on the new feature?' and 'did we see better performance from people who saw the new banner?'.  There are questions about pathing ('why did more people go to the search bar instead of going to checkout?') and there are questions about these users.  Then we can also provide all the in-built data segments from the testing tool itself.  Whichever tool you use, I am confident it will have new vs return users; users by geographic region; users by traffic source; by landing page; by search term... any way of segmenting your normal website traffic data can be unleashed onto your test data and fill up those slides with pie charts and tables.

After all, segmentation is key, right?  All those out-of-the-box segments are there in the tool because they're useful and can provide insight.

Well, I would argue that while they can provide more analysis, I'm not sure about more insights (as I wrote several years ago).  And I strongly suspect that the out-of-the-box segments are there because they were easy to define and apply back when website analytics was new.  Nowadays, they're there because they've always been there,  and because managers who were there at the dawn of the World Wide Web have come to know and love them (even if they're useless.  The metrics, not the managers).

Does it really help to know that users who came to your site from Bing performed better in Recipe B versus Recipe A?  Well, it might - if the traffic profile during the test run was typical for your site.  If it is, then go ahead and target Recipe B for users who came from Bing.  And please ask your data why the traffic from Bing so clearly preferred Recipe B (don't just leave it at that).

Visitors from Bing performed better in Recipe B?  So what?

Is it useful to know that return users performed better in Recipe C compared to Recipe A?

Not if most of your users make a purchase on their first visit:  they browse the comparison sites, the expert review sites and they even look on eBay, and then they come to your site and buy on their first visit.  So what if Recipe C was better for return users?  Most of your users purchase on their first visit, and what you're seeing is a long-tail effect with a law of diminishing returns.  And don't let the argument that 'All new users become return users eventually' sway you.  Some new users just don't come back - they give up and don't try again.  In a competitive marketplace where speed, efficiency and ease-of-use are now basic requirements instead of luxuries, if your site doesn't work on the first visit, then very few users will come back - they'll find somewhere easier instead.  

And, and, and:  if return users perform better, then why?  Is it because they've had to adjust to your new and unwieldy design?  Did they give up on their first visit, but decide to persevere with it and come back for more punishment because the offer was better and worth the extra effort?  This is hardly a compelling argument for implementing Recipe C.  (Alternatively, if you operate a subscription model, and your whole website is designed and built for regular return visitors, you might be on to something).  It depends on the size of the segments.  If a tiny fraction of your traffic performed better, then that's not really helpful.  If a large section of your traffic - a consistent, steady source of traffic - performed better, then that's worth looking at.

So - how do we segment the data intelligently?

It comes back to those questions that our stakeholders ask us: "How many people clicked?" and "What happened to the people who clicked, and those who didn't?"  These are the questions that are rarely answered with out-of-the-box segments.  "Show me what happened to the people who clicked and those who didn't" leads to answers like, "We should make this feature more visible because people who clicked it converted at a 5% higher rate." You might get the answer that, "This feature gained a very high click rate, but made no impact [or had a negative effect] on conversion." This isn't a feature: it's a distraction, or worse, a roadblock.

The best result is, "People who clicked on this feature spent 10% more than those who didn't."

And - this is more challenging but also more insightful - what about people who SAW the new feature, but didn't click?  We get so hung up on measuring clicks (because clicks are the currency of online commerce) that we forget that people don't read with their mouse button.  Just because somebody didn't click on the message doesn't mean they didn't see it: they saw it and thought, "Not interesting," "not relevant" or "Okay, that's good to know but I don't need to learn more".  The message that says, "10% off with coupon code SAVETEN - Click here for more" doesn't NEED to be clicked.  And ask yourself "Why?" - why are they clicking, why aren't they?  Does your message convey sufficient information without further clicking, or is it just a headline that introduces further important content.  People will rarely click Terms and Conditions links, after all, but they will have seen the link.

We forget that people don't read with their mouse button.

So we're going to need to have a better understanding of impressions (views) - and not just at a page level, but at an element level.  Yes, we all love to have our messages, features and widgets at the top of the page, in what my high school Maths teacher called "Flashing Red Ink".  However, we also have to understand that it may have to be below the fold, and there, we will need to get a better measure of how many people actually scrolled far enough to see the message - and then determine performance for those people.  Fortunately, there's an abundance of tools that do this; unfortunately, we may have to do some extra work to get our numerators and denominators to align.  Clicks may be currency, but they don't pay the bills.

So:  segmentation - yes.  Lazy segmentation - no.

Other articles I've written on Website Analytics that you may find relevant:

Web Analytics - Gathering Requirements from Stakeholders

Analysis is Easy, Interpretation Less So - when to segment, and how.
Telling a Story with Web Analytics Data - how to explain your data in a clear way
Reporting, Analysing, Testing and Forecasting - the differences, and how to do them well
Pages with Zero Traffic - identifying which pages you're wasting effort on.

A Quick Checklist for Good Data Visualisation

One thing I've observed during the recent pandemic is that people are now much more interested in data visualisation.  Line graphs (or equivalent bar charts) have become commonplace and are being scrutinised by people who haven't looked at them since they were at school.  We're seeing heatmaps more frequently, and tables of data are being shared more often than usual.  This was prevalent during the pandemic, and people have generally retained their interest in data presentation (although they wouldn't call it that).

This made me consider:  as data analysts and website optimisers, are we doing our best to convey our data as accurately and clearly as possible in order to make our insights actionable.  We want to share information in a way that is easy to understand and easy to base decisions on, and there are some simple ways to do this (even with 'simple' data), even without glamorous new visualisation techniques.

Here's the shortlist of data visualisation rules

- Tables of data should be presented consistently either vertically or horizontally, don't mix them up
- Graphs should be either vertical bars or horizontal bars; be consistent
- If you're transferring from vertical to horizontal, then make sure that top-to-bottom matches left-to-right
- If you use colour, use it consistently and intuitively.

For example, let's consider the basic table of data:  here's one from a sporting context:  the English Premiership's Teams in Form:  results from a series of six games.

Pos	Team	P	Pts	F	A	GD	Sequence
1	Liverpool	6	16	13	2	11	W W W W W D
2	Tottenham	6	15	10	4	6	W L W W W W
3	West Ham	6	14	17	7	10	D W W W W D

The actual data itself isn't important (unless you're a Liverpool fan), but the layout is what I'm looking at here.  Let's look at the raw data layout:

Pos	Category	Metric 1	Metric 2	Metric 3	Metric 4	Derived metric	Sequence
1	Liverpool	6	16	13	2	11	W W W W W D
2	Tottenham	6	15	10	4	6	W L W W W W
3	West Ham	6	14	17	7	10	D W W W W D

The derived metric "GD" is Goal Difference, the total For minus the total Against (e.g. 13-2=11).

Here, the categories are in a column, sorted by rank, and different metrics are arranged in subsequent columns - it's standard for a league table to be shown like this, and we grasp it intuitively.  Here's an example from the US, for comparison:

Player	Pass Yds	Yds/Att	Att	Cmp	Cmp %	TD	INT	Rate	1st	1st%	20+
Deshaun Watson	4823	8.9	544	382	0.702	33	7	112.4	221	0.406	69
Patrick Mahomes	4740	8.1	588	390	0.663	38	6	108.2	238	0.405	67
Tom Brady	4633	7.6	610	401	0.657	40	12	102.2	233	0.382	63

You have to understand American Football to grasp all the nuances of the data, but the principle is the same.   For example, Yds/Att is yards per attempt, which is Pass Yds divided by Att.  Columns of metrics, ranked vertically - in this case, by player.

A real life example of good data visualisation

Here's another example; this is taken from Next Green Car comparison tools:

The first thing you notice is that the categories are arranged in the top row, and the metrics are listed in the first column, because here we're comparing data instead of ranking them.  The actual website is worth a look; it compares dozens of car performance metrics in a page that scrolls on and on.  It's vertical.

When comparing data, it helps to arrange the categories like this, with the metrics in a vertical list - for a start, we're able to 'scroll' in our minds better vertically than horizontally (most books are in a portrait layout, rather than landscape).

The challenge (or the cognitive challenges) come when we ask our readers to compare data in long rows, instead of columns... and it gets more challenging if we start mixing the two layouts within the same document/presentation.  In fact, challenging isn't the word. The word is confusing.

The same applies for bar charts - we generally learn to draw and interpret vertical bars in graphs, and then to do the same for horizontal bars.

Either is fine. A mixture is confusing, especially if the sequence of categories is reversed as well. We read left-to-right and top-to-bottom, and a mixture here is going to be misunderstood almost immediately, and irreversibly.

For example, this table of data (from above)

Pos	Category	Metric 1	Metric 2	Metric 3	Metric 4	Derived metric	Sequence
1	Liverpool	6	16	13	2	11	W W W W W D
2	Tottenham	6	15	10	4	6	W L W W W W
3	West Ham	6	14	17	7	10	D W W W W D

Should not be graphed like this, where the horizontal data has been converted to a vertical layout:

And it should certainly not be graphed like this:  yes, the data is arranged in rows and that's remained consistent, but the sequence has been reversed!  For some strange reason, this is the default layout in Excel, and it's difficult to fix.

The best way to present the tabular data in a graphical form - i.e. putting the graph into a table - is to match the layout and the sequence.

And keep this consistent across all the data points on all the slides in your presentation.  You don't want your audience performing mental gymnastics to make sense of your data.  It would be like reading a book, then having to turn the page by 90 degrees after a few pages, then going back again on the next page, then turning it the other way after a few more pages.  

You want your audience to spend their mental power analysing and considering how to take action on your insights, and not to spend it trying to read your data.

Other articles with a data theme

Web analytics - a data emergency
Too Big Data? 
The science of a good hypothesis

It's Not Zero!

 I started this blog many years ago.  It pre-dates at least two of my children, and possibly all three - back in the days when I had time to spare, time to write and time to think of interesting topics to write about.  Nowadays, it's a very different story, and I discovered that my last blog post was back in June.  I used to aim for one blog article per month, so that's two full months with no digital output here (I have another blog and a YouTube channel, and they keep me busy too).

I remember those first few months, though, trying to generate some traffic for the blog (and for another one I've started more recently, and which has seen a traffic jump in the last few days).  

Was my tracking code working?  Was I going to be able to see which pages were getting any traffic, and where they were coming from?  What was the search term (yes, this goes back to those wonderful days when Google would actually tell you your visitors' search keywords)?

I had weeks and weeks of zero traffic, except for me checking my pages.  Then I discovered my first genuine user - who wasn't me - actually visiting my website.  Yes, it was a hard-coded HTML website and I had dutifully copied and pasted my tag code into each page...  did it work?  Yes, and I could prove it:  traffic wasn't zero.

So, if you're in the point (and some people are) of building out a blog, website or other online presence - or if you can remember the days when you did - remember the day that traffic wasn't zero.  We all implemented the tag code at some point; or sent the first marketing email, and it's always a moment of relief when that traffic starts to appear.

Small beginnings:  this is the session graph for the first ten months of 2010, for this blog.  It's not filtered, and it suggests that I was visiting it occasionally to check that posts had uploaded correctly!  Sometimes, it's okay to celebrate that something isn't zero any more.

And, although you didn't ask, here's the same period January-October 2020, which quietly proves that my traffic increases (through September) when I don't write new articles.  Who knew?

How long should I run my test for?

 A question I've been facing more frequently recently is "How long can you run this test for?", and its close neighbour "Could you have run it for longer?"

Different testing programs have different requirements:  in fact, different tests have different requirements.  The test flight of the helicopter Ingenuity on Mars lasted 39.1 seconds, straight up and down.  The Wright Brothers' first flight lasted 12 seconds, and covered 120 feet.  Which was the more informative test?  Which should have run longer?

There are various ideas around testing, but the main principle is this:  test for long enough to get enough data to prove or disprove your hypothesis.  If your hypothesis is weak, you may never get enough data.  If you're looking for a straightforward winner/loser, then make sure you understand the concept of confidence and significance.

What is enough data?  It could be 100 orders.  It could be clicks on a banner : the first test recipe to reach 100 clicks - or 1,000, or 10,000 - is the winner (assuming it has a large enough lead over the other recipes). 

An important limitation to consider is this:  what happens if your test recipe is losing?  Losing money; losing leads; losing quotes; losing video views.  Can you keep running a test just to get enough data to show why it's losing?  Testing suddenly becomes an expensive business, when each extra day is costing you revenue.   One of the key advantages of testing over 'launch it and see' is the ability to switch the test off if it loses; how much of that advantage do you want to give up just to get more data on your test recipe?

Maybe your test recipe started badly.  After all, many do:  the change of experience from the normal site design to your new, all-improved, management-funded, executive-endorsed design is going to come as a shock to your loyal customers, and it's no surprise when your test recipe takes a nose-dive in performance for a few days.  Or weeks.  But how long can you give your design before you have to admit that it's not just the shock of the new design, (sometimes called 'confidence sickness') but that there are aspects of the new design that need to be changed before it will reach parity with your current site?  A week?  Two weeks?  A month?  Looking at data over time will help here.  How was performance in week 1?  Week 2?  Week 3?  It's possible for a test to recover, but if the initial drop was severe, then you may never recover the overall picture, but if you can find that the fourth week was actually flat (for new and return visitors) then you've found the point where users have adjusted to your new design.

If, however, the weekly gaps are widening, or staying the same, then it's time to pack up and call it a day.

Let's not forget that you probably have other tests in your pipeline which are waiting for the traffic that you're using on your test.  How long can they wait until launch?

So, how long should you run your test for?  As long as possible to get the data you need, and maybe longer if you can, unless it's
- suffering from confidence sickness (keep it running)
- losing badly, and consistently (unless you're prepared to pay for your test data)
- losing and holding up your testing pipeline

Similar posts I've written about online testing

Getting an online testing program off the ground
Building Momentum in Online testing
How many of your tests win?

Wright Brothers Picture:

"Released to Public: Wilber and Orville Wright with Flyer II at Huffman Prairie, 1904 (NASA GPN-2002-000126)" by pingnews.com is marked with CC PDM 1.0

Analysis versus Interpretation

We have had a disappointingly mild winter.

It snowed on two days...

You will easily notice the bias in that sentence. Friends and long-time readers will know that I love snow, for many reasons. The data from the Meteorological Office puts the winter (1 December - 29 February) into context, using a technique that I've mentioned before - ranking the specific period against the rest of the data set.

So, by any measure, it was a wet and mild winter. Far more rain than usual (across the country), and temperatures were above average.

This was posted on Facebook, a website renowned for its lack of intelligent and considered discussion, and known for the sharp-shooting debates.  Was it really wetter than usual? Is global warming to blame? Is this an upward trend (there is insufficient data here) or a fluke?

And then there's the series of distraction questions - how long have records been held? Have the temperature and rainfall data been recorded since the same original date? Is any of that relevant? No.

In my experience, analysis is hard, but anybody, it seems, can carry out the interpretation.  However, interpretation is wide open to personal basis, and the real skill is in treating the data impartially and without bias, and interpreting it from that viewpoint. It requires additional data research - for example, is February's data an anomaly or is it a trend? Time to go and look in the archive and support your interpretation with more data.

The General Election (Inferences from Quantitative Data)

The Election

The UK has just had a general election: all the government representatives who sit in the House of Commons have all been selected by regional votes.  The UK is split into 650 areas, called constituencies, each of which has an elected Member of Parliament (MP). Each MP has been elected by voting in their constituency, and the candidate with the highest number of votes represents that constituency in the House of Commons.


There are two main political parties in the UK - the Conservative party (pursuing centre-right capitalist policies, and represented by a blue colour), and the Labour party (which pursues more socialist policies, and represented by as red colour).  I'll skip the political history, and move directly to the data:  the Conservative party achieved 318 MPs in the election; the Labour party achieved 262; the rest were spread between smaller parties. With 650 MPs in total, the Conservative party did not achieve a majority and have had to reach out to one of the smaller parties to reach the majority they require to obtain a working majority.

Anyway:  as the results for most of the constituencies had been announced, the news reporters started their job of interviewing famous politicians of the past and present.  They asked questions about what this meant for each political party; what this said about the political feeling in the country and so on.

And the Conservative politicians put a brave face on the loss of so many seats.  And the Labour politicians contained their delight at gaining so many seats and preventing a Conservative majority.

The pressing issue of the day is Brexit (the UK's departure from the European Union).  Some politicians said, "This tells us that the electorate don't want a 'hard' Brexit [i.e. to cut all ties completely with the EU], and that they want a softer approach." - views that they held personally, and which they thought they could infer from the election result.  Others said, "This shows a vote against austerity,"; "This vote shows dissatisfaction with immigration." and so on.

The problem is:  the question on election day is not, "Which of these policies do you like/dislike?" The question is, "Which of these people do you want to represent you in government?"   Anything beyond that is guesswork and supposition - whether that's educated, informed, biased, or speculative.

Website Data
There's a danger in reading too much into quantitative data, and especially bringing your own bias (intentionally or unintentionally) to bear on it.  Imagine on a website that 50% of people who reach your checkout don't complete their purchase.  Can you say why?

- They found out how much you charge for shipping, and balked at it.
- They discovered that you do a three-for-two deal and went back to find another item, which they found much later (or not at all)
- They got called away from their computer and didn't get chance to complete the purchase
- Their mobile phone battery ran out
- They had trouble entering their credit card number

You can view the data, you can look at the other pages they viewed during their visit.  You can even look at the items they had in their basket.  You may be able to write hypotheses about why visitors left, but you can't say for sure.  If you can design a test to study these questions, you may be able to improve your website's performance.  For example, can you devise a way to show visitors your shipping costs before they reach checkout?  Can you provide more contextual links to special offers such as three-for-two deals to make it easier for users to spend more money with you?  Is your credit card validation working correctly?  No amount of quantitative data will truly give you qualitative answers.

A word of warning:  it doesn't always work out as you'd expect.

The UK, in its national referendum in June 2016, voted to leave the EU.  The count was taken for each constituency, and then total number of votes was counted; the overall result was that "leave" won by 52% to 48%.  

However, this varied by region, and the highest leave percentage was in Stoke-on-Trent Central, where 69% of voters opted to leave.  This was identified by the United Kingdom Independence Party (UKIP) and their leader, Paul Nuttall, took the opportunity to stand as a candidate for election as an MP in the Stoke-on-Trent Central constituency in February 2017.  His working hypothesis was (I assume) that voters who wanted to leave the EU would also vote for him and his party, which puts forward policies such as zero-immigration, reduced or no funding for overseas aid, and so on - very UK-centric policies that you might imagine would be consistent with people who want to leave a multi-national group.  However, his hypothesis was disproved when the election results came in:

Labour Party - 7853
UKIP (Paul Nuttall) - 5233
Conservative Party - 5154
Liberal Democrat Party - 2083

He repeated his attempt in a different constituency in the General Election in June; he took 3,308 votes in Boston and Skegness - more than 10,000 fewer votes than the party's result in 2015.  Shortly afterwards, he stood down as the leader of UKIP.

So, beware: inferring too much from quantitative data - especially if you have a personal bias - can leave you high and dry, in politics and in website analysis.

Ten Things I Learned In Fantasy Football

This year, for the second year, I joined my workplace Fantasy NFL Football league, even though I'm nowhere near my 'workplace'.  I work from home in the UK, and most of my colleagues are based in Texas, so I don't get much chance to engage with them outside of a work environment - so I seized this opportunity.  Some of my colleagues asked me if I knew that this was American football (some of them with more sarcasm than others), but they were all very welcoming.  And I can assure you that I know enough about football (I'm going to call it football instead of American Football - it's just quicker to type) to understand the rules of the game, the aims of the game and the basic stats (yards, passes, interceptions and so on).  We use the Yahoo fantasy football scoring system (points per 100 yards, typically, with extra points for touchdowns) - which I soon got to grips with (and produced my own Excel spreadsheet to identify the good players, as you do).

Now, although I understand the rules, I had no idea about who the best players were, so I really did start from scratch - reviewing the previous year's data and rankings, understanding how Yahoo scores each player, and so on.  This means I had no preconceptions (also known as 'experience') about the best players or the most successful teams. They are all just names to me.  Le'Veon Bell's arrest for drugs; Cam Newton's Christian faith (and his fashion sense); Derek Carr's philanthropy... I wasn't aware of any of them.

However, here's what I learned:

1. Some Americans are extremely competitive. Not just the actual football teams and players, but my workmates - and some of them take this very, very seriously.  (I have the advantage of having nothing to lose - after all, is an Englishman supposed to know anything about the NFL?  Don't English men just drink tea and play cricket?).  I had heard about trash talk, but now I know what it means - and thankfully last season, most of it was directed between other players.  This season, there was almost none at all.  Perhaps my American colleagues just weren't trying hard enough?

From "If Brits Played American Football" YouTube video.

2.  As they say when advertising risky financial products, previous performance is not really an indicator of future performance.  It's okay to review a previous season, or even a previous game, but it's not going to give you all the answers.  It's good as an indication of a player's abilities and potential performance, but it's not comprehensive or totally reliable.  More detailed information about player form and fitness, and the strength of their opposition is also important. Fitness levels are important -more than just the "Questionable" that Yahoo listings provide:  wider reading is recommended. For example, Derek Carr (QB) scored 47 points one week... and just 7 the next.  I bet you didn't see that coming.

3.  Yahoo's own points projections are unreliable at best.  I suspect they're produced at the start of the season and not adapted or updated based on circumstances or form throughout the season, because there have been times when my players have massively outperformed them (Le'Veon Bell (RB) and Julio Jones (WR) are two examples) and yet they've not seen their projections change for the following week.

4. MVP (Most Valuable Player) can also stand for Most Variable Player.  I had Cam Newton (QB) on my team last year, and again this year.  I also drafted Derek Carr (QB), who has had a season of two halves.  There was even a week where I played Marcus Mariota (QB), (which worked out for me).

Overall, Derek Carr scored 328 points, 14% more than Cam Newton's 287.  However, Derek Carr was less consistent:  his maximum scores were 47 (week 8) and 31 (week 4), and his minimum scores were 7 (week 9) and 4 (week 14).  Yes, just 4 points.  His overall spread of results is 4 - 47, which is 43 points.  For Cam, the maximum scores are 40 and 26, the minimum scores are 12 and 13, and his spread is just 27 (compared to 43 for Derek).

So, who do you pick?  There's considerable variation in both players:  Derek scored 328, Cam scored 287, but if I'd picked the better player each week (retrospectively), their combined score is 418.  This game is not just about drafting good players, it's also about playing the best one on a week-by-week basis.

How are you supposed to forecast the performances in weeks 8 (47 points) and 9 (7 points)?

5.  I have to pick my draft selections in advance, as I'm six hours ahead of my Texan friends and the draft session is too late in the day.  This is not a significant disadvantage (nor am I complaining), but it does mean I have to choose my list all at once, without knowing which of my first picks I drafted successfully.  It's a lot like running an A/B test (and I have treated the whole Fantasy Football thing like a series of A/B tests) - you have to set up your recipe before you start running the test!

I should probably confess that in my first year, I didn't realise in NFL that you can change/transfer your players each week (it's not like soccer, where there are specific transfer windows) and hence I drafted two kickers - a lead kicker and a substitute.  I didn't make that mistake this year.

Yahoo gets all sassy with my team selections in my first season

6.  It's okay to make transfers to change your team - like I said, this is really just iterative testing with more noise than usual.  It's frowned upon (halfway through the first season, I received the "Most OCD Manager" for the most roster changes), but not against the rules.

Partway through this season, I picked up Jay Ajayi (RB) and Marcus Mariota (QB).  Marcus is the new quarterback for Tennessee - this was his second season - but a few weeks into this season, I noticed his performance based on, and drafted him and played him once.  Similarly, Jay Ajayi has really developed this season, and very quickly became my second running back - my first running back slot was taken since I discovered Le'Veon Bell last season ;-)

7.  It's okay to use the wisdom of the crowd.  There are sites which compile player rankings from multiple sites and will enable you to compare one player against another, week by week (taking into account effects like injuries, opposition, and so on).  This is extremely useful if you have two players in mind - either both players in your squad, or one that you own and one that you're considering picking up as a free agent.  My personal favourite is FantasyPros.com.  These compile the rankings from sites like Yahoo, but also take into account expert rankings which are updated and reviewed every week (unlike Yahoo, as I mentioned in paragraph 3).

8.  Le'Veon Bell (RB) is an extremely good player.  He was suspended for the first four weeks of the season (as I discovered after drafting him) but has still been one of the highest scoring running backs this year.  In week 15, he achieved 55 fantasy points, which was just over a third of my team's total for the week, and possibly the best for a RB in the whole season.
9.  Bye weeks: after a few introductory weeks, each team has a week off during the season, so you'll need strength and depth to carry your team when your best player(s) are not playing.  And it helps if you can stagger your team's bye weeks, so that you don't have a large number of players out in the same week - as I discovered last year, and then remembered too late this year.  This year, I didn't pay enough attention, and had a week where two or three of my best players were all out at the same time.  A note that bye weeks are not the same as in the English Premier League, where an International weekend means that nobody plays.

10.  It's not great when you have players in your fantasy team who are playing directly against each other in a given week.  Are both players going to have good weeks, if only one of them can score points when they have possession?  This is also important when you pick your defence - it's really not a good idea to have your quarterback play against your defence - only one of them can do really well.  And if you're spelling defence with a 'c', and stressing the second syllable instead of the first, 

11.  Yes, I'm having 11 lessons, because the article title is as accurate as a Yahoo player projection.  Lesson number 11 is that if you win, you become the 'commissioner' for the next year.  From what I can tell, this is a thankless task, where you set up all the parameters for the season (the points awarded for yards, touchdowns, field goals and so on) and how many teams make it into the playoffs.  Do it well, and nobody notices.  Do it badly (or less well), and everybody complains, especially at the end of the season when everybody claims they've won; that they scored the most points; conceded the fewest; made the most player transfers (I thought this was a bad thing, but apparently not); and won in the playoffs (which everybody, for some strange reason) was entered into.  Our commissioner this year did a great job.  That's all I'm going to say :-)
My results?  I achieved 6-7-0 for this season, making the play-offs by coming third in our league of eight, and then coming third in the play-offs  My aim was to be not-last in our league, and I exceeded my own expectations.  I even made some of my colleagues nervous by winning my first two games, and climbing towards the top spot.  My weekly points average was 128.62, with a high of 192 in week 10 (Le'Veon Bell 33, Stefon Diggs 31, Cam Newton 26, Julio Jones 25) and a dreadful low of 66 in Week 3 (Willie Snead 0, Julio Jones 2).

Next season: I'll read in advance of the start of the season to identify any suspensions or injuries, then review the best players from this year. My spreadsheet is ready!

Other articles I've written about sports, spreadsheets and data:

Checkout Conversion:  A Penalty Shootout
When should you switch off an A/B test?
The Importance of Being Earnest with your KPIs
Should Chelsea sack Jose Mourinho? (It was a relevant question at the time, and I looked at what the data said)
How Exciting is the English Premier League?  what does the data say about goals per game?