Learning Points
When I joined Mixbook 4 years ago, we were running very large, coarse experiments, where we’d spend 3-6 months with 2-4 people building some new feature until we thought it was done, then we’d ship. These days, we’re a bit leaner, with the same sized teams typically shipping a new experiment within 2-4 weeks. But we don’t want to stop there. Lately we’ve been talking about how to get it down to 1-2 days per experiment, so we can ship 5 or more per sprint.
Towards that end I’ve dusted off a tool I created a couple years ago when one my team’s was struggling to reach their growth targets. They were building expensive new features, seeing what would stick, then trying something else, but the features were so costly to make, that they weren’t learning very quickly at all. With a unprofitable win to loss ratio, it didn’t make sense to keep funding the project–we needed to get the ratio up.
So we started tracking a new metric: learning points.
Learning Points
Learning Points represent progress towards understanding how to dramatically grow the business. Projects earn learning points as follows:
+0 Learning Point
- started without a hypothesis on file
- OR impact is not measured and/or is not measurable
+1 Learning Point
- started with a measurable hypothesis
- AND the results were measured, including both benefit and cost
- AND the hypothesis was proved or disproved
- AND we gained a non-trivial insight
- AND the hypothesis and results were recorded for easy reference
+1 Learning Point
- started with franchise KPIs clearly defined
- AND a KPI improved with 95% confidence and 10k+ observations
+1 Learning Point
- got the LP for a KPI improvement
- AND there’s strong evidence to support the same pattern can be repeated 3 or more times more, achieving similar gains
Note that the size of the KPI improvement (or degradation) does not affect the LP value. This is intentional–bigger improvements in KPIs may be worth more in the short run, but they don’t necessarily teach more, especially if they take a long time to complete, i.e. they have a high LP/SP ratio. This scoring system is designed to reward learning velocity and ’lean’-er projects that get us the most learning value at the lowest possible cost in effort.
Similar to how non-backlog work is considered ‘waste’, as it does not increase story point velocity on a Scrum team, work that does not produce learning point velocity is considered waste on a Lean team and should be scrutinized and targeted for elimination.
Back To Our Story
After we started using Learning Points, we shifted from a gut based, ‘we like this idea more’ approach to prioritizing the backlog, to one that asked the questions:
- what are we trying to learn?
- why are we trying to learn that?
- how much upside is there to learning that, i.e. does it scale?
- how do we avoid an inconclusive result?
- how can we learn that same thing, but cheaper?
We got better and better at answering those questions, our learning velocity went up, and our ROI went up. We never achieved the growth we needed to to keep going with that project, but we learned a lot about learning faster and we’ve applied many of the ideas to other projects and other teams, which is how we’ve managed to get leaner and learner. Perhaps the time has come again to measure our learning velocity and see if it will help us push our learning efficiency further.