g
ame, g
a
me. repea
t
.

When I joined Mixbook 4 years ago, we were running very large, coarse experiments, where we’d spend 3-6 months with 2-4 people building some new feature until we thought it was done, then we’d ship. These days, we’re a bit leaner, with the same sized teams typically shipping a new experiment within 2-4 weeks. But we don’t want to stop there. Lately we’ve been talking about how to get it down to 1-2 days per experiment, so we can ship 5 or more per sprint.

Towards that end I’ve dusted off a tool I created a couple years ago when one my team’s was struggling to reach their growth targets. They were building expensive new features, seeing what would stick, then trying something else, but the features were so costly to make, that they weren’t learning very quickly at all. With a unprofitable win to loss ratio, it didn’t make sense to keep funding the project–we needed to get the ratio up.

So we started tracking a new metric: learning points.

Learning Points

Learning Points represent progress towards understanding how to dramatically grow the business. Projects earn learning points as follows:

+0 Learning Point

+1 Learning Point

+1 Learning Point

+1 Learning Point

Note that the size of the KPI improvement (or degradation) does not affect the LP value. This is intentional–bigger improvements in KPIs may be worth more in the short run, but they don’t necessarily teach more, especially if they take a long time to complete, i.e. they have a high LP/SP ratio. This scoring system is designed to reward learning velocity and ‘lean’-er projects that get us the most learning value at the lowest possible cost in effort.

Similar to how non-backlog work is considered ‘waste’, as it does not increase story point velocity on a Scrum team, work that does not produce learning point velocity is considered waste on a Lean team and should be scrutinized and targeted for elimination.

Back To Our Story

After we started using Learning Points, we shifted from a gut based, ‘we like this idea more’ approach to prioritizing the backlog, to one that asked the questions:

We got better and better at answering those questions, our learning velocity went up, and our ROI went up. We never achieved the growth we needed to to keep going with that project, but we learned a lot about learning faster and we’ve applied many of the ideas to other projects and other teams, which is how we’ve managed to get leaner and learner. Perhaps the time has come again to measure our learning velocity and see if it will help us push our learning efficiency further.