Always tinker

I learnt recently that even when all indications are that your business (or life. or city. or whatever) is running fine, something could be wrong – in plain view – that those indicators can’t tell you.

The only way to discover these problems is to tinker around with data. Ask questions of it.

Here’s what happened.

Ours is a prepaid subscriptions business. You sign up, put money into your account and pick your subscriptions.

Our signups, payments and new subscriptions – the three primary indicators of our health [1] – were growing at expected rates relative to each other. Nothing seemed to be wrong.

Until one day, I discovered that many new signups didn’t have any subscriptions. That was unexpected. It meant that most of our new subscriptions were via older subscribers.

That meant – and this was quickly confirmed – that most of our payments were also made by older subscribers. This is a problem, and we commissioned a quick survey to find out what was wrong with our new signups.

But then we tinkered further. We plotted a histogram of (normalized) new subscriptions started (all of this is excluding renewals) versus how long ago the subscribers had signed up, and we found this:


Click for a larger image - bet you can't read the tiny text


Astonishing. The older the subscriber, the more the number of new subscriptions they started recently [2]. We had a larger problem than we expected; our older subscribers were so active, they’d hidden how un-engaged our newer subscribers were for several months.

While we took immediate steps to fix this, we also realized that it’s hard to build a dashboard for stuff like this. You can – and should – track primary measures of success, results of specific campaigns, and suchlike. But under-the-surface stuff like this – we’d never have figured it out if we hadn’t tinkered with data.

[1] There’s also ARPU and churn, but they aren’t material to this discussion.

[2] The data for months 8 and 9 is skewed by a small set of people with a lot of subscriptions each, but they’re still much higher than any other month, and the trend is the same

(Cross posted from the MyToday blog.)

One Reply to “Always tinker”

  1. Love the subject line. Always Tinker. It succinctly summarizes the most important lesson of my phd.

Comments are closed.