Always tinker

I learnt recently that even when all indications are that your business (or life. or city. or whatever) is running fine, something could be wrong – in plain view – that those indicators can’t tell you.

The only way to discover these problems is to tinker around with data. Ask questions of it.

Here’s what happened.

Ours is a prepaid subscriptions business. You sign up, put money into your account and pick your subscriptions.

Our signups, payments and new subscriptions – the three primary indicators of our health [1] – were growing at expected rates relative to each other. Nothing seemed to be wrong.

Until one day, I discovered that many new signups didn’t have any subscriptions. That was unexpected. It meant that most of our new subscriptions were via older subscribers.

That meant – and this was quickly confirmed – that most of our payments were also made by older subscribers. This is a problem, and we commissioned a quick survey to find out what was wrong with our new signups.

But then we tinkered further. We plotted a histogram of (normalized) new subscriptions started (all of this is excluding renewals) versus how long ago the subscribers had signed up, and we found this:

 

 

Astonishing. The older the subscriber, the more the number of new subscriptions they started recently [2]. We had a larger problem than we expected; our older subscribers were so active, they’d hidden how un-engaged our newer subscribers were for several months.

While we took immediate steps to fix this, we also realized that it’s hard to build a dashboard for stuff like this. You can – and should – track primary measures of success, results of specific campaigns, and suchlike. But under-the-surface stuff like this – we’d never have figured it out if we hadn’t tinkered with data.

—
[1] There’s also ARPU and churn, but they aren’t material to this discussion.

[2] The data for months 8 and 9 is skewed by a small set of people with a lot of subscriptions each, but they’re still much higher than any other month, and the trend is the same

(Cross posted from the MyToday blog.)