Ten thousand ponies! or How to build a dashboard

So usually you identify One Key Metric to measure success. Or maybe more. And those metrics are probably the right ones for your product/service – pageviews, signups, app downloads, dollars, ponies.

The thing is, when everything’s right and you’re growing like gangbusters, nothing really matters. You just stand in front of that giant screen that measures the One Metric in real-time, and down shots with every milestone (ten thousand ponies!).

But then you’re not doing so well. And so then you switch to detective mode and dive into your logs to make sense of what’s happening – or what’s not.

It’s going to be hard to do that every time something goes wrong. And yes, things will go wrong more than once. Because once your baby has launched, you’re going to obsess over every little negative point and trend, however fleeting.

You don’t need an event log. You need a dashboard. Something in addition to your single-point metric. That’ll help you find why that One Metric isn’t doing so well [1].

Three general principles that will likely help:

1. Model and track your ideal customer flow. Create a state diagram of how you want your customers to navigate through your app or your website. Now track state transitions, not state populations.

For our paid subscription service with a free trial, our One Metric was payment transactions. When that slowed, we just didn’t know why. We just knew a lot of people were signing up and now weren’t paying enough. Why didn’t they like the service?

Then we created the following state diagram [2].



Which then told us that of the people who stated the free trial, too few people were paying us (x%) and too many were letting their trials expire (y%). It wasn’t that they disliked the service as they just didn’t *do* anything [3].

2. Track over time. Track both your One Metric and customer flow every day (or minute?). When you tinker with something, you’ll see how it affects your customer behaviour. Today, and a week from now. Note: you can get carried away with tinkering this way.

3. Export, don’t just display. Week-on-week on-screen charts are great, but have both the OneMetric and customer flow snapshots exportable as CSV. Being able to find correlations between data sets will help your cause/effect analyses no end. Chimps can do pattern recognition too. But they can’t rock a vlookup.

So then. To a million ponies.


[1] Look, chances are you’ll muck around with your log file or raw data exports anyway, but your dashboard’s going to tell you exactly where to look.

[2] Very briefly: the states ending in ‘f’ were free trial states, the others were paid states; a customer began in SSF (service started, free) and either paid (SSP or service started, paid), or actively stopped during the trial (USF or unsubscribed, free), or had their free trial expire (PPF or payment pending, free), and so on.

[3] We first thought they just didn’t care (terrible!). When we surveyed a sample, we found that communication was a problem – we were doing a terrible job at telling them just how they could pay.