The Case for Metrics

"If you want to improve something, measure it. If you really want to improve something, measure it often."

I don't know who said this first, but it is 100% true. So, why don't more people do this more often?

• It's hard to determine what the right metrics are for a given issue.
• It's tedious to calculate the values of the metrics, especially if you have to do it manually and often.
• The metrics that come to mind don't really capture exactly what you have in mind.
• The metrics might get used against you.

What are the right metrics?

For example, if you want to make software changes that will improve client satisfaction, how will you measure client satisfaction? How will you take into the account the fact that different clients have different levels of skill in their organizations? How do you take into the account the fact that client satisfaction, number of support calls, and other measures vary so much from month to month even when you aren't trying to improve anything?

The answer to these valid questions is that almost any reasonable metric is better than nothing. And, two reasonable metrics are better than one, and may be as good as one better one.

When starting to measure an important business barometer, start simple. Accept the fact that you may never get the perfect metric for customer satisfaction with your product, but that there are some really good proxies for that metric that will tell you over the long haul whether you're getting the result you hoped for.

By way of example, let's say you are trying to make the installation process better. You're about to release a new version of the software and you want to know if it accomplishes this goal. What do you do?

1. Choose a couple of metrics. In this case, you can use the number of support calls received during installation and the number of support minutes (or hours or days) required to get a client up and running. You could also choose to use client satisfaction survey results or even a subjective rating by your installer (if one is involved). There is nothing wrong with deciding which metrics you will use based on the degree of difficulty of getting the data - select the most convenient ones.
2. Start measuring these metrics before you implement the enhancement. The longer before the better, but anyting is better than nothing.
3. Implement the change and watch the metrics. If possible, measure them more frequently than you had been so that you can get "early returns".
4. Give the new process some time to take hold. Respond to emergencies, but try to give the new process a reasonable period of time before you make more changes so that you can get some good data.
5. Review the results, determine what can be further improved, and repeat from step 3.

When you've gotten all the improvement you need or all that you can afford to chase, it's time to move on to another opportunity.

How can I make it less tedious?

This is an easy one. Simply choose more available measures. Try not to invent something for the purpose of measuring unless it's absolutlely necessary. Remember, metrics are a proxy for the real thing and are correlated with it. It should not be necessary to develop the best possible metrics.

How can I avoid having the metrics from being mis-used?

You can't. You can get out in front by drawing the right conclusions first. Be honest when something isn't working, and then figure out how to fix it. Get rightful credit for identifying problems and then fixing them.

Conclusion

Metrics are the best way to quantify a problem and evaluate its solution. By making the discussion on a problem quantitative, you are making it objective and less personal. When reasonable people have access to the same data, they more often reach consensus than when they limit their inputs to anecdotes and subjective theories.