Data vs Judgement:

There are two ways to measure any particular performance – via data or through judgement of someone who knows the job or work being measured.

In the recent past, as it has become easier to collect data, the preference in all kinds of settings has been moving towards collecting a lot of data and as being the primary way to measure performance.

What we need to understand is that measured data is almost never an alternative to judgement. In fact, measurement demands judgement – judgement about whether to measure, what to measure, how to evaluate the significance of what is being measured and decide if rewards and penalties need be attached to the results. Lastly, judgement about who should have access to the measured data.

Guidelines on measurements:  

At the end of the book, Jerry does share a checklist on when and how to create performance metrics and what we need to keep in mind while analyzing these metrics.

Below are some guidelines based on his checklist added with my perspectives.

What do you plan to measure:

The kind of information you plan to measure has an important impact on how you design your metrics. The more the object to be measured resembles inanimate matter, the more likely it is measurable (natural sciences or engineering).

When the object being measured are influenced by the process of measurement, then the measurement becomes less reliable. Even more unreliable if the measurement is about human activity.

People are conscious and will respond to the process of being measured and if rewards and punishments are involved, they will react in a way to skew the validity of the measurements.

What will the measures be used for:

Metrics used by practitioners to track performance to compare themselves with their peers and identify potential areas of improving performances are more useful.

Metrics that will be used by external parties (includes senior management, government or any other external party) who might not recognize the limitations of the metrics and use it to dish out rewards or punishments leads to behavior that will lead to gaming the system and are more harmful in the mid-to-long term.

Offering recognition to those who excel and assistance to those who are behind is good.

KPI’s that help accentuate the internal motivations of the people being measured so that they can continue to improve tend to be better than if they are designed to get people to behave in ways that they are not intrinsically motivated or appeal to extrinsic motivations.

Low stakes metrics are almost always better than high stakes metrics.

If the performance being measured does not require or has the potential for any intrinsic rewards, pay for performance might work really well.

How useful is the measurement:

Just because something is measurable doesn’t mean that it is worth measuring. In some cases, the ease of measurement could be inversely proportional to significance of what is being measured.

Ask yourself – “Is what you are measuring a proxy for what you really want to know?”. If the answer is no for both (usefulness or a proxy for what you really want to know”, you are better off not measuring it at all.

How useful are more metrics

Measured performance, when useful are more effective in finding outliers, especially poor performers or true misconduct. They might not be very useful in distinguishing people in the middle or on the top of the ladder.

The more you measure, the marginal cost of measuring will become more than the marginal improvements through measurement. So, the fact that metrics are helpful doesn’t mean that more metrics are more helpful.

In fact, it is easier, simpler and faster to use less metrics, but one’s that is relevant to drive behavior and continue to improve performance. One can change the KPI depending on the maturity of performance instead of having the same KPI to measure the performance of everyone (irrespective of how mature the person or process doing the activity is).  

Costs of Measurement:

There is always a cost of acquiring the data for the metrics. Every moment that we devote to production of metrics, it is time not spent on doing what is being measured. So, before we initiate a new metric, we need to identify the costs of measuring.

We can also look if there are other sources of information about performance that can be used instead, based on the judgement and experience of customers or someone else.

Who develops the KPI’s and what they mean:

How and by whom are the measures of performance developed. Measurements are more likely to be beneficial if developed bottom up with inputs from the people who will be measured (experience and judgement).

KPI’s defined by people who don’t have the context in which the performance being measured typically tend to be the one’s that are gamed or manipulated as they might neither add value nor increase the difficulty of performing the activity.

In conclusion:

KPI’s are effective to the extent that the people who are being measured believe in its worth and effectiveness. We need to remember that even the best KPI’s are subject to gaming and corruption. Keep an eye of the result of what is being measured.  

Sometimes, recognizing the limits of the possible is the beginning of wisdom. Not all problems are soluble and even fewer are soluble by metrics. It’s not true that everything can be improved by measurement and whatever can be measured can be improved.

We need to exercise judgement, which comes from experience, before introducing any new metrics. Metrics should inform judgement and vice versa.

As Jerry concludes in his book, sometimes the best use of metrics is not to use them at all. 

PS: You can watch Jerry talking about his book “The Tyranny of Metrics”