It is not enough to simply create a numeric measure. The measure should accurately reflect the process. We use metrics to base decisions on and to focus our actions. It is not only important to measure the right indicators, it is important to measure them well.
To be effective and reliable, the metrics we choose to use need to have ten key characteristics. The following table was adapted from Keebler (1999) which suggest the qualitites to look for in indicators.
A good measure: | Description: |
Is quantitative | The measure can be expressed as an objective value |
Is easy to understand | The measure conveys at a glance what it is measuring, and how it is derived |
Encourages appropriate behavior | The measure is balanced to reward productive behavior and discourage “game playing” |
Is visible | The effects of the measure are readily apparent to all involved in the process being measured |
Is defined and mutually understood | The measure has been defined by and/or agreed to by all key process participants (internally and externally) |
Encompasses both outputs and inputs | The measure integrates factors from all aspects of the process measured |
Measures only what is important | The measure focuses on a key performance indicator that is of real value to managing the process |
Is multidimensional | The measure is properly balanced between utilization, productivity, and performance, and shows the trade-offs |
Uses economies of effort | The benefits of the measure outweigh the costs of collection and analysis |
Facilitates trust | The measure validates the participation among the various parties |
Choosing the right metrics is critical to success, but the road to good metrics is fraught with pitfalls. As your endeavors to become more metrics-driven, beware of errors in the design and use of metrics.
Common mistakes include:
- Metrics for the sake of metrics (not aligned)
- Too many metrics (no action)
- Metrics not driving the intended action
- Lack of follow up
- No record of methodology
- No benchmark
- Underestimation of the data extraction
Tim,
ReplyDeleteGreat reminder of the value of good metrics, and how to create them. Having spent what is now 14 years (egads!) in project planning & analysis, there's always those who use measurement to show what they want, not what is really happening. Although that solves a short-term problem, in the future, when looking at past data to determine future plans, using those bogus metrics only leads to terrible plans - which results in increasing variances and more and more need to "massage" the numbers.
The process itself is a waste (rework, defects, overproduction) and the interpretations of the resulting data only lead to other wastes within the organizations, too.
David, I believe that measures should be minimal. We need to apply lean thinking to the measures we use to capture the performance of our processes. The minimum set of measures possible to provide feedback on the system is desirable (and the goal).
DeleteThank you for your good posts of a major subject. I am a process improvment guy in a manufacturing facility (now 23-1/2 years and counting) which rolled out "lean" back in 2008. We have improved in many areas and continue to drive for more improvement which leads to the increasing need for value-added accurate metrics. I've found it's difficult to measure performance related to 5S even though it's obvious that we have a safer work environment and we are more efficent on the shop floor. It would be fabulous to be able to capture some actual numbers for this activity. It's not difficult in our Energy and Waste efforts, VSM efforts or our TPM efforts. Any suggestions?
ReplyDeleteThanks,
Bill
Bill, 5S like many of the Lean tools and techniques are sometimes difficult to measure the full impact of the improvement. This is in part because of the intangible benefits that can come in the form of people development. Anyway, like any measure I like to start with the benefits of the initiative and then work out how we will see that improvement. If you think you will see 5S measures in the P&L I think you will be looking for a while. I would argue overtime you will see that improvement though.
DeleteMy friend Jon Miller wrote 2 good posts on this subject you should review:
http://www.gembapantarei.com/2010/01/whats_the_roi_of_a_5s_program.html
http://www.gembapantarei.com/2010/04/a_tracking_method_for_5s_programs.html
This is probably a good topic for a post. Stay tuned for an update.
Thank you Tim. I'll review Jon's posts. Also, although we can completely prove it here at our company it's interesting to note that we have the lowest number of safety related lost work days since starting 5S in 2008 and we are working less overtime while shipping record numbers of product with a higher on-time average. All good stuff.
ReplyDeleteI really liked your 10 Characteristics of a Good Measure but you didn't define what you meant by "a measure" and I am wondering whether you would define it the same as I do. To me a measure is the charcteristic(s) by which performance will be judged or whaat we look for to see if our objective has been achieved. Thus a process might have, repeatability, safety, resource utlization, capacity etc as its measures. But to these one needs to choose units of measure and a method of measurement. I also think its how these are expressed that is important. e.g. in a hospital they often have waiting time as a measure but express in such a way that the administrators can cheat by permitting them to effect when a patient is put on the list and taken off.
ReplyDeleteDavid you make a good point.
DeleteA measure is a number or quantity that records a directly observable value or performance.
All measures have a unit attached to them.
In contrast, an indicator is an indirect measure or a predictor (such as a leading Indicator) of performance.
I prefer raw data, sent to cloud, which can later be re-analyzed in light of new interest, or events.
ReplyDeleteThe Flaw of Averages is good book to read before you start using stat methods of any kind.
I think you left off one crucial characteristic: it has to be readily and consistently measurable. In other words, if you measure the same thing several times by different people you should get the same answer each time. Lack of this characteristic is sometimes the downfall of measurement systems.
ReplyDeleteDennis, that is a good characteristic. I agree with you. Many places collect data but never bother to a) understand if it is any good and b)even analyze it. Consistency and readily measurable are important.
DeleteGreat discussion on a topic near and dear to my heart. I often use as an example of "overproduction" (one of the lean wastes) the collection of data that are never used or analyzed.
ReplyDelete