In his classic 2011 book, The Lean Startup, Eric Ries introduced the concept of Vanity metrics, presented as being in opposition to Actionable metrics. Vanity metrics are described as broad, cumulative metrics that give you a warm feeling that things are good, but they don’t really inform or indicate why. We’ve all had that experience where things are going well and we’re encouraged to “keep doing whatever it is that you are doing,” but that doesn’t help meaningfully improve your product or business.
Common characteristics of vanity metrics include:
- Lack of substance
- Overly simplistic
- Lacking nuance and context
- Can be misleading or manipulated
Chasing Green Time
As a former Missile Maintenance Officer in the United States Air Force, I can share a fitting example of a vanity metric that illustrates why they don’t often serve leaders well. In missile maintenance, our vanity metric was known informally as Green Time and formally as the Alert Rate, a measure of the time missiles are functional and on alert divided by total time in a given period and expressed as a percentage. A 50% alert rate means the missiles are operational and ready 50% of the time, and the remainder of the time they are down for repair or restorative action.
Green Time came from matrixed cells on a spreadsheet indicating time periods. For each period, if a missile is operational, the cell is green and if not, it is red. When you glanced at the spreadsheet and saw a field of green, it meant Alert Rates were high. If there was too much red, it was a problem.
Alert Rates had the characteristics of a Vanity Metric. It was the primary measure by which missile maintenance units were judged but it didn’t give any information about the how or why behind the numbers. My wing was in Missouri and our peer wings were in Montana, North Dakota, South Dakota and Wyoming. We usually led the group with the best Alert Rates. Our most confounding challenges were weather related and a major factor in the response time to address maintenance and repair issues was the ability to get to the missile sites.
Missouri had typically milder winter weather and easier treks to missile sites, especially in the winter. Additionally, from base-to-base missiles were spread out over as small of an area as 9,600 square miles to as wide of an area as 16,000 square miles, through terrain varying from flat to mountainous. Alert Rates between wings were simplistic measures that didn’t provide apples-to-apples comparisons. They lacked the nuance and context of each bases’ unique environment, but that didn’t stop wings from competing for the highest Alert Rates.
Vanity metrics can also be manipulated and be misleading. Ries refers to “theatrics” that can be used to make the numbers look good. In missile maintenance, decisions could be made to chase Green Time rather than do what’s best. If the wing set a threshold Alert Rate goal and an issue arose at a missile site, rather than address it immediately, it might be possible to put the maintenance off and keep the missile on alert. Doing so can help stay above the immediate goal threshold, but it can degrade overall long-term capability as chasing Green Time can lead to the missile maintenance equivalent of technical debt, where minor problems build up to the extent that they can eventually degrade the system.
Chasing Green Time aligns with the root behind Ries’ use of the word “vanity.” According to the Cambridge Dictionary, vanity is defined as:
“The fact that you are too interested in appearances or achievements.”
Maintenance teams would chase Green Time so at their next meeting with superior officers, they could paint a rosy picture indicating that everything looks good. The superior officers then take the news up the chain. Alert Rates gave a broad indication of how things were going but they gave no information to learn about how to improve, what might be going wrong or other underlying factors. If the spreadsheets were green, everyone was happy.
How You Can Identify Actionable Metrics
In the Lean Startup, Ries contrasts Vanity metrics with Actionable metrics. In the webinar, Optimizing Your Agile Transformation, xScion CTO Mason Chaudhry and Agile expert Reid Lowery discuss Vanity metrics versus Actionable metrics in the context of Agile transformation. Reid’s guidance for establishing Actionable metrics starts with a clear understanding of the organization’s goals so that actions are taking you in the right direction and helping achieve business objectives. To consider whether a metric is actionable, Reid poses three questions:
- What’s the business decision you can make from the metric?
- How can you intentionally reproduce the result?
- Does the data reflect the truth?
In missile maintenance, Alert Rates or Green Time didn’t answer the first two questions. “It looks good, keep doing whatever you are doing” isn’t a solid business decision and can’t intentionally reproduce the result because Alert Rates don’t indicate how they were achieved. If the rates are good, our action would be to keep doing what we’re doing and hope things worked out. While Alert Rates are broad indicators of truth, they miss important context that hinders comparison between wings or with past Alert Rates. Additionally, good Alert Rates can potentially hide the truth, that there might be ongoing neglect of small maintenance and repair issues building up to cause larger problems in the future.
For tips about optimizing your Agile Transformation and to learn about Vanity versus Actionable metrics, watch Part 2 of the Measuring Success in Your Agile Transformation webinar series, Optimizing Your Agile Transformation.
