A Metric Is Vanity Only Against Your Own Outcome Data

# A Metric Is Vanity Only Against Your Own Outcome Data Whether a metric is "vanity" is a property of that metric in *your* dataset, not a universal label — only regression against your own outcome data tells you whether it predicts anything, so you never dismiss a metric you haven't tested. ## Scope & conditions - Holds when you have a defined outcome and enough data to regress against it. Penn's reframe: a vanity metric is a "non-zero" metric — zero means you've failed, but growth above zero proves nothing about outcomes until you've tested the correlation in your own numbers. The label travels around social media; the evidence doesn't. - The failure it guards against is statistical illiteracy dressed as sophistication — parroting "X is a vanity metric" without the regression that would actually earn the label for your context. - Breaks (or needs a proxy) when no outcome target exists — purely non-commercial or community goals where you must first define the "outcome" before anything can be regressed against it. ## Links - [[DevRel MOC]] — applies locally validated outcome measures to DevRel program evaluation. - [[Product & Startup Instincts MOC]] — connects metric interpretation to product decisions grounded in actual outcomes. - [[3. Resources/Permanent Notes/Instincts Are Reliable Ideas Are Not So Test Small Bets on Proven Foundations]] — applies-in: annual retention supplies the outcome data that distinguishes a promising small bet from an attractive but unproven idea. - [[Unveiling the Impact of CHAOSS DEI Metrics in Open Source]] — applies-in: CHAOSS metrics become decision-worthy when a project relates them to its own recruitment, retention, safety, and participation outcomes. - [[The Four Objectives Framework Replaces Vanity Metrics in DevRel]] — because extends; the framework maps every metric to Awareness/Activation/Engagement/Retention — this note adds the test that decides whether a given metric actually predicts its objective in your data, rather than assuming the mapping holds - [[Rewarding AI Usage Manufactures Usage Not Value]] — because applies-in; a usage metric looks like a KPI but Goodharts into vanity once it's gamed — only regression against the real outcome catches the gap between measured usage and value - [[A System Failure That Looks Like a Performance Failure Never Gets Fixed]] — because contrasts; there a real signal gets misread as a people problem, here a metric gets mislabeled vanity — both are measurement-attribution errors fixed by checking against outcomes, not reputation - [[Making Data-Driven Product Decisions]] — evidence-for; Ashauer's independent-platform checks, order exports, and customer conversations show how to test a metric against outcomes before acting - [[Scaling Developer Content Production at Snyk]] — applies-in; Snyk's sessions-per-content-type estimates become decision-grade only when tested against the team's growth outcome - [[3. Resources/Literature Notes/The Importance of Customer Obsession and Data-Driven Insights in DevRel at Vercel]] — applies-in: Vercel's segmented customer and product data needs an explicit outcome test before it can guide DevRel decisions - [[The Complexity of Measuring Value in Knowledge Work]] — evidence-for; knowledge work's intangible output is the broader reason proxy metrics need validation against real outcomes. - [[The Principle of Least Resistance in Knowledge Work]] — evidence-for: weak feedback about outcomes lets ease, rather than impact, determine which work gets selected. - [[The Shift from Clarity to Ambiguity in Knowledge Work]] — evidence-for: ambiguous output is the condition that makes unvalidated proxy metrics especially tempting. - [[3. Resources/Literature Notes/Purpose of the 2023 Developer Relations Report]] — applies-in: respondent geography supplies the outcome context needed to interpret compensation data without false generalization. - [[3. Resources/Permanent Notes/Search Visibility and Business Value Need Separate Metrics]] — applies-in: AI-search exposure is a candidate leading indicator, while qualified leads and revenue supply the outcome test. - [[3. Resources/Permanent Notes/Builders Partners and Customers Need Separate Funnels Because They Make Different Decisions]] — constrained by: one blended engagement number hides three decisions (Morrow). - [[3. Resources/Permanent Notes/A Metric Nobody Else Acts On Is a Diary]] — contrasts: a DevRel metric earns its place when someone outside DevRel changes a decision because of it (Gorton). - [[3. Resources/Permanent Notes/A Coding Agent Mentioning Your Tool Does Not Mean It Installs Your Tool]] — applied in: mention rate is top of funnel; install rate is the outcome (Silverman and Barnes). - [[3. Resources/MOCs/AI Cost & Tokenomics MOC]] — applies-in: the map pairs this note with the pre-AI-baseline note, on the point that no tracked baseline means no attributable return. - [[3. Resources/MOCs/Book Review Synthesis MOC]] — applies-in: listed in this synthesis map; read it as the outcome-data test to apply to any metric a reviewed book recommends. - [[3. Resources/Permanent Notes/A Code of Conduct File Being Present and Being Maintained Are Separate Facts]] — constrained by: a code of conduct's edit history is checkable, but no edit pattern was found to predict community health, so an untouched file is a question and not a verdict. ## JTBD source trail - [[4. Archives/Book reviews/The Jobs to Be Done Playbook]] — interview insight forms hypotheses; representative evidence is needed before treating desired outcomes as market prevalence. - [[4. Archives/Book reviews/Jobs to Be Done — Theory to Practice]] — an opportunity score inherits the validity of its outcome statement, sample, and decision rule. - [[4. Archives/Book reviews/The Jobs-to-be-Done Pyramid]] — opportunity ratings need an explicit method and outcome baseline before they can guide strategy.