# The 8 Software Engineering Metrics AI Broke > [!question] Lingering Questions > - Does any vendor (Jellyfish, LinearB, Pluralsight Flow) have published empirical data on AI-correlated metric inflation, or is this practitioner anecdote so far? > - Is there longitudinal research on skill/expertise atrophy under sustained AI coding assistance, or is "expertise debt" still speculative? > [!tip] Article Ideas > - A post on Goodhart's Law becoming *automatic* under AI tool use — the pre/post-AI distinction (gaming used to require intent; now it's a side effect of normal use) is a sharp, shareable hook. > - A one-page "metrics audit template" distinguishing activity vs. outcome metrics, built off Ankit Jain's five-step reframing. > [!link] Cross Reference > - [[3. Resources/Permanent Notes/AI Activity Metrics Measure Tool Throughput Not Engineering Value]] — already captures this article's core claim (activity metrics measure tool adoption, not engineering value) > - [[3. Resources/Permanent Notes/A Metric Is Vanity Only Against Your Own Outcome Data]] — the outcome-regression test this article's "3 surviving metrics" implicitly apply > - [[3. Resources/Permanent Notes/Prevented Work Is Invisible to Velocity Metrics]] — same dashboard blind spot, different angle (review labor vs. generation volume) > [!quote] Vocabulary > - **Goodhart's Law** — "when a measure becomes a target, it ceases to be a good measure"; the article's twist is that AI makes this automatic rather than requiring deliberate gaming > - **DXI / DevEx frameworks** — Developer Experience measurement frameworks tracking feedback speed, cognitive load, flow state ## Metadata - Source: [[3. Resources/Readwise/Articles/The 8 software engineering metrics AI broke]] - Author: Michael Hill, LeadDev - URL: https://leaddev.com/ai/the-8-software-engineering-metrics-ai-broke ## Author's thesis AI coding tools severed the assumption that engineering effort and output scale together, which invalidates eight widely-used proxy metrics (deployment frequency, cycle time, change failure rate, SPACE's satisfaction/activity/efficiency, lines of code, DevEx frameworks). Three metrics survive because they measure real events or outcomes rather than activity: time to recover, business/customer outcomes, and escaped defect rate. ## Key arguments (≤3) 1. **The structural assumption broke, not just the numbers.** Metrics assumed effort-as-value; AI made effort nearly free, so every activity-based proxy built on that assumption is now unreliable. 2. **Goodhart's Law went from intentional to automatic.** Pre-AI, gaming a metric took deliberate effort. Post-AI, inflating PR volume or deployment frequency is a side effect of normal tool use — no bad intent required. 3. **Surviving metrics share one trait: they measure what happened, not how much work happened.** Time to recover, customer/business outcomes, and escaped defect rate are structurally immune to AI inflation because an agent can't fake an incident timestamp, adoption, or an escaped bug. ## Claim atoms ### AI severs the effort-output link that metrics assumed > [!quote] ID:rw_metrics_ai_01 > "Metrics were always measuring effort as value, but AI has made effort nearly free. What's left is the harder question your dashboards were never designed to answer." Located in the article's introductory framing. This is the load-bearing claim — every individual metric failure (deployment frequency, cycle time, lines of code, etc.) is an instance of this one structural argument. ### Goodhart's Law becomes automatic under AI tool use > [!quote] ID:rw_metrics_ai_02 > "In the pre-AI era, gaming a metric required deliberate effort. Now it's a side effect of normal tool use." Mid-article, in the Goodhart's Law section. Sharpens a well-known principle into a practically new form — useful when arguing for metric redesign to leaders who assume gaming requires intent. ### Surviving metrics measure outcomes, not activities > [!quote] ID:rw_metrics_ai_03 > "The metrics that survive the AI era will share one characteristic: they measure what happened as a result of engineering work, not how much engineering work happened." From the "Three engineering metrics surviving AI" section. The actionable prescription — a decision criterion usable immediately on any existing dashboard. ## Permanent candidates - **Activity metrics measure tools, not engineers** (H1) — already captured by [[3. Resources/Permanent Notes/AI Activity Metrics Measure Tool Throughput Not Engineering Value]]. Do not duplicate; connect instead. - **Goodhart's Law becomes automatic under AI tool use** (H2) — not yet captured as a standalone claim. Promoted this run: [[3. Resources/Permanent Notes/Goodharts Law Becomes Automatic Under AI Tool Use]]. ## Processing checklist - [x] Checked existing Permanent Notes for duplicate claims before creating — found [[3. Resources/Permanent Notes/AI Activity Metrics Measure Tool Throughput Not Engineering Value]] already covers H1/P1 - [x] Promoted the Goodhart's-Law-automatic claim (H2) as a new Permanent Note - [ ] Verify Nicholas Arcolano's "20% larger AI-generated code" stat against a primary Jellyfish source - [x] Connect into [[3. Resources/MOCs/AI & the Future of Knowledge Work MOC]] ## Links - [[3. Resources/MOCs/AI & the Future of Knowledge Work MOC]] — the article's metric-redesign argument is a concrete instance of measuring AI-mediated knowledge work