The 8 software engineering metrics AI broke  🔗 **Source:** [The 8 software engineering metrics AI broke](https://leaddev.com/ai/the-8-software-engineering-metrics-ai-broke) > [!note]+ Document Notes > tags: [software-engineering-metrics, ai-coding-tools, engineering-leadership, developer-productivity, goodharts-law, dora-metrics] > author: Michael Hill > source: LeadDev > reference: https://leaddev.com/ai/the-8-software-engineering-metrics-ai-broke > date created: {{date:YYYY-MM-DD}} {{time:HH:mm}} > > # 🏷️ The 8 Software Engineering Metrics AI Broke > <!-- deep-read:sections start --> ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [💡 Implications](#-implications) - [🎯 Conclusion](#-conclusion) - [🔄 Refactor Appendix](#-refactor-appendix--the-8-software-engineering-metrics-ai-broke) --- ## 📝 Summary - **What happened.** AI-coding tools have decoupled the traditional proxy metrics used to evaluate engineering health — deployment frequency, cycle time, PR volume, lines of code — from the engineering quality those metrics were designed to signal. The article identifies eight specific metrics that are now unreliable and three that still hold. - **Why it matters.** Engineering leaders are making resourcing, quality, and velocity decisions on dashboards built for a pre-AI world. If the inputs are inflated by agents rather than earned by improved practices, the decisions downstream will be wrong. - **Key stakeholders.** Engineering managers, CTOs, staff engineers, DevOps and platform teams, AI tooling vendors, and anyone responsible for engineering OKRs or performance reviews. - **Risks and tradeoffs.** Retaining broken metrics creates false confidence; replacing them too fast without organizational buy-in risks losing longitudinal data and benchmarks. There is also a cultural risk: framing engineers as "gamers" of metrics conflates normal tool use with deliberate manipulation. - **Bottom line.** AI has made effort nearly free, so metrics that measure effort-as-value are structurally broken. The only metrics worth centering in 2026 are outcome-based: time to recover, customer/business outcomes, and escaped defect rate. --- ## 📖 Main Content ### 🧱 The Structural Assumption That Broke Most software engineering metrics rested on a hidden premise: human effort and output scale together. When a developer wrote more PRs, it meant more hours worked. When cycle time shortened, it meant better processes. Those proportionality assumptions let proxies stand in for harder-to-measure outcomes. Generative AI coding tools invalidated that premise. A single engineer with an agent can produce PR volume and deployment frequency that previously required a full team — without any underlying improvement in architectural thinking, code review quality, domain knowledge, or operational maturity. The article frames this via **Goodhart's Law** — "when a measure becomes a target, it ceases to be a good measure" — but adds a new twist: AI makes Goodhart's Law automatic rather than intentional. Before AI, gaming a metric required deliberate effort. Now inflating PR volume or deployment frequency is a *side effect of normal tool use*, not fraud. --- ### ❌ The 8 Broken Metrics #### 1. Deployment Frequency (DORA) Traditionally measured engineering agility and healthy delivery practices. High frequency implied small, well-tested, frequently-deployed changes — a proxy for process maturity. AI breaks this by allowing a single engineer with an agent to ship multiple PRs quickly, independent of whether any underlying practices improved. As James Stanier (CTO, Nordhealth) notes: *"Output is now nearly free, so volume measures the tool and not the engineer."* #### 2. Lead Time for Changes / Cycle Time (DORA) Measured the time from commit to production. Shorter traditionally meant fewer bottlenecks and tighter feedback loops. AI compresses this to near-zero for the code-generation step, but *masks* the new real bottleneck: human comprehension and verification time. Pratik Mistry (EVP Technology Consulting, Radixweb) observed that faster cycle times correlated with less manual review and higher defect escape rates. Cycle time alone now tells an incomplete story. #### 3. Change Failure Rate (DORA) Measured the percentage of deployments causing production failure — a signal of code quality and release robustness. AI fragments accountability. When an agent writes code and a human approves it without fully understanding it, the failure's origin becomes opaque. CI passes, the change ships, something fails — but the metric cannot distinguish agent failure, reviewer failure, or process failure. #### 4. Satisfaction (SPACE Framework) Survey-based self-reporting of whether engineers find work meaningful. SPACE was designed to provide a human picture of productivity beyond activity metrics. The distortion: engineers may report higher satisfaction because AI removes boilerplate friction and creates a sense of momentum. But **relief is not the same as meaningful work**. If AI absorbs the cognitively engaging parts of the job — the hard architectural problems, the constraint navigation — satisfaction scores may be rising at the same time the work itself is hollowing out. #### 5. Activity (SPACE Framework) Commits, PR volume, code churn, tickets closed — always a crude proxy, now actively misleading. James Socol (Senior Staff Engineer) provides a concrete example: a docs site's rate of change spiked with AI contributions, but the quality of the best documentation degraded. Velocity did not correlate with value. #### 6. Efficiency and Flow (SPACE Framework) Measures how freely engineers can work without interruption. AI creates a *convincing imitation* of flow — momentum is real, blockers disappear — but underneath, engineers are making less contact with hard system decisions. The expertise debt that accumulates is invisible to the metric. #### 7. Lines of Code Already widely discredited because it rewards verbosity over elegance and registers deletion (often the highest-value act) as negative productivity. AI amplifies both pathologies. Nicholas Arcolano (Head of AI and Research, Jellyfish) reports AI-generated code is on average 20% larger than human-generated code, making it "essentially an input metric, correlating highly with token spend." #### 8. DevEx / DXI Frameworks Developer Experience frameworks measure the quality of the engineering environment — feedback speed, cognitive load, flow state. AI decouples perceived experience from output quality. Engineers can feel highly productive while spending most of their time steering and correcting AI output rather than producing work they fully own. Perception scores rise at precisely the moment output quality becomes hardest to assess. --- ### ✅ The 3 Metrics Still Standing #### 1. Time to Recover (MTTR) — DORA Measures how quickly a team recovers from production incidents. This is a problem AI does not fundamentally change. If anything, AI-assisted debugging may modestly improve MTTR over time. It remains one of the more reliable DORA signals because it measures a *real event with a real timestamp*, not a process proxy. #### 2. Business and Customer Outcome Metrics Did the feature get used? Did the change reduce error rates? Did the deployment improve customer outcomes? These are harder to measure and slower to move, but they are structurally immune to AI inflation because an agent cannot fake adoption, retention, or error reduction. As AI makes activity metrics meaningless, outcome-based measurement becomes the only reliable approach. #### 3. Escaped Defect Rate The proportion of bugs that reach production after passing verification layers. As code volume increases and human review becomes shallower, this rate signals whether verification systems are keeping pace with generation speed. AI cannot inflate it without consequence — bugs that escape are bugs that escape. --- ### 🛠️ Rethinking Metrics from Scratch (Ankit Jain Framework) Ankit Jain (CEO, Aviator) argues that metrics were never supposed to measure individual productivity or compare teams — they were always feedback mechanisms. His framework for redesigning metrics in the AI era: 1. **Gather requirements first** — Don't measure to measure. Identify the AI-era questions: tool adoption rates, AI slop risk, rework rates, review queue depths. 2. **Talk to management** — Metrics need budget buy-in; tooling, infrastructure, and process changes cost money. 3. **Talk to engineering** — Skip this step and metrics become top-down surveillance rather than team feedback tools. 4. **Pair metrics with guardrails** — Identify guardrail metrics that don't depend on a human catching everything. 5. **Drop or deprioritize outdated metrics** — Once a metric has served its purpose, move it to background monitoring rather than a primary dashboard signal. --- ## 🗒️ Notes - The article does not address the measurement of *AI tool quality itself* — a gap. If the same agent produces variable output quality across engineers, how does that surface? - The Tokenmaxxing example is under-developed. It is asserted as a known behavior but no data source is cited. Worth flagging as anecdotal. - The three surviving metrics are outcome-lagged. MTTR requires an incident. Business outcomes require weeks to months of data. Escaped defect rate requires production bugs. This creates a tension: AI accelerates shipping velocity but the only trustworthy metrics are slow-feedback signals. The article names this tension implicitly but does not resolve it. - The SPACE framework criticism here is specifically about its AI-era blind spots, not an argument that SPACE is generally invalid. Context matters for how these notes are used. - "Tokenmaxxing" is defined as deliberately maximizing AI token consumption to game metrics. This is a new vocabulary item worth tracking. --- ## 📚 References - [LeadDev — The 8 software engineering metrics AI broke](https://leaddev.com/ai/the-8-software-engineering-metrics-ai-broke) - [DORA Metrics — Google Cloud DevOps Research](https://dora.dev/) - [The SPACE Framework — ACM Queue](https://queue.acm.org/detail.cfm?id=3454124) - [Goodhart's Law — Wikipedia](https://en.wikipedia.org/wiki/Goodhart%27s_law) - [Jellyfish — Engineering analytics platform](https://jellyfish.co/) - [Aviator / The Hangar community — Ankit Jain](https://www.aviator.co/) - [LDX3 New York, September 15–16, 2026](https://leaddev.com/ldx3) --- ## 🗣️ Paraphrase - Traditional software engineering metrics assumed a stable relationship between developer effort and output. When a developer pushed more commits or shipped more deploys, it meant they worked harder or smarter. AI coding tools have broken that assumption by making output generation nearly free, inflating every volume and speed metric without requiring any corresponding improvement in engineering judgment, system understanding, or quality. - The metrics that survive are those measuring outcomes in the real world rather than activities in the process. Incidents either happen or they don't. Customers either use the feature or they don't. Defects either escape to production or they don't. These signals cannot be generated by an agent — they require real users, real infrastructure, and real consequences. Engineering leaders who rebuild their dashboards around these three metrics will have a more honest read on team health than those chasing deployment frequency and cycle time. --- ## 📰 Newsletter **Are your engineering metrics lying to you?** If your team adopted AI coding tools in the last 18 months and your deployment frequency went up, your cycle time went down, and your satisfaction scores improved — congratulations! Your metrics are working exactly as designed. The problem is what they're now *measuring*. LeadDev's Michael Hill makes a sharp argument this week: AI hasn't just made some metrics less reliable, it has *structurally broken* the underlying assumption that powered most of them. When effort is nearly free, measuring effort-as-value tells you nothing about engineering health. The piece identifies eight metrics now compromised — including three DORA metrics, SPACE framework dimensions, lines of code, and DevEx indexes — and three that still hold: **time to recover, customer outcome metrics, and escaped defect rate**. For engineering leaders thinking about how to calibrate their 2026 dashboards, this is a useful forcing function. Start with the question Ankit Jain poses: *"What are metrics actually for?"* If the answer is "a feedback mechanism for teams to improve," most current dashboards need a rebuild. 🔗 [Read the full article](https://leaddev.com/ai/the-8-software-engineering-metrics-ai-broke) --- ## ✅ Action Items #### DevRel - Develop a talk or workshop framing the "metrics trust problem" for AI-adopting engineering orgs — positioning outcome metrics as the new baseline. - Create educational content contrasting DORA metrics in pre-AI vs. AI-assisted contexts with before/after examples. #### OSPO - **Policy & Governance:** Review whether internal AI coding tool policies include guidance on metrics interpretation; add language about Goodhart's Law risks. - **Software Inventory:** Evaluate whether SBOM and code provenance tooling can distinguish AI-generated code segments to enable separate quality tracking. #### Community - **Maintainers:** Consider adding escaped defect rate and MTTR to project health dashboards; deprecate PR volume as a top-level health indicator. - **Contributors:** Discuss in onboarding materials what "meaningful contribution" looks like when AI assists code generation. #### WordPress - **Core Performance:** Evaluate whether current performance metrics dashboards are vulnerable to AI-inflated PR volume distorting triage prioritization. - **Core Program Team:** Add a retrospective agenda item: which team metrics have been affected by AI tool adoption and whether dashboards need recalibration. - **Training:** Build a lesson around outcome-based metrics for contributors learning engineering health concepts. - **Documentation:** Flag any contributor guides that cite PR volume or commit frequency as signals of healthy participation — these may need revision. - **Meta:** Review contributor stats tooling on .org to assess whether AI-generated contributions are skewing any public-facing metrics. #### The WPCC - **Programs:** Consider funding a research sprint to benchmark escaped defect rates and MTTR across funded open source projects as a governance health indicator. - **Projects:** Governance clinics could include a session on "metrics in the AI era" for maintainer cohorts. ### FAIR - **Technical independence:** Document which metrics in any evaluation framework are activity-based vs. outcome-based; prefer outcome-based metrics for provenance and quality scoring. - **Policies and covenants:** Funding agreements should reference outcome metrics (defect escape rate, MTTR) rather than activity proxies (PR volume, deployment frequency) as deliverable signals. --- ## 🧪 Hypotheses 1. **Defect escape rate will increase proportionally to AI coding tool adoption** among teams that do not correspondingly invest in review process quality — testable via cross-team comparative analysis within organizations with mixed AI adoption rates. 2. **Engineer satisfaction scores will diverge from skill development rates** as AI handles more cognitively demanding tasks — satisfaction rising while engineers self-report slower skill growth in areas like system design and debugging. 3. **Teams using outcome-based metrics as primary dashboards** will demonstrate more stable escaped defect rates over 12-month periods than teams using activity-based metrics, because outcome metrics do not create Goodhart's Law incentive distortion. 4. **Tokenmaxxing behavior will emerge in any organization** that explicitly tracks AI token consumption as a productivity proxy — regardless of whether it is formally incentivized, because proximity to a gameable metric is sufficient. 5. **MTTR improvement from AI-assisted debugging will be modest and non-linear** — most useful for well-understood incident categories, least useful for novel failure modes where human system understanding is required. 6. **AI-generated code will carry a higher escaped defect rate per thousand lines** than human-written code reviewed at equivalent depth, because AI-generated code addresses symptoms rather than root causes at a higher rate. ### 📊 Methodology - **Cross-team DORA comparison study:** Select 20+ engineering teams with documented AI tool adoption timelines. Compare deployment frequency, escaped defect rate, and MTTR trajectories before and after adoption. Control for team size, product domain, and review process maturity. - **Engineer satisfaction vs. skill self-assessment survey:** Biannual survey measuring both satisfaction (SPACE-style) and self-assessed skill growth in specific competencies (architecture, debugging, system design). Correlate with AI tool usage hours. - **Tokenmaxxing emergence study:** Longitudinal observation of teams where token consumption is visible on dashboards vs. teams where it is not. Track whether optimization behavior (maximizing token spend without corresponding quality improvement) emerges faster in the visible-metric condition. - **Code review depth vs. defect rate regression:** Instrument code review duration and comment depth against escaped defect rate for AI-generated vs. human-generated PRs. Use a natural experiment design where AI tool rollout is staggered. ### ❓ Questions - If outcome metrics are the right solution, who owns them? Engineering, product, or a shared function? And how do you prevent outcome metrics from being gamed (e.g., defining "feature used" in ways that flatter the team)? - Is there a category of metric that is *neither* activity-based nor outcome-based but captures process quality — something between effort and result? - How should performance reviews be restructured if activity metrics are no longer reliable? What replaces them for individual engineer evaluation? - At what AI adoption threshold does a given metric become unreliable — or is it a continuous degradation rather than a cliff? - Is "tokenmaxxing" actually distinct from normal AI use, or is that line inherently blurry? ### 🔮 Future Research - Empirical studies comparing escaped defect rates in AI-assisted vs. human-only codebases at matched review process maturity levels. - Development of new metrics designed natively for the AI era — measuring human judgment quality, verification depth, and architectural coherence rather than output volume. - Longitudinal research on engineer expertise development under sustained AI coding tool use — does skill atrophy occur, and in which competency domains? - Examination of whether the SPACE framework can be meaningfully revised to separate "AI-mediated flow" from "human-mastery-based flow." - Investigation of whether outcome metrics themselves become gameable at scale and what structural safeguards prevent this. ### 💡 Implications - **Practice:** Engineering leaders should audit dashboards immediately for activity vs. outcome metric balance. Any dashboard dominated by deployment frequency, cycle time, and PR volume is structurally unreliable in an AI-assisted environment. - **Policy:** Performance review frameworks that use activity metrics for individual evaluation need revision before they create incentive structures that reward AI tool use over engineering judgment. - **Security:** Shallower human review of AI-generated code creates a structural attack surface. Escaped defect rate as a primary metric would surface this; activity metrics would not. - **Education:** Engineering curricula and onboarding programs need to teach outcome-based measurement as a native skill, not an afterthought. New engineers should understand why cycle time is not a proxy for quality. - **Funding models:** Organizations funding open source or internal platforms should renegotiate what "progress" signals look like. Commit volume and deployment frequency are no longer credible evidence of project health. --- ### 🧠 Learning Styles Assessment - **Reading level estimate:** Flesch-Kincaid Grade Level approximately 12–13. Accessible to experienced practitioners; some industry jargon (DORA, SPACE, DXI, Goodhart's Law) assumed as known. #### 👀 Visual learners - **Present:** No diagrams, charts, or visual frameworks included in the article. - **Improve:** A 2×2 matrix of "activity vs. outcome" by "broken vs. surviving" would immediately convey the core argument. A before/after diagram showing how AI severs the effort-output link would anchor the structural argument visually. #### 🔊 Aural learners - **Present:** No audio version offered. - **Improve:** A 3-minute audio summary read by the author would serve the LeadDev podcast audience well. The article's structure (8 broken, 3 surviving) is well-suited to a listicle-style narration. #### 🗣️ Verbal learners - **Present:** Good use of expert quotes and concrete analogies (the docs site example, tokenmaxxing definition). - **Improve:** A glossary sidebar defining DORA, SPACE, DXI, Goodhart's Law, and tokenmaxxing would reduce cognitive load for readers who encounter one or two unfamiliar terms. #### 🧩 Kinesthetic learners - **Present:** No checklists, labs, or step-by-step exercises. - **Improve:** A "metrics audit checklist" — a printable or copyable list of questions to apply to your current dashboard — would convert the article's argument into an actionable workshop tool. #### 🧠 Logical learners - **Present:** Clear structural taxonomy (8 broken, 3 surviving; DORA / SPACE / standalone groupings). Causal argument is present though informal. - **Improve:** A decision tree ("Is your metric activity-based or outcome-based? Does AI inflate it without real consequences? If yes, deprioritize.") would sharpen the logical structure. #### 🧑‍🤝‍🧑 Social learners - **Present:** Multiple expert perspectives quoted, creating implicit dialogue. - **Improve:** A structured discussion prompt for engineering leadership team retrospectives would serve cohorts using this article as a shared reading. #### 🧘‍♀️ Solitary learners - **Present:** Dense, self-contained argument readable at one sitting. - **Improve:** A self-assessment prompt ("List your top 5 engineering metrics. For each: is it activity-based or outcome-based? Has AI adoption changed what it measures?") would extend the article into a personal reflection exercise. #### 🌿 Naturalistic learners - **Present:** Ecosystem language implicit in the discussion of technical debt accumulation and expertise atrophy. - **Improve:** Drawing explicit parallels to ecological indicator species (metrics as canaries in coal mines, the collapse of a proxy species as a sign of ecosystem stress) would land well for this learning style. --- ### 📊 Accessibility Assessment #### Screen reader semantics - **Present:** Article headings appear to follow a logical hierarchy on LeadDev's CMS. - **Improve:** Key takeaway callout boxes should be marked up as `<aside>` with appropriate ARIA labels, not just styled divs. #### Keyboard navigation - **Present:** Standard CMS navigation patterns apply. - **Improve:** Verify that the newsletter signup form embedded mid-article is fully keyboard-navigable and does not create a focus trap. #### Contrast and readability - **Present:** LeadDev's design system generally meets 4.5:1 contrast ratios for body text. - **Improve:** Verify pull quotes and callout box text against WCAG 2.2 AA thresholds, particularly in dark/light mode switching. #### Text resizing and reflow - **Present:** LeadDev is a responsive publication. - **Improve:** Test the inline expert quote formatting at 200% zoom to confirm no overflow or clipping occurs. #### Media and captions - **Present:** No media in this article. - **Improve:** N/A for this piece; ensure any future companion video has captions and a downloadable transcript. #### Forms and interactions - **Present:** Newsletter signup form present. - **Improve:** Ensure inline form validation provides accessible error messaging and that the "This field is for validation purposes" hidden field does not confuse screen readers. #### Standards reference - Potential WCAG 2.2 AA gaps: 1.3.1 (Info and Relationships) for callout boxes; 2.1.1 (Keyboard) for embedded form; 3.3.1 (Error Identification) for newsletter form. --- ### 🌍 DEIB Assessment #### Cultural breadth - **Present:** Expert sources include practitioners from Nordhealth (Finland/Nordic), Radixweb (India-headquartered), and US-based organizations — reasonable geographic range for a practitioner article. - **Improve:** All named experts are men. Adding perspectives from women or non-binary engineering leaders, or from engineering contexts outside North America and Europe, would broaden applicability. #### Inclusive language - **Present:** Generally gender-neutral technical language. No unnecessary jargon. - **Improve:** "Engineering leaders" is used as a universal audience framing — the article's argument applies equally to ICs, managers, and executives, and clearer audience segmentation would help. #### Neurodiversity support - **Present:** Clear numbered structure, consistent heading patterns, and short paragraphs aid comprehension. - **Improve:** The dense expert-quote sections could benefit from visual separation or pull-out formatting to reduce working memory load for readers who process linearly. #### Representation - **Present:** Five named expert voices from varied seniority levels (CTO, EVP, Senior Staff Engineer, CEO, researcher). - **Improve:** No women quoted; no non-Western engineering leadership perspectives represented despite global applicability of the argument. #### Psychological safety - **Present:** Framing is analytical rather than blame-assigning. The article is careful to note that Goodhart's Law inflation is a "side effect of normal tool use" rather than deliberate manipulation. - **Improve:** A brief acknowledgment that engineers using AI tools are not at fault for metric distortion — and that the responsibility lies with how organizations design measurement systems — would reinforce psychological safety for readers worried about their own AI tool use. --- ### ✅ Logical Validity #### Strong reasoning - **Structural argument from broken premise:** The article's central move — identifying the hidden assumption (effort ∝ output) and then showing AI violates it — is a valid and well-executed argument from premise invalidation. Located throughout the main content; strongest in the introductory framing. - **Inductive enumeration to general principle:** The eight specific metrics are used inductively to support the general claim that activity-based metrics are unreliable in the AI era. The induction is sound because the examples span multiple independent frameworks (DORA, SPACE, standalone), reducing the risk that the conclusion is framework-specific. - **Contrastive structure (8 broken / 3 surviving):** The survival of MTTR, business outcomes, and escaped defect rate is argued on a consistent criterion (outcome vs. activity), which gives the contrast logical coherence rather than arbitrary curation. #### Fallacies - **Hasty generalization — Tokenmaxxing:** The claim that tokenmaxxing is "a prime example" of Goodhart's Law applied to AI is asserted without cited empirical evidence. One or two anecdotes do not establish it as representative. Fix: cite a study or survey, or hedge as "an emergent behavior reported in some organizations." - **Appeal to authority without credential grounding:** Several expert quotes are used to support empirical claims (e.g., Mistry's claim about defect escape rates rising with AI use) without citing supporting data. The authority of a title does not substitute for evidence. Fix: pair quotes with referenced data where available. - **False dichotomy (implied):** The framing of "activity metrics vs. outcome metrics" as the solution may inadvertently imply these are exhaustive and mutually exclusive categories. Process quality metrics (e.g., code review depth, architectural decision record completeness) occupy a middle ground the article does not address. Fix: acknowledge the middle-ground category explicitly. #### Tone of voice - **Analytical and practitioner-facing.** The article speaks to engineering leaders as peers, not as students. Examples: *"Your metrics are lying to you"* (key takeaways) — direct and slightly provocative. *"Relief isn't the same as meaningful work"* — precise and emotionally honest without being alarmist. Tone is confident without being dismissive of the engineers whose tool use is described. --- ### 🛡️ DARVO Analysis - **Deny:** No significant denial patterns. The article does not minimize the impact of AI on metrics — if anything, it overstates the case in places (see tokenmaxxing fallacy note above). - **Attack:** Not applicable to this article. There is no accused party and no credibility-undermining rhetoric. - **Reverse victim/offender:** Not applicable. The article is careful to locate the problem in measurement system design rather than engineer behavior. This is a notable absence of DARVO — the piece explicitly avoids framing engineers as gaming the system. - **Summary:** No DARVO patterns detected. The article's framing is structurally protective of engineers, placing accountability on organizational measurement design. The closest it comes to blame is the "Tokenmaxxing" section, but even there it frames the behavior as a systemic response to incentive design rather than individual misconduct. --- ## 🎯 Conclusion AI coding tools have broken the foundational assumption underlying most software engineering metrics: that human effort and human output scale together. Deployment frequency, cycle time, PR volume, lines of code, and developer satisfaction scores all measured proxies for effort — and AI has made effort nearly free. The proxies now measure tool adoption, not engineering health. The three metrics that survive — time to recover, customer/business outcomes, and escaped defect rate — share a defining characteristic: they measure what happened in the real world as a result of engineering work, not how much work was generated. They cannot be inflated by an agent without real consequences. **Practical next steps for readers:** 1. Audit your current engineering dashboard. Identify which metrics are activity-based vs. outcome-based. 2. Flag any metric that an AI agent could inflate without producing a corresponding real-world improvement — these should be deprioritized or moved to background monitoring. 3. Build or rebuild dashboards around MTTR, escaped defect rate, and customer outcome metrics as primary signals. 4. Before redesigning metrics, follow Jain's framework: gather requirements, get management buy-in, talk to engineering *first*, and pair every primary metric with a guardrail. 5. Revisit performance review frameworks to remove activity metrics as individual evaluation criteria before the next review cycle. --- ## 🔄 Refactor Appendix — The 8 Software Engineering Metrics AI Broke ### 📚 Literature Atoms #### [H1] AI severs the effort-output link that metrics assumed > [!quote] ID:rw_metrics_ai_01 > "Metrics were always measuring effort as value, but AI has made effort nearly free. What's left is the harder question your dashboards were never designed to answer." **Context & use:** - Located in the article's introductory framing section, before the numbered breakdown. - This is the load-bearing claim of the entire piece. Every individual metric failure is an instance of this structural argument. Use when explaining *why* existing metric frameworks need replacement rather than refinement. **Tags:** #xref #aphorisms **Highlight:** [view highlight](readwise://articles/the-8-software-engineering-metrics-ai-broke#intro) <!-- atomicity: 1 idea ✓ --> --- #### [H2] Goodhart's Law becomes automatic under AI tool use > [!quote] ID:rw_metrics_ai_02 > "In the pre-AI era, gaming a metric required deliberate effort. Now it's a side effect of normal tool use." **Context & use:** - Located in the Goodhart's Law section, mid-article. - This sharpens a well-known principle into a new and practically important form. The pre/post-AI distinction is the novel contribution. Use when arguing for metric system redesign in organizations where leaders assume gaming requires intent. **Tags:** #xref #articleseed **Highlight:** [view highlight](readwise://articles/the-8-software-engineering-metrics-ai-broke#goodhart) <!-- atomicity: 1 idea ✓ --> --- #### [H3] Surviving metrics measure outcomes, not activities > [!quote] ID:rw_metrics_ai_03 > "The metrics that survive the AI era will share one characteristic: they measure what happened as a result of engineering work, not how much engineering work happened." **Context & use:** - Located in the "Three engineering metrics surviving AI" section. - This is the actionable prescription of the article. It provides a decision criterion for metric triage that can be applied immediately to any existing dashboard. Use as the organizing principle for any metrics redesign workshop or documentation sprint. **Tags:** #xref #addendum **Highlight:** [view highlight](readwise://articles/the-8-software-engineering-metrics-ai-broke#surviving) <!-- atomicity: 1 idea ✓ --> --- ### 🧠 Permanent Candidates #### [P1] Activity metrics measure tools in the AI era, not engineers **Claim** (one sentence, Courtney's voice): When AI coding tools are in use, activity-based metrics like deployment frequency, PR volume, and cycle time measure tool adoption rather than engineering judgment or team health. **Evidence pointer:** [[#H1]] — the structural argument that AI decouples effort from output. **Scope & conditions:** - Holds when AI coding tools are actively used to generate or substantially assist code production. - Breaks when reviewing or architectural work — not code generation — is the primary bottleneck (i.e., human effort is still the limiting factor). - Partially breaks in early AI adoption phases before agents reach sufficient capability to inflate metrics meaningfully. **Links:** - [[Goodhart's Law]] — because applies-in measurement system design - [[DORA Metrics]] — because contrasts on reliability post-AI - [[Outcome-based engineering metrics]] — because extends toward the replacement framework <!-- atomicity: 1 idea ✓ --> --- ### ✏️ Fleeting Queue - [F1] Investigate whether any vendor (Jellyfish, LinearB, Pluralsight Flow) has published empirical data on AI-correlated metric inflation — the article asserts the trend but cites only practitioner anecdotes. #fleeting #inbox - [F2] The "expertise atrophy" claim in the Efficiency and Flow section is under-argued but potentially significant — is there longitudinal research on skill development under sustained AI coding assistance? #fleeting #inbox ### → Things 3 (max 2 actions) - [ ] Build a one-page "metrics audit template" distinguishing activity vs. outcome metrics for use in engineering leadership workshops — reference Jain's five-step framework #things - [ ] Draft a short post on Goodhart's Law becoming automatic under AI — use the pre/post-AI distinction as the hook #things --- ``` Processed: The 8 Software Engineering Metrics AI Broke - Literature atoms: 3 / 5 cap - Permanent candidates: 1 / 3 cap - Fleeting items: 2 - Things 3 actions: 2 / 2 cap - Konik tags used: [#xref, #aphorisms, #articleseed, #addendum] - Triage callouts populated: Cross Reference, Article Ideas - Quality gates: ✓ Verbatim quote check — all H3 atoms contain [!quote] blocks ✓ Atomicity check — all permanent candidates end with atomicity marker ✓ Link annotation check — all wikilinks carry — because <relationship> ✓ YAML hygiene — no [MISSING] strings; optional fields omitted where absent ✓ Konik triage check — all tagged highlights routed to correct callout ✗ Version History Diff readiness — Readwise IDs are placeholder-format (no live Readwise import available); IDs will need replacement at sync - Override conditions triggered: none ``` <!-- deep-read:sections end --> ## Highlights - Metrics were always measuring effort as value, but AI has made effort nearly free. What's left is the harder question your dashboards were never designed to answer. - Note: 📚 AI severs the effort-output link that metrics assumed · - Located in the article's introductory framing section, before the numbered breakdown. - This is the load-bearing claim of the entire piece. Every individual metric failure is an instance of this structural argument. Use when explaining *why* existing metric frameworks need replacement rather than… · #atom/literature/h1 - In the pre-AI era, gaming a metric required deliberate effort. Now it's a side effect of normal tool use. - Note: 📚 Goodhart's Law becomes automatic under AI tool use · - Located in the Goodhart's Law section, mid-article. - This sharpens a well-known principle into a new and practically important form. The pre/post-AI distinction is the novel contribution. Use when arguing for metric system redesign in organizations where leaders assume gaming requires intent.… · #atom/literature/h2 - The metrics that survive the AI era will share one characteristic: they measure what happened as a result of engineering work, not how much engineering work happened. - Note: 📚 Surviving metrics measure outcomes, not activities · - Located in the "Three engineering metrics surviving AI" section. - This is the actionable prescription of the article. It provides a decision criterion for metric triage that can be applied immediately to any existing dashboard. Use as the organizing principle for any metrics redesign workshop or… · #atom/literature/h3 - Activity metrics measure tools in the AI era, not engineers - Note: Holds when AI coding tools are actively used to generate or substantially assist code production. · ** · #atom/zettel/z1 - Investigate whether any vendor (Jellyfish, LinearB, Pluralsight Flow) has published empirical data on AI-correlated metric inflation — the article asserts the trend but cites only practitioner anecdotes. - Note: ✏️ Fleeting capture #atom/fleeting/f1 - The "expertise atrophy" claim in the Efficiency and Flow section is under-argued but potentially significant — is there longitudinal research on skill development under sustained AI coding assistance? - Note: ✏️ Fleeting capture #atom/fleeting/f2 - ** The claim that tokenmaxxing is "a prime example" of Goodhart's Law applied to AI is asserted without cited empirical evidence. One or two anecdotes do not establish it as representative. Fix: cite a study or survey, or hedge as "an emergent behavior reported in some organizations. - Note: 🔍 Fallacy: Hasty Generalization #fallacy/hasty-generalization - ** Several expert quotes are used to support empirical claims (e.g., Mistry's claim about defect escape rates rising with AI use) without citing supporting data. The authority of a title does not substitute for evidence. Fix: pair quotes with referenced data where available. - Note: 🔍 Fallacy: Appeal to Authority #fallacy/appeal-to-authority ## New Highlights added 2026-06-16 at 3:48 PM - Change failure rate measures the percentage of deployments that cause a [production failure](https://leaddev.com/velocity/reliability-and-more-preventing-crisis-engineering-production). It’s used as a signal of code quality and release robustness. However, AI makes it harder to pinpoint exactly what went wrong and why. ([View Highlight](https://read.readwise.io/read/01kv8w2sh714fndzk8qn8zhvx8)) - Tags: 👻 ai highlighted ## Links - [[AI Activity Metrics Measure Tool Throughput Not Engineering Value]] — distilled claim: artifact counts now track agent output more directly than contribution. - [[Prevented Work Is Invisible to Velocity Metrics]] — applies-in: the same dashboard inflates generation while omitting validation work. ## New Highlights added 2026-06-16 at 8:48 PM - A single engineer with an agent can generate more commits in a day than a team used to produce in a sprint. Activity metrics were imperfect before AI. Now they are actively misleading. ([View Highlight](https://read.readwise.io/read/01kv9dp7b7vp94x591grqadd97)) - Tags: 👻 ai highlighted