# An Agent Optimizing a Metric Will Find Hollow Wins ## Claim Handing an agent a metric to optimise produces technically-valid, practically-useless changes unless someone keeps checking whether each one helps a user rather than only the number. <!-- atomicity: 1 idea ✓ --> ## Evidence **Source**: [[3. Resources/Readwise/Articles/Planning Next Moves Improving Performance When Half Your Stack Is Someone Else's Problem|Planning Next Moves]] > [!quote] > when a measure becomes a target, it stops being a good measure altogether Goodhart's law is old. What is new is the rate. A human optimising a score is bounded by how many ideas they can generate and how much tedium they will tolerate; both act as informal filters, because obviously-hollow changes are not worth a person's afternoon. An agent has neither bound, so it will surface the entire space of technically-valid score-movers including every one a person would have dismissed unexamined. The talk's own worked case shows what a *real* finding looks like by contrast: > [!quote] > returning users over HTTP1 with blobs out of cache was the worst case That is a specific population with a specific failure — the kind of finding a tail-focused investigation produces and an average-focused optimiser buries. ## Scope & conditions - Holds wherever the metric is a proxy rather than the goal itself, which covers nearly all performance, engagement, and quality metrics. - The mitigation is not a better metric — proxies are unavoidable — but a standing human check on whether each accepted change corresponds to a user-visible improvement. - Weakens where the metric *is* the goal (correctness tests, compliance thresholds), because there is no gap for a hollow win to live in. - Related tail-vs-median discipline from the same source: a healthy P50 can conceal a completely different, fixable P99 failure mode, so latency work has to look at the tail specifically. ## Links - [[AI Agent Safety MOC]] — why unattended agent optimisation needs a standing human check on user-visible benefit - [[3. Resources/Permanent Notes/Rewarding AI Usage Manufactures Usage Not Value]] — the human-incentive twin of the same Goodhart failure - [[3. Resources/Permanent Notes/The Maker Should Not Grade Its Own Homework]] — why the checking party cannot be the optimising party - [[3. Resources/Permanent Notes/Encoding What Good Looks Like Is What Makes Agent Verification Trustworthy]] — the constructive response: write down the thing the metric was standing in for - [[3. Resources/Permanent Notes/Adoption Frequency Does Not Measure Output Reliability]] — another proxy that detaches from what it was chosen to represent