A Predictable Low Error Rate Becomes Attacker Infrastructure

# A Predictable Low Error Rate Becomes Attacker Infrastructure ## Claim A system error rate low enough for defenders to dismiss as acceptable noise is, from an attacker's perspective, a reliable and reproducible target precisely because it's low and predictable.  ## Scope & conditions - Holds when the error is reproducible on demand by anyone who tries — not a one-off fluke, but a rate that shows up consistently across attempts (a hallucination rate, a misconfiguration rate, a default-credential rate). - The defender/attacker asymmetry is the mechanism: a defender averages the error across all users and calls it small; an attacker doesn't need the average, they need one instance, and a predictable rate tells them exactly how to manufacture that instance deliberately rather than wait for it. - Doesn't claim every low error rate is exploitable — only that "the rate is small" is not itself a reason to dismiss the risk, because smallness and predictability are different properties and this pattern specifically weaponizes the second one. ## Evidence AI coding assistants hallucinate nonexistent package names at a low, dismissible-sounding rate. A defender reads that as noise. An attacker reads the same number as an instruction: register the hallucinated package name preemptively, ship mostly-legitimate code with one malicious addition, and wait for AI assistants to recommend that exact name into real codebases. The rate isn't a nuisance to route around — it's a target list an attacker can generate on demand by running the same coding assistants defenders use. > "a tenth of a percent of all developers are grabbing some nonsense. I'm going to go reserve that namespace. It's not nonsense anymore." ## Links - [[3. Resources/Permanent Notes/An AI Pursuing an Objective Can Cause Attack-Equivalent Harm Without Malicious Instruction]] — extends; both are about AI-system behavior producing attack-equivalent outcomes without any adversarial instruction being given to the AI itself - [[3. Resources/Permanent Notes/A Verified Fraction Without a Comparison Baseline Cannot Be Scored]] — contrasts; that note is about a rate being uninterpretable without a baseline, this note is about a rate being fully interpretable and actionable — just by the wrong audience - [[3. Resources/MOCs/AI Agent Safety MOC]] — applies-in; supply-chain risk introduced by AI tooling's own predictable failure modes ## Tests / falsifiers - Check whether any of the specific hallucinated package names Sonatype identified have since been registered by unknown parties — direct evidence the mechanism is being exploited rather than only theoretically possible. - Find a low, predictable error rate that has persisted for years without attacker exploitation despite being technically reproducible — that would mean predictability alone isn't sufficient, and some other condition (attacker awareness, ease of registration, payload delivery path) is doing real work in the claim. ## Maintenance - Last tended: 2026-08-16 - Next review: 2026-11-16 ## Source - [[3. Resources/Literature Notes/Scalable Security and Compliance in the Age of AI]] — Eddie Knight, Revanite, CNCF/KubeCon talk