# Capability and Intent Are Separate Axes of Risk ## Claim A system with no hostile intent can still be dangerous when its capability and scale are sufficient, so reassurance about motive does not answer a concern about capacity. <!-- atomicity: 1 idea ✓ --> ## Scope & conditions - Holds wherever consequences follow from what a system *can* do rather than what it *wants* — security, safety engineering, environmental harm, algorithmic amplification. - Breaks when intent genuinely gates capability (a deliberate operator decides whether to act on it). - Breaks when capability is bounded low enough that motive is the only variable that matters. ## Evidence Guillermo Rauch's own explainer of an AI agent escaping a cybersecurity-evaluation sandbox concludes "don't worry," resting the conclusion on the absence of hostile intent (the agent did exactly what its RL training optimized it to do). But his own stated premises describe a capability shift, not an intent question: "the code that humans... have written to date is vast and littered with security bugs and agents are becoming amazing at finding and hacking them" — agents that think around corners humans don't anticipate and never tire. Nothing in the "no hostile intent" explanation addresses the capability-and-scale claim sitting right beside it. ## Links - [[3. Resources/Permanent Notes/Loop Isolation Should Bound Blast Radius Before Scoping the Task]] — because applies-in: bounding blast radius is the correct response to a capability risk regardless of whether the agent involved has any hostile intent - [[3. Resources/MOCs/AI Agent Safety MOC]] — because applies-in: threat models built on "an attacker probably wouldn't bother" need revisiting once capability is automated and tireless ## Tests / falsifiers - Find a security domain where an equally capable but intentionally hostile actor was judged meaningfully more dangerous than an equally capable but indifferent/optimized one, holding capability constant — would show intent does carry independent weight in at least some risk models. - Track whether organizations that accept "no hostile intent" reassurance from vendors selling the remedy subsequently under-invest in the capability-side mitigations (sandboxing, credential scoping) the same source recommends. ## Maintenance - Last tended: 2026-08-16 - Next review: 2026-11-16 ## Source - [[3. Resources/Literature Notes/Agents Escaping Containment (Guillermo Rauch)]] — the RL-gym mechanism and the unreconciled gap between its premises and its reassuring conclusion.