# Company-Facing AI Agents Need Release Discipline Not Ad-Hoc Edits ## Claim An AI agent that speaks for a company should be changed under the same discipline as shipped software — versioned, tested against a fixed adversarial suite before going live, and rollback-able in one action — not edited ad hoc. <!-- atomicity: 1 idea ✓ --> ## Scope & conditions - Holds for AI systems with company-facing, trust-bearing output: support agents, sales agents, documentation agents. - Less critical for purely internal, low-stakes experimentation where a bad answer costs little. - Doesn't establish which specific eval suite size or pass-rate threshold is sufficient — GravityKit's 44-scenario suite and 39/44 pass rate is one data point, not a benchmark. ## Evidence GravityKit locked its DocsBot-powered support widget's prompt only after running it against a fixed 44-scenario adversarial test suite drawn from real customer conversations, scoring factual accuracy, source-grounded answers, correct routing, and escalation behavior. Prompt and source changes are versioned with one-call rollback and gated behind re-running the eval suite before going live — the article frames this explicitly as "a release-managed support channel," treating the agent as a deployed software artifact rather than a chatbot install. ## Links - [[3. Resources/Permanent Notes/Safe Agent Changes Come From Guardrails and Observability Not Trust]] — because evidence-for; this is a concrete case study instantiating that general claim's guardrail-gating and blast-radius-control principles for a customer-facing support agent specifically. - [[3. Resources/Permanent Notes/Adversarial Practice Builds Transferable Confidence Through Worst-Case Exposure]] — because applies-in; the adversarial eval suite is worst-case exposure testing applied to a bot instead of a person. - [[3. Resources/MOCs/AI Agent Safety MOC]] — because applies-in; release discipline is one of the concrete mechanisms that MOC tracks. ## Tests / falsifiers - Compare incident rate (factual errors reaching customers, mis-routed escalations) between teams that gate agent changes behind a fixed eval suite and teams that edit prompts ad hoc. - If teams with informal, undisciplined change processes show equally low incident rates over a comparable volume of interactions, the claim weakens — release discipline would be sufficient but not necessary. ## Maintenance - Last tended: 2026-08-16 - Next review: 2026-11-16 ## Source - [[3. Resources/Literature Notes/How GravityKit Made AI Support Earn Its Place on the Front Line]] — reports the 44-scenario eval suite, the June prompt lock, and the versioned-rollback release pattern this claim generalizes from.