# An Agent Evaluation Without an Incumbent Baseline Cannot Establish Value ## Claim Measuring an agent only against itself, or against nothing, can show that it improved but can never show that it was needed — the comparison that actually answers whether an agent is worth using is against the non-AI tool it would replace. <!-- atomicity: 1 idea ✓ --> ## Scope & conditions - Holds anywhere a team evaluates a new agent or tool without a defined incumbent to compare against — the incumbent doesn't have to be sophisticated, just already in use and already measured. - Strongest in domains where the incumbent has known precision/recall (static analyzers, fuzzers, existing automated checks) — the comparison is cheap to run and hard to fake there. - Doesn't establish that every domain has a clean incumbent to compare against; some agent capabilities genuinely have no non-AI precedent, and this claim doesn't apply until one exists. ## Evidence Trail of Bits builds skill-level benchmarks for their agent product (bug-finding per language, statement-of-work drafting) and names the requirement most eval efforts skip: "you have to evaluate AI versus existing algorithmic tools. You can't just ignore that the rest of the world, you know, exists." The comparison is the one teams avoid precisely because it's the one that can make the new tool look worse — nobody schedules the test that might embarrass the project. But it's the only comparison that answers "does this find bugs the tools I already run would have missed," rather than "does this find bugs." ## Links - [[AI & the Future of Knowledge Work MOC]] — improvement against nothing is not evidence of need — the comparison that decides adoption - [[AI Agent Safety MOC]] — the missing comparison that makes most agent evaluations unreadable - [[AI Cost & Tokenomics MOC]] — the comparison that decides whether agent spend is justified at all - [[3. Resources/Permanent Notes/Goodharts Law Becomes Automatic Under AI Tool Use]] — because applies-in; an eval without a baseline is exactly the setup where automatic Goodharting goes undetected, since there's nothing external to catch the drift. - [[3. Resources/Permanent Notes/A Metric Is Vanity Only Against Your Own Outcome Data]] — because extends; that note says a metric only earns meaning against your own outcome data — this note names the specific outcome data an agent eval needs: the incumbent's actual performance. - [[3. Resources/Permanent Notes/Vendor Usage Anecdotes Need Verification Before Adoption Decisions]] — because applies-in; the speaker describing this discipline is also the vendor selling the agent product being benchmarked, which is exactly the situation that note warns needs independent verification. ## Tests / falsifiers - Find a published agent-vs-incumbent-tool comparison in security tooling and check whether the agent's advantage survives once framed as "caught what the existing analyzer missed" rather than "found bugs." - Track whether Trellis (or any named agent product) ever publishes what its comparison against existing algorithmic tools actually showed — the clip raises the comparison and never reports the result. ## Maintenance - Last tended: 2026-08-16 - Next review: 2026-11-16 ## Source - [[3. Resources/Readwise/Articles/Give Your Agents a Performance Review]] — Trail of Bits clip, 2026-07-31; the baseline-comparison requirement and skill-level benchmark design.