> [!question] Lingering Questions > - What did Trellis's comparisons against existing algorithmic tools actually show? The clip raises the comparison and never reports the result. > - Is there a held-out eval slice, and who is barred from seeing it? Never mentioned. > [!tip] Article Ideas > - "Name your baseline before you build the eval" — short piece for teams adopting agent tooling, built around the one sharp instruction in this clip. > [!link] Cross Reference > - One of six sibling clips from the same Trail of Bits talk in this batch — read together with *A 7-day Cooldown on Every Dependency*, *A Disagreement on Principles Not on Skills*, *The CEO Goes First*, *All of It's Becoming Due at the Same Time*, *The Machine Doesn't Cook for You*. > - [[3. Resources/Permanent Notes/A Metric Is Vanity Only Against Your Own Outcome Data]] — same discipline, applied to agent benchmarking specifically. > [!quote] Vocabulary > - **evaluation dataset / benchmark** — positive and negative cases plus a comparison arm against the incumbent non-AI tool > - **skill-level benchmark** — evaluating a capability (finding bugs in a language, drafting a document) rather than a model, so the benchmark survives model swaps ## Metadata - Source: [[3. Resources/Readwise/Articles/Give Your Agents a Performance Review]] - Category: Video (YouTube short-form clip) - Published: 2026-07-31 ## Author's thesis Organizations investing in agent systems owe themselves proof the agents work, delivered as a per-skill evaluation dataset containing positive cases, negative cases, and — the requirement most eval efforts skip — a head-to-head comparison against the existing non-AI tool. ## Key arguments 1. **Investment creates an obligation to verify**, and the instrument is a benchmark treated as recurring and comparative, not a one-time acceptance test. 2. **The comparison against existing algorithmic tools is the one everyone skips**, because it's the only comparison that can make the new tool look worse — and it's the one that actually answers "does this find bugs the tools I already run would have missed." 3. **Benchmarks are development infrastructure, not a gate** — once they exist, harness, skill, and model changes become measurable rather than vibes-based. ## Claim atoms ### Benchmark against the tool you'd replace, not against nothing > [!quote] ID:rw_01kywp9w32cyck0n95tq4b0eht > You have to evaluate AI versus existing algorithmic tools. You can't just ignore that the rest of the world, you know, exists. *Where:* ~00:00:30. *Why:* the comparison most likely to be skipped and most likely to change the adoption decision — in security the incumbent has measurable precision and recall, so its absence is a choice, not an oversight. ### Benchmarks make agent iteration measurable > [!quote] ID:rw_01kywp9w32cyck0n95tq4b0eht > once you have these benchmarks, then you can make tweaks to your agent harness, to your skills, to insert new models, and you can figure out whether they're actually better. *Where:* ~00:00:30. *Why:* reframes evals from a gate you pass once to the instrument that makes future changes evaluable at all. ### The unit of evaluation is the skill, not the model *Where:* ~00:00:00. *Why:* Trellis benchmarks "core skills" — bug-finding per language, document drafting — which matches what's actually deployed and survives model swaps underneath it. ## Permanent candidates - **An evaluation without the incumbent as a comparison arm cannot establish value** — strong, general, directly testable in any org adopting agent tooling. ## Processing checklist - [ ] Check whether Trail of Bits has published benchmark methodology publicly - [ ] Look up current agent-benchmark overfitting literature - [x] Connect into the relevant MOC — [[3. Resources/MOCs/AI Agent Safety MOC]] ## Links - [[3. Resources/MOCs/AI Agent Safety MOC]] — eval design and the limits of self-graded benchmarks are core agent-safety infrastructure questions. - [[3. Resources/MOCs/AI & the Future of Knowledge Work MOC]] — agent evaluation practice for teams adopting these tools at work. - [[3. Resources/Permanent Notes/An Agent Evaluation Without an Incumbent Baseline Cannot Establish Value]] — the Permanent note this Literature note generated. - [[3. Resources/Permanent Notes/A Vendors Capability Claim Needs a Paired Comparison Not a Demo]] — extends: the same rule for vendor claims, where a demo alone proves little and a paired comparison against the alternative is what counts - [[3. Resources/Permanent Notes/A Verified Fraction Without a Comparison Baseline Cannot Be Scored]] — evidence-for: a verified-findings percentage needs a comparable baseline before it means anything, which is the head-to-head arm this clip asks for - [[3. Resources/Permanent Notes/A Metric Is Vanity Only Against Your Own Outcome Data]] — applies-in: the clip's benchmark discipline is that note's test, applied to agent skills