# Overview of AI Evaluation (The Context Window #05) > [!question] Lingering Questions > - When is LLM-as-a-judge trustworthy enough to gate a release, and when must a deterministic verifier replace it entirely? > - If a lab tunes its model against an open benchmark like Ollie Bench, does the benchmark stop measuring what it was built to measure? > [!tip] Article Ideas > - "Evals Are the Loop's Checker" — connect Grafana's online/offline eval taxonomy directly to the loop-engineering cluster's maker/checker rule; they're the same discipline from two different source communities. > [!link] Cross Reference > - [[3. Resources/Permanent Notes/The Maker Should Not Grade Its Own Homework]] — fact-based rubrics and independent judges are this rule's production-observability implementation. > - [[3. Resources/MOCs/AI Agent Safety MOC]] > [!quote] Vocabulary > - *pass@k vs. pass^k* — pass@k measures whether an agent got a task right at least once in k tries (capability); pass^k measures whether it got it right every time (reliability). Yas reports a 10–20% gap between the two. > - *Fact-based rubric* — a grading approach that runs known-good queries against a controlled environment so a judge can catch a fabricated-but-plausible answer, not just a badly-worded one. ## Bib capsule Nicole Vanderhoven interviewing Yas, "Overview of AI Evaluation (The Context Window #05)," Grafana, 2026. [[3. Resources/Literature Notes/Overview of AI Evaluation (The Context Window 05)]] ## Author's thesis As coding agents write most of the code, the bottleneck shifts from writing to verifying — evals turn a fuzzy "is the agent good?" into a tracked, improvable number, split into online evals (score real production traffic) and offline evals (a reproducible golden set in CI/CD), with fact-based rubrics and pass^k reliability scoring as the tools that catch plausible-but-wrong answers. ## Key arguments 1. Online evals are production observability with an AI flavor (catch a vendor silently swapping models); offline evals are reproducible integration tests (a golden set that gates every change). 2. Fact-based rubrics — known-good queries run against a controlled environment — catch fabricated-but-plausible answers that a text-only LLM-judge misses. 3. pass@k (capability) and pass^k (reliability) are different numbers with a real, reported 10–20% gap between them; reliability, not just capability, is what a shipped feature needs. ## Claim atoms ### An eval tests whether an agent behaves as intended > [!quote] ID:rw_h1 > Evals is essentially your testing to see if the AI agent is behaving like you want. The plain-language anchor definition for everything else in the episode. ### Evals split into offline and online families > [!quote] ID:rw_h2 > So you got broadly speaking like two big sides of evals... offline evals and online evals. The organizing taxonomy, mapped directly onto pre-prod/production testing conventions. ### Developers shift from writing code to verifying intent > [!quote] ID:rw_h3 > Now developers are becoming shepherds of intent. So we're not really writing the code anymore but we are making sure that the intent is clear and that it is implemented. Names the bottleneck shift this whole episode is organized around. ### Fact-based rubrics catch plausible-but-wrong answers Paraphrase (no single verbatim line captured for this atom): running known-good queries against a controlled environment lets a grader catch a fabricated P95 latency number that "looks plausible" but is wrong — a failure mode a text-only LLM-judge would miss. Directly extends the maker/checker rule into observability tooling. ## Permanent candidates - "Reliability differs from capability in repeated trials" (pass@k vs. pass^k) — promoted. - "Fact-based rubrics catch plausible-but-wrong answers judges miss" — promoted. ## Processing checklist - [ ] Find the named "deep software engineering" benchmark referenced as flipping Opus vs. GPT-5.5 rankings and verify the claim - [ ] Check whether Ollie Bench's leaderboard methodology is public enough to audit for tuning-to-the-test - [x] Connect into the relevant MOC — [[3. Resources/MOCs/AI Agent Safety MOC]] ## Links - [[3. Resources/Permanent Notes/The Maker Should Not Grade Its Own Homework]] — fact-based rubrics and independent judges are this rule's production-eval implementation. - [[3. Resources/MOCs/AI Agent Safety MOC]] — eval methodology is core agent-safety infrastructure.