Overview of AI Evaluation  🔗 **Source:** [Overview of AI Evaluation](https://www.youtube.com/watch?v=_XGLBqVpL-Y#transcript) > [!note]+ Document Notes > tags: [ai-evaluation, llm-evals, observability, grafana, ai-engineering, devrel] > author: Nicole Vanderhoven > source: Grafana Labs — The Context Window (podcast) > reference: https://www.youtube.com/watch?v=_XGLBqVpL-Y > date created: 2026-06-28 09:00 > > # 🏷️ Title: Overview of AI Evaluation (The Context Window #05) > <!-- deep-read:sections start --> ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [The Context Window #05 (video)](https://www.youtube.com/watch?v=_XGLBqVpL-Y) - [Anthropic — "Demystifying Evals"](https://www.anthropic.com) - [Ollie Bench (Grafana AI benchmark)](https://github.com/grafana) - [LMArena.ai](https://lmarena.ai) - [Matt Schumer — "Something Big Is Happening"](https://www.mattschumer.com) - [Unprompted (Grafana Labs Medium blog)](https://medium.com) ## 📝 Summary - **What happened.** Grafana developer advocate Nicole Vanderhoven interviews AI engineer Yas (pre-recorded ahead of his two-month break) for a full episode on AI evaluation — "evals." They define what an eval is, split the practice into offline vs online evals, walk through graders/verifiers, and demo Grafana's public **Ollie Bench** benchmark plus the internal **LLM Spec** CLI tool and its self-improving "eval loop." - **Why it matters.** As coding agents write most of the code (Yas says manual code he writes is "very close to zero"), the bottleneck shifts from writing to *verifying*. Evals are how teams turn fuzzy "is the agent good?" questions into a tracked number so they can improve reliably instead of vibe-shipping. - **Key stakeholders.** AI/observability engineers, DevRel teams, product managers (being recruited to author evals), open-source model maintainers (Qwen, Kimi appear on the leaderboard), and labs that might tune against an open benchmark. - **Risks and tradeoffs.** LLM-as-a-judge is itself unreliable; benchmarks can be gamed/tuned-against; agents produce *plausible but wrong* answers that pass shallow checks; reproducibility vs realism is a live tension; AI-generated tests get silently edited to pass. - **Bottom line.** Evals are the durable differentiator. Measure → change → measure, ideally with a coding agent in the loop and verifiable graders, and keep a human reviewing the trajectory, not just the final answer. ## 📖 Main Content The episode reframes the testing discipline Courtney's audience already knows (pre-prod vs production, integration tests, synthetic monitoring) into AI terms. The core move: an AI agent is "programmed" in natural language, so there is no clean mapping from instructions to behavior. The only way to probe that is to run many tasks and grade outcomes. Two families: - **Online evals** — score real production conversations (a sample or all of them), typically via LLM-as-a-judge. Analogous to RUM / synthetic monitoring. Catches live regressions (e.g., a vendor silently swapping a model and quality dropping) but only tests the deployed variant and is hard to verify. - **Offline evals** — a controlled "golden set" run in CI/CD against a real-ish environment. Analogous to integration tests. Reproducible, lets you compare *all* changes (prompts, code, tools), and supports drill-down (PromQL vs LogQL, dashboarding, etc.). Terms defined: a **harness** = the agent plus its tools (and now skills). A **grader/verifier** grades the full transcript of steps against the task. **Fact-based rubrics** run known-good queries against the controlled environment so the judge can catch nuanced wrong answers (e.g., a fabricated P95 latency that "looks plausible"). **pass@k vs pass^k** distinguishes "got it right at least once in three tries" from "got it right all three times" — capability vs reliability, where Yas notes 10–20% gaps. The **eval loop**: measure current state, learn from transcripts, make targeted changes (new tool, tool instructions, system prompt, bug fixes), measure again — and crucially, hand that clean feedback signal to a coding agent so it iterates toward 100% on its own. Contested/uncertain claims: whether security-tuned models (Claude "Mythos" preview) are meaningfully better or marketing; whether benchmark rankings (the new "deep software engineering" benchmark flipping Opus vs GPT-5.5) are reliable when labs tune against them. ### 🗒️ Notes #### ⏱️ [00:00:00](https://www.youtube.com/watch?v=_XGLBqVpL-Y&t=0) Intro and framing - Pre-recorded because Yas is taking a two-month break (not leaving Grafana). Hosts riff on Dutch/Turkish names and Grafana's 3-year/5-year tenure perks. - Yas: AI engineer ~5 years; the old ML team (forecasting, anomaly detection) got absorbed into a larger AI department. #### ⏱️ [00:04:00](https://www.youtube.com/watch?v=_XGLBqVpL-Y&t=240) AI is useful but needs guidance - AI "definitely has use," not crypto/NFT vaporware. Guidance per session has dropped from ~50–70 steerings to ~5–10, but isn't zero. - "If you just blindly v code your way out of this you're not going to end up in a good place." #### ⏱️ [00:07:00](https://www.youtube.com/watch?v=_XGLBqVpL-Y&t=420) Announcements - Grafana Assistant now available for OSS/self-hosted; new "Unprompted" community blog; Yas's first engineering post on building an evaluation loop; weekly AI team demo videos ("warts and all"). - News: Warp open-sourced; ChatGPT's "goblin" personality bug; GPT-5.5; Gemini 3.1 Ultra 2M-token window; Anthropic Claude "Mythos" security preview; Matt Schumer's "Something Big Is Happening." #### ⏱️ [00:14:00](https://www.youtube.com/watch?v=_XGLBqVpL-Y&t=840) Why trusting AI is hard - Models are nondeterministic; cost of writing code → near zero, so PRs are bigger and more numerous; review burden is higher than writing burden. - "Shepherds of intent" framing. Using AI to review AI erodes your own taste for good/bad code. - Process vs outcome: a plausible-looking outcome with a wrong process is not trustworthy ("final answer problem"). #### ⏱️ [00:22:00](https://www.youtube.com/watch?v=_XGLBqVpL-Y&t=1320) Human eval and defining evals - Thumbs up/down and LMArena-style A/B human prefs are useful but don't scale into CI/CD. - Eval defined: does the AI agent behave the way you want? Simple example — ask "who are you," expect "I am Grafana Assistant." #### ⏱️ [00:27:00](https://www.youtube.com/watch?v=_XGLBqVpL-Y&t=1620) Offline vs online - Online = production observability with an AI flavor; turn text into a trackable number, alert on it. Catches vendor model swaps. - Offline = golden set in CI/CD; reproducible; tests every change and supports per-capability drill-down. #### ⏱️ [00:34:00](https://www.youtube.com/watch?v=_XGLBqVpL-Y&t=2040) Anatomy of an eval (Anthropic "Demystifying Evals") - Single-turn (e.g., incident summary): prompt → LLM → graded for readability, completeness, length. Easy. - Agentic: variable trajectories (5 vs 7 steps), runs against a real environment with tools, then grade the whole transcript. Harder and slower. #### ⏱️ [00:42:00](https://www.youtube.com/watch?v=_XGLBqVpL-Y&t=2520) Building evals for Assistant - Tension: as real as possible vs reproducible/controllable. Started with manual test scripts (4–5 scenarios), doesn't scale to 500. - Front-end assistant tested via Playwright (runs as the user in the browser); host considers K6's browser module reaching parity. #### ⏱️ [00:52:00](https://www.youtube.com/watch?v=_XGLBqVpL-Y&t=3120) Ollie Bench - Public benchmark, ~six task categories (dashboarding, investigation, logs, metrics, traces). Built on Harbor (same framework as Terminal Bench); separates harness / environment / tasks. - Synthetic Prometheus/Loki/Tempo data generated in Docker so answers are known up front; fact-based rubrics check against known-good queries. Leaderboard includes open-weight models (Qwen 3.6, Kimi). pass@k vs pass^k reliability gap discussed. #### ⏱️ [01:04:00](https://www.youtube.com/watch?v=_XGLBqVpL-Y&t=3840) Internal evals, the loop, and the future - Internal set = larger superset (private data, Slack-markdown quirks). LLM Spec CLI; `lmspec improve` skill runs measure→improve automatically. - LLM Spec 1.0 → 2.0 (speed/robustness) → Ollie Bench → 3.0. Next: environment fidelity, more/harder tasks, getting PMs and community to contribute evals (K6-extension-style model), bridging online→offline automatically, and "bring your own model" guidance. ## 📚 References - [Anthropic — "Demystifying Evals"](https://www.anthropic.com) — primary source Yas screen-shares on eval structure. - [Ollie Bench repo + leaderboard](https://github.com/grafana) — Grafana's public observability agent benchmark. - [Harbor framework](https://github.com) — benchmarking framework (also behind Terminal Bench). - [LMArena.ai](https://lmarena.ai) — blind A/B human preference evals. - [Playwright](https://playwright.dev) / [K6 browser module](https://k6.io) — front-end test harnesses. - [Matt Schumer — "Something Big Is Happening"](https://www.mattschumer.com) — "no longer needed for the actual technical work." - "Deep software engineering" benchmark — newly released, flips the SWE ranking (cited, not linked in transcript). ## 🗣️ Paraphrase - AI has real, day-to-day utility, but still needs guidance; the number of corrections per session has fallen sharply yet hasn't reached zero, so blind acceptance is a mistake. - Because writing code is now nearly free, the work and the risk move to review; developers become "shepherds of intent" verifying that intent landed. - A plausible-looking output is more dangerous than an obviously broken one, so you must check the process and the facts, not just the final answer. - An eval simply asks whether the agent behaves as intended; online evals score live production traffic, offline evals run a reproducible golden set in CI/CD. - Agentic evals grade the whole trajectory against a real environment, using fact-based rubrics and known-good queries to catch subtle fabrications. - pass@k measures capability; pass^k measures reliability, and the gap between them is where "looks good in marketing" agents fail. - The eval loop — measure, change, measure — can be driven by a coding agent when graders are verifiable, pushing the agent toward 100% on its own. ## 📰 Newsletter **Evals: the unglamorous habit that separates shipping from vibing.** This Grafana episode is a clean primer for anyone wiring AI into a product — including WordPress. The big idea translates directly to Core AI: don't trust an agent because its answer *looks* right; build a "golden set" of tasks with verifiable graders and watch the number over time. Grafana split the work into online evals (score real conversations, alert when a vendor silently swaps a model) and offline evals (reproducible CI/CD runs you can drill into). They also open-sourced **Ollie Bench**, generating synthetic data so the correct answer is known before the agent ever runs — a pattern WordPress could borrow for Playground-based, reproducible AI test fixtures. Best line for the community: developers are becoming "shepherds of intent," and the review burden now outweighs the writing burden. If Core AI, Training, and Docs want a shared vocabulary for "is this agent any good," start here. Bonus: their pitch to recruit product managers to *write* evals is a ready-made WordPress contributor on-ramp. ## ✅ Action Items #### DevRel - Developer advocacy: produce a "WordPress eval 101" talk/demo mirroring this episode's online vs offline framing. - Education: workshop on building a golden-set eval with verifiable graders; office hours on LLM-as-a-judge pitfalls. - Developer experience: ship an example eval harness repo (tasks + environment + grader) others can fork. - Developer marketing: blog series on "shepherds of intent" and the review-burden shift. - Community management: survey contributors on what AI tasks they actually run. - Internal/external: align eval vocabulary across Core AI, Training, and Docs. #### OSPO - Policy & Governance: contribution policy for community-authored evals (ownership, review). - License Compliance: scan benchmark data/fixtures for license cleanliness before public release. - Security Management: evaluate "security-tuned" model claims with an internal eval set, not vendor marketing. - Software Inventory: track which models/harnesses are tested against, with freshness SLOs. - Developer Support: guardrails doc for "don't merge AI-edited tests that just make the branch pass." - Community Engagement: explore funding/coordination for a shared open observability-style WP benchmark. - Strategic Alignment: add eval pass-rate as an OKR for any AI feature. #### Community - Maintainers: release-hygiene rule that AI features ship with an eval set. - Contributors: "write-an-eval" first-issue rotation (PM-friendly, low-code). - Sponsors: fund reproducible test-environment infrastructure (Playground/Docker). - Ecosystem: cross-project drill on AI-output regression detection. #### WordPress - Core: define release criteria that any agentic feature includes regression evals. - Core Performance: track agent latency/step-count as an eval metric, alert on drift. - Core AI: stand up a golden-set eval harness; adopt online vs offline split; pilot LLM-as-a-judge with fact-based rubrics. - **Core Program Team:** add eval authoring to cross-team rituals; coordinate PM-written evals. - Mobile: telemetry-based online evals for any in-app AI. - Playground: reproducible synthetic environments for offline evals (Grafana's Docker pattern fits Playground perfectly). - Community: meetup kit on evaluating AI assistants. - Training: learning path "Evaluating AI Outputs," with pass@k vs pass^k explained. - Marketing: narrative on reliability over hype. - Support: knowledge-base entries on diagnosing "plausible but wrong" AI answers. - Design: UX for surfacing model-fit/leaderboard info to "bring your own model" users. - Documentation: examples-first guide to writing graders/verifiers. - Photos: N/A — not applicable to this source (no media-library angle). - TV: caption/transcribe any WP eval talks for indexing. - Polyglots: glossary for eval terms (harness, grader, golden set) across locales. - Accessibility: ensure any eval dashboards meet WCAG 2.2 AA. - Meta: API reliability evals for .org AI endpoints. - CLI: a `wp eval`-style command pattern for running task suites (mirrors LLM Spec CLI). - Hosting: cross-host checklist for AI feature reliability telemetry. - Tide: add agentic-output false-positive review loops. - Openverse: evaluate AI search/recommendation quality with rubrics. - Themes: N/A — not applicable to this source. - Plugins: provenance/security checks for plugins bundling AI evals. #### The WPCC - Programs: microgrant for building an open WP AI eval suite. - Projects: documentation sprint on eval methodology. - Funding experiments: milestone payouts tied to eval coverage growth. ### FAIR - Technical independence: local, reproducible eval environments (no dependence on a single vendor's judge model). - Policies and covenants: transparency on which models/benchmarks are used and how. - Productized governance artifacts: an eval-coverage scorecard for AI features. ## 🧪 Hypotheses - Teams that maintain a verifiable offline golden set ship AI changes with fewer production regressions than teams relying on human spot-checks. - pass^k (reliability) correlates better with user-perceived quality than pass@k (capability). - Fact-based rubrics (known-good queries) catch more "plausible but wrong" failures than LLM-as-a-judge alone. - Recruiting product managers to author evals increases task coverage of real user intents faster than engineer-only authoring. - Open-sourcing a benchmark causes labs to tune models toward it, measurably improving those models on the benchmark's domain within ~2 release cycles. - Driving the eval loop with a coding agent reaches a fixed quality target in fewer human-hours than manual iteration. ## 📊 Methodology - **Online evals:** sample or score all production conversations; LLM-as-a-judge grades full transcripts; reduce to a tracked metric; alert on movement. Tests only the deployed variant. - **Offline evals:** fixed "golden set" of tasks run in CI/CD against a controlled-but-real environment; reproducible so scores are comparable across runs and people; supports per-capability drill-down. - **Harness separation:** tasks / environment / harness decoupled (Harbor) so the same tasks grade MCP, GCX CLI, Claude Code, or any agent on outcome, not method. - **Synthetic ground truth:** generate Prometheus/Loki/Tempo data in Docker so correct answers are known before the run; graders run known-good queries (fact-based rubrics) and compare. - **Reliability sampling:** run each task 3× and report pass@3 (≥1 success) vs pass^3 (3/3 success). - **The loop:** measure → analyze transcripts → change (tool/prompt/bug) → measure, with a coding agent iterating against a clean feedback signal. ## ❓ Questions - When is LLM-as-a-judge trustworthy enough to gate a release, and when must a deterministic verifier replace it? - How do you keep an offline golden set representative as product usage drifts? - What's the right cadence to regenerate tasks from real (online) conversations into offline evals? - Can "bring your own model" guidance be reduced to a single per-task leaderboard a user reads before opting in? - How do you prevent agents from silently editing tests to pass while still letting them author tests? - If labs tune to your open benchmark, does the benchmark stop measuring what you care about? ## 🔮 Future Research - Automated online→offline pipeline: turn real conversations into verifiable tasks without manual authoring. - Environment-fidelity techniques for multi-plugin, browser-based (front-end) agents. - Standardized, contributable evaluator modules (a "K6 extensions" model for graders). - Cross-language verifier portability (Go internal vs Python/Harbor public). - Benchmark robustness research: detecting and discounting tuning-to-the-test. ## 💡 Implications - **Practice:** evals become a first-class deliverable alongside the AI feature; review, not writing, is the bottleneck. - **Policy:** procurement/governance should require eval evidence over vendor claims (esp. "security-tuned" models). - **Security:** reproducible eval environments reduce reliance on a single judge model and expose subtle failure modes. - **Education:** new curricula needed for prompt/eval literacy; PM-authored evals broaden who contributes. - **Funding:** open benchmarks can attract "free fine-tuning" from labs but need sustained maintenance funding to stay representative. ## 🧠 Learning Styles Assessment - Reading level estimate: Flesch–Kincaid Grade ~9–10 (conversational, but dense with jargon: harness, grader, pass^k, golden set). Spoken-transcript filler lowers formal complexity while raising cognitive load for newcomers. #### 👀 Visual learners - Present: screen-shares of the Anthropic eval post, Ollie Bench leaderboard, and git repo. - Improve: an architecture diagram of harness/environment/grader and an online-vs-offline data-flow visual. #### 🔊 Aural learners - Present: full podcast audio with two-host dialogue. - Improve: a 3-minute audio recap and chapter markers. #### 🗣️ Verbal learners - Present: strong analogies (pre-prod/prod, integration tests, RUM, "shepherds of intent"). - Improve: a glossary callout box for eval terms. #### 🧩 Kinesthetic learners - Present: a public, runnable benchmark (Ollie Bench) to clone and run. - Improve: a step-by-step "write your first eval" lab. #### 🧠 Logical learners - Present: clear taxonomy (online/offline, pass@k/pass^k, single-turn/agentic). - Improve: a decision tree for "which eval type for which question" and a metrics table. #### 🧑‍🤝‍🧑 Social learners - Present: invitation to comment with eval questions; PM-alliance collaboration. - Improve: a community eval-authoring cohort or office hours. #### 🧘‍♀️ Solitary learners - Present: self-paced repo exploration. - Improve: reflection prompts and a self-check quiz on the terms. #### 🌿 Naturalistic learners - Present: ecosystem framing (models, harnesses, environments interacting). - Improve: lifecycle thinking — how an eval set evolves as usage drifts. ## 📊 Accessibility Assessment #### Screen reader semantics - Present: YouTube auto-structure only. - Improve: chaptered description with headings; descriptive alt text for screen-shared dashboards. Maps to WCAG 1.3.1. #### Keyboard navigation - Present: standard YouTube player controls. - Improve: ensure any linked Ollie Bench dashboard is keyboard reachable. WCAG 2.1.1. #### Contrast and readability - Present: unknown for screen-shared UI (small text noted by host "make this a bit bigger"). - Improve: zoom/high-contrast when sharing dashboards. WCAG 1.4.3 / 1.4.11. #### Text resizing and reflow - Present: not addressed. - Improve: provide a text transcript that reflows at 200%. WCAG 1.4.4 / 1.4.10. #### Media and captions - Present: auto-captions likely; quality unverified (transcript shows ASR errors — "Vanderhovven," "v code"). - Improve: human-corrected captions and a downloadable transcript. WCAG 1.2.2 / 1.2.4. #### Forms and interactions - Not applicable to this source — it's a video, no forms. (One-line reason: passive media.) #### Standards reference - Primary gaps map to WCAG 2.2 AA: 1.2.2, 1.2.4, 1.4.3, 1.4.4, 1.3.1. ## 🌍 DEIB Assessment #### Cultural breadth - Present: international team (Belgium-based, Dutch/Turkish heritage, talks in Madrid); open-weight models from non-US labs (Qwen, Kimi). - Improve: examples beyond English/observability; note non-English agent behavior in evals. #### Inclusive language - Present: generally neutral, collegial tone; jargon mostly defined. - Improve: plain-language summary; define harness/grader/pass^k on first use for non-experts. #### Neurodiversity support - Present: conversational chunking by topic. - Improve: consistent headings and a structured outline; the rambling intro raises load. #### Representation - Present: two technical voices, one woman host leading the interview, one engineer guest. - Improve: broader range of practitioners (PMs, community contributors) on future episodes. #### Psychological safety - Present: respectful, self-deprecating humor; "warts and all" framing reduces perfectionism pressure. - Improve: explicit framing that not-knowing eval methods is normal; invite beginner questions. ## ✅ Logical Validity #### Strong reasoning - **Analogical (strong):** mapping online/offline evals to RUM/synthetic monitoring and integration tests (≈[00:27:00]–[00:34:00]) is apt and grounds new terms in known testing practice. - **Causal (strong):** "cost of writing code → near zero, therefore review becomes the bottleneck" is a clean, well-supported causal chain. - **Inductive (strong):** pass@k vs pass^k generalizes from observed 10–20% gaps to a capability-vs-reliability distinction; clarify by reporting sample sizes/variance. #### Fallacies - **Hasty generalization:** "the guidance has come down... now I just need to do that five or 10 times" generalizes from personal sessions. Fix: cite measured data across users/tasks. - **Appeal to novelty / anecdote:** dismissing "use this memory layer = 100x" hype while endorsing OpenSpec/bare-bones setup is largely anecdotal. Fix: A/B the workflows with an eval. - **Circularity (acknowledged):** using AI to review AI and LLM-as-a-judge to grade LLMs — the speakers name this ("if you use AI to review the AI, how much can you trust"). Fix: anchor with deterministic fact-based rubrics, which they do. #### Tone of voice - Candid, peer-to-peer, lightly comedic (goblins, PJs, name jokes), pragmatic and non-hype. Example: "warts and all" demo framing. ## 🛡️ DARVO Analysis - No DARVO present. This is a cooperative technical interview with no accusation, defense, or victim-offender dynamic. The closest rhetorical move is mild skepticism toward vendor hype (security-tuned models, "100x" workflow claims), which is critique, not denial/attack/reversal. ## 🎯 Conclusion - **Bottom line.** Evals convert "is this agent good?" into a tracked, improvable number. Use online evals to watch production, offline golden sets to gate changes reproducibly, fact-based rubrics to catch plausible-but-wrong answers, and pass^k to measure reliability — then let a coding agent drive the measure-change-measure loop while a human still reviews the trajectory. - **Next steps for readers (WordPress angle).** Core AI: stand up a small golden set with verifiable graders; Playground: build reproducible synthetic environments; Training/Docs: ship an "evaluating AI outputs" path; recruit PMs and community to author evals as a low-code contributor on-ramp. ## 🔄 Refactor Appendix — Overview of AI Evaluation (The Context Window #05) > Density: 2–5 Literature atoms (H3), 0–3 Permanent candidates (H4). > Resonance filter applied: dropped weak atoms in favor of load-bearing claims. > Tag pool: #bmf · #addendum · #xref · #articleseed · #storystem · #share · #aphorisms ### 📚 Literature Atoms (H3) #### [H1] An eval tests whether an AI agent behaves as intended > [!quote] ID:rw_h1 > Evals is essentially your testing to see if the AI agent is behaving like you want. **Context & use**: - ~[00:24:00], defining the core concept. - The plain-language anchor for every other eval idea; mirrors the host's "what is eval?" prompt. **Tags**: #xref #aphorisms **Highlight**: [view highlight](readwise://) <!-- atomicity: 1 idea ✓ --> #### [H2] Evals split into offline and online families > [!quote] ID:rw_h2 > So you got broadly speaking like two big sides of evals. Like in our case we talk about offline evals and online evals. **Context & use**: - ~[00:27:00], the taxonomy backbone. - Maps to pre-prod vs production testing; the organizing distinction Courtney's testing audience already knows. **Tags**: #xref **Highlight**: [view highlight](readwise://) <!-- atomicity: 1 idea ✓ --> #### [H3] Developers shift from writing code to verifying intent > [!quote] ID:rw_h3 > now developers are becoming shepherds of intent. So we're not really writing the code anymore but we are making sure that the intent is clear and that it is implemented. **Context & use**: - ~[00:20:00], on the review-burden shift. - Strong DevRel/community framing for "why evals matter now." **Tags**: #xref #aphorisms #share <!-- atomicity: 1 idea ✓ --> #### [H4] Reliability is pass-every-time, not pass-at-least-once > [!quote] ID:rw_h4 > And that's kind of like the difference between okay, the model has somewhere the rough capabilities to solve this task versus the model is actually reliably solving this task like every time that you ask it to solve this task. **Context & use**: - ~[01:00:00], explaining pass@k vs pass^k. - The distinction that separates "demo-good" from "production-reliable" agents. **Tags**: #xref #articleseed <!-- atomicity: 1 idea ✓ --> #### [H5] The eval loop is measure, change, measure — repeatedly > [!quote] ID:rw_h5 > We make the changes and then we measure again and then we make changes and then we measure again. **Context & use**: - ~[01:06:00], defining the evaluation loop a coding agent can drive. - The operational heartbeat of continuous AI improvement. **Tags**: #xref <!-- atomicity: 1 idea ✓ --> ### 🧠 Permanent Candidates (H4) #### [P1] Reliability, not capability, is the real eval target **Claim** (one sentence, Courtney's voice): An AI feature is only trustworthy when it solves a task every time, so reliability (pass^k) — not occasional capability (pass@k) — is the metric to ship on. **Evidence pointer**: [[#H4]] **Scope & conditions**: Holds for agentic tasks with verifiable answers; breaks when "correct" is subjective or unverifiable. **Links**: - [[AI evaluation]] — because extends - [[Plausible but wrong outputs]] — because contrasts on capability-vs-reliability - [[WordPress Core AI release criteria]] — because applies-in gating AI features <!-- atomicity: 1 idea ✓ --> #### [P2] Verify the process, because plausible answers hide failures **Claim** (one sentence, Courtney's voice): Because top models produce plausible-looking wrong answers, grading must check the process and ground-truth facts, not just the final output. **Evidence pointer**: [[#H1]] **Scope & conditions**: Holds when ground truth is computable (fact-based rubrics); weaker when no known-good query exists. **Links**: - [[LLM-as-a-judge limits]] — because contrasts on verifiability - [[Fact-based rubrics]] — because evidence-for - [[Testing process vs outcome]] — because extends <!-- atomicity: 1 idea ✓ --> ### ✏️ Fleeting Queue - [F1] Try Ollie Bench's harness/environment/task separation as a model for a WordPress Playground-based AI eval fixture — https://github.com/grafana #fleeting #inbox - [F2] Read Anthropic "Demystifying Evals" and extract the eval-structure checklist #fleeting #inbox - [F3] Quote to reuse in a talk: "if you just blindly v code your way out of this you're not going to end up in a good place." #fleeting #inbox ### → Things 3 (max 2 actions) - [ ] Draft a "WordPress eval 101" DevRel talk outline using the online-vs-offline framing — #things - [ ] Open a discussion proposing PM/community-authored evals as a low-code contributor on-ramp — #things <!-- deep-read:sections end --> 📄 **Full text:** [[Full Document Contents/Articles/Overview of AI Evaluation.md|Overview of AI Evaluation by Grafana — The Context Window]] ## Highlights - >> Yeah, I like the I forget where I got it from but I like the the phrase that now developers are becoming shepherds of intent. So we're not really writing the code anymore but we are making sure that the intent is clear and that it is implemented. >> Yeah. [clears throat] But on the review the I think the burden is Yeah. ([View Highlight](https://read.readwise.io/read/01kw81jzsky9hr01jrthjw6rcq)) - Note: Zettel: "developers are becoming shepherds of intent." - Tags: ai-engineering, atom/zettel/z1 - And that's useful but it's also not really scalable like you can't run that in a CI/CD pipeline you know. So useful but limited. >> Yeah, we can like on the eval side I I think they are all complimentary, right? So you got broadly speaking like two big sides of evals. Like in our case we talk about offline evals and online evals. ([View Highlight](https://read.readwise.io/read/01kw81jx085f97xywkj459157j)) - Note: Two sides of evals: offline vs online. - Tags: atom/literature/h2, ai-evaluation - Um I mean also we keep talking about evals but what is eval? Evals is essentially your testing to see if the AI agent is behaving like you want. Right? So a simple eval could be let's say I want to make sure like that my uh agent identifies itself correctly. So then your eval is you ask it as a user who are you and you expect uh the LM to answer I am graphana assistant that's a simple one it's much more complicated ones uh around like you know uh querying dashboarding uh like or like you know software engineering benchmarks are like very different as well but that's at at its core it's really about an eval is how do you figure out if this AI agent is any good and the reason we do that is because when we make changes to our agent, how do we know if it's better? ([View Highlight](https://read.readwise.io/read/01kw81jt9cc3at7kwrccp7wk2h)) - Note: Definition: an eval is a test of whether the agent behaves as intended. - Tags: ai-evaluation, atom/literature/h1 - So, we run the same task three times and then we check, did it pass every time of those three times? Like you get three shots. Did you score all three shots or did you just happen to be lucky and you shoot three times and you only score once? And that's kind of like the difference between okay, the model has somewhere the rough capabilities to solve this task versus the model is actually reliably solving this task like every time that you ask it to solve this task. ([View Highlight](https://read.readwise.io/read/01kw81k30vk7szqqsqq9mg29at)) - Note: Reliability vs capability: pass@k repeated-run testing. - Tags: atom/literature/h3, ai-evaluation - We make the changes and then we measure again and then we make changes and then we measure again. We make changes we measure again. The thing is that this loop uh that is can be done by a coding agent. It's not us manually doing this. So the whole setup is that if your evals are defined well enough and robust enough so you got like really good ways to verify if the answer was correct and you can pass this signal like this clean feedback signal back to a coding agent. ([View Highlight](https://read.readwise.io/read/01kw81k5z1e498rt2r9gpqkqej)) - Note: Robust evals become the clean feedback signal that lets a coding agent self-improve. - Tags: atom/literature/h3, ai-evaluation