# Evals Function as Executable Requirements Not PRD Replacements ## Claim Writing an eval before building a fix turns a vague user complaint into a reproducible, gradable test case — the same discipline test-driven development applies to code, applied to product decisions — while a PRD still does the separate job of holding context and stakeholder alignment. <!-- atomicity: 1 idea ✓ --> ## Scope & conditions - Holds where user feedback is genuinely ambiguous ("this feels wrong," "it doesn't work well") and needs translation into a concrete, checkable case before anyone can act on it. - Doesn't claim evals replace PRDs — a PRD's job (context, rationale, stakeholder buy-in) is different from an eval's job (a checkable pass/fail case), and conflating the two is the "false dichotomy" reading Penn explicitly warns against. - Weakens where the desired behavior can't be cheaply or reliably scored (open-ended taste judgments, novel domains without a clear "good" to encode) — there, an eval adds false precision rather than clarity. ## Evidence Dianne Penn describes her team's practice as "test-driven development for PMs; you write the test first" — the team's job is translating vague user feedback into reproducible failure categories and evals *before* asking researchers to change model behavior, so the request arrives as something checkable rather than a restated complaint. This complements, rather than substitutes for, a PRD's traditional role of carrying context and alignment across the team. ## Links - [[AI & the Future of Knowledge Work MOC]] — turning vague complaints into gradable cases, which is what makes AI work reviewable - [[3. Resources/Permanent Notes/The Demo-to-Product Gap Is Human Work Not Model Work]] — because extends; evals-as-requirements is one concrete piece of the human-supplied infrastructure that closes the demo-to-product gap. - [[3. Resources/Permanent Notes/Eval Quality Is the Rate-Limiter on Safe Agent Autonomy]] — because contrasts on purpose; that note treats evals as an autonomy-safety gate, this one treats evals as a requirements-writing discipline for human product decisions — same artifact, different function. - [[3. Resources/MOCs/Product & Startup Instincts MOC]] — because applies-in; a reusable practice for any product team translating fuzzy feedback into actionable work. ## Tests / falsifiers - Find a product team that adopted "eval-first" feedback translation and check whether time-from-complaint-to-fix shortened compared to their prior PRD-only process — if it didn't, the claimed benefit doesn't hold outside Anthropic's specific context. - Look for a case where an eval was written but a PRD's context/alignment role turned out to be unnecessary (or vice versa) — if one artifact reliably substitutes for the other, "not PRD replacements" is wrong. ## Maintenance - Last tended: 2026-08-16 - Next review: 2026-11-16 ## Source - [[3. Resources/Literature Notes/Why AI is going vertical (again) - Dianne Penn (Anthropic)]] — Lenny's Podcast interview, 47:00 timestamp.