# Why AI is going vertical (again) | Dianne Penn (Anthropic) > [!question] Lingering Questions > - "Evals are the new PRDs" is a memorable compression that Penn herself later qualifies (PRDs still hold context and alignment) — where exactly does the eval-as-requirement framing stop applying? > - How is Anthropic's eval suite protected from Goodhart's law as it becomes more central to product decisions? > - This is single-company testimony from a product leader — which claims here would replicate at a company without frontier-lab-scale research/product integration? > [!tip] Article Ideas > - "Evals as executable requirements" — cross-apply the test-driven-development-for-PMs framing to WordPress feature development: what would a WordPress core feature's "eval" look like as an alternative/complement to a written PRD? > [!link] Cross Reference > - [[3. Resources/Permanent Notes/The Demo-to-Product Gap Is Human Work Not Model Work]] — same underlying claim (product surface, not model capability, is where value gets built) from a different source; connect rather than duplicate > - [[3. Resources/Permanent Notes/Eval Quality Is the Rate-Limiter on Safe Agent Autonomy]] — adjacent but distinct: that note is about evals gating safe agent autonomy, this source is about evals as a PM requirements-writing practice > [!quote] Vocabulary > - *research product manager* — Penn's term for a PM embedded directly with researchers, translating user feedback into reproducible failure categories and evals rather than working from a separate roadmap > - *Labs* (Anthropic) — framed in the episode as a venue for discontinuous bets outside the core roadmap ## Bib capsule Lenny Rachitsky interviews Dianne Penn, a product leader at Anthropic, on product management inside a frontier AI lab — how research and product functions integrate, and why general model capability alone doesn't produce adoption. ## Author's thesis A capable model doesn't create user adoption by itself — it needs a product surface (a "vehicle"), domain context, and a feedback loop that translates vague user complaints into reproducible, testable failure cases. As implementation gets cheaper and faster, the scarce resource shifts from building the thing to deciding what's worth building — which raises, not lowers, the value of detailed user-centered product judgment. ## Key arguments (≤3) 1. **Models need a product vehicle to reach users.** Penn credits Opus 4.5's impact partly to Claude Code as the product experience through which people actually encountered the model — capability without a fitting surface doesn't translate into adoption. 2. **Evals function as executable requirements.** Penn frames the practice as "test-driven development for PMs" — writing the eval first turns a vague user complaint into a reproducible, gradable case that researchers can act on, complementing (not replacing) PRDs. 3. **Cheaper building raises the premium on judgment.** As implementation speed increases, the bottleneck shifts to deciding what deserves to be built at all — user-centered, first-principles reasoning becomes more valuable, not less, as SaaS-era pattern-matching stops predicting new AI-native user behavior. ## Claim atoms ### Models need products to reach users > [!quote] ID:rw_1040672517 > "What was magical about Opus 4.5 is we also now not just had a model but a vehicle, a great product experience like Claude Code." Where: 00:30. A capability becomes valuable through a product experience that makes it usable — not through capability alone. ### Evals make feedback executable > [!quote] ID:rw_1040672520 > "It's basically test-driven development for PMs; you write the test first." Where: 47:00. Reproducible eval cases shorten the distance from a vague user complaint to concrete research action. ### User-centered work grows more important > [!quote] ID:rw_1040672521 > "the role of people who are user-centric, who go into the details of understanding what users are trying to accomplish" Where: 83:31. Cheap implementation raises the premium on judgment about what deserves to exist in the first place. ## Permanent candidates - **[P1] Evals function as executable requirements, distinct from PRDs** — genuinely new angle: the vault has existing notes on eval quality gating agent autonomy and on the demo-to-product gap being human work, but not on evals-as-requirements-writing-practice for product managers specifically. Worth its own atom. - **[P2 — skip, largely covered]** The "model needs a product vehicle" claim (H1) restates ground already held by [[3. Resources/Permanent Notes/The Demo-to-Product Gap Is Human Work Not Model Work]] — connect, don't duplicate. ## Processing checklist - [ ] Look for independent (non-Anthropic) accounts of the "evals as PRDs" practice before treating it as more than one company's internal method - [x] Connect into relevant MOC — [[3. Resources/MOCs/AI & the Future of Knowledge Work MOC]] and [[3. Resources/MOCs/Product & Startup Instincts MOC]] ## Links - [[3. Resources/MOCs/Product & Startup Instincts MOC]] — core subject match: AI product management practice - [[3. Resources/MOCs/AI & the Future of Knowledge Work MOC]] — the "cheaper building raises the premium on judgment" argument - [[3. Resources/Permanent Notes/The Demo-to-Product Gap Is Human Work Not Model Work]] — parent pattern for the model-needs-a-vehicle claim