Why AI is going vertical (again) | Dianne Penn (Anthropic) ![rw-book-cover](https://d34adp677peecb.cloudfront.net/static/images/article1.be68295a7e40.png) 🔗 **Source:** [Why AI is going vertical (again) | Dianne Penn (Anthropic)]() > [!note]+ Document Notes > tags: [ai-product-management, model-evaluation, anthropic, product-strategy, user-research, vertical-ai] > author: Lenny Rachitsky and Dianne Penn > source: Lenny's Podcast > reference: https://www.youtube.com/watch?v=tivaWTTVRhY > date created: 2026-08-04 10:08 EDT > > > [!info] Generated analysis — not the author's words. Sections below are machine-written > > from the captured source. Only text inside `> [!quote]` blocks is verbatim from the author. > > Do not quote or attribute any other line to the named author. > > # 🏷️ Title: Why AI is going vertical (again) | Dianne Penn (Anthropic) > > ## 🧭 Table of Contents > - [📝 Summary](#-summary) > - [📖 Main Content](#-main-content) > - [🗒️ Notes](#️-notes) > - [📚 References](#-references) > - [🗣️ Paraphrase](#️-paraphrase) > - [📰 Newsletter](#-newsletter) > - [✅ Action Items](#-action-items) > - [🧪 Hypotheses](#-hypotheses) > - [📊 Methodology](#-methodology) > - [❓ Questions](#-questions) > - [🔮 Future Research](#-future-research) > - [💡 Implications](#-implications) > - [🧠 Learning Styles Assessment](#-learning-styles-assessment) > - [📊 Accessibility Assessment](#-accessibility-assessment) > - [🌍 DEIB Assessment](#-deib-assessment) > - [✅ Logical Validity](#-logical-validity) > - [🛡️ DARVO Analysis](#-darvo-analysis) > - [🎯 Conclusion](#-conclusion) > - [🔄 Refactor Appendix](#-refactor-appendix--why-ai-is-going-vertical-again-dianne-penn-anthropic) > > ### 🔗 References > - [Source video](https://www.youtube.com/watch?v=tivaWTTVRhY) > - References named in the transcript are contextual leads; verify them independently before reuse. > > ## 📝 Summary > - **What happened:** Dianne Penn describes product management inside Anthropic as a bridge between fast-moving model research and specific user outcomes. Model capability alone doesn't create adoption; products, harnesses, context, feedback translation, and evaluations turn general models into useful vertical experiences. > - **Why it matters:** The episode offers an inside account of research-product work and the changing PM role. It is valuable testimony from a leader at one company, not an independent measure of Anthropic's performance or safety. > - **Key stakeholders:** AI product managers, researchers, engineers, founders, users in specialized domains, evaluators, safety teams, and hiring candidates. > - **Risks and tradeoffs:** Vendor-centered evidence, product names mistranscribed in captions, unclear claims about inclusion, rapid-plan churn, concentrating advanced capabilities inside labs, and treating eval scores as complete representations of user value. > - **Bottom line:** Build the model and product surface together, translate feedback into reproducible failures, and keep first-principles user judgment central as production gets faster. > > ## 📖 Main Content > Dianne Penn describes product management inside Anthropic as a bridge between fast-moving model research and specific user outcomes. Model capability alone doesn't create adoption; products, harnesses, context, feedback translation, and evaluations turn general models into useful vertical experiences. > > The episode offers an inside account of research-product work and the changing PM role. It is valuable testimony from a leader at one company, not an independent measure of Anthropic's performance or safety. > > The source is an interview or educational presentation, not a controlled study. Claims about products, markets, organizations, and outcomes should be treated as speaker testimony unless a cited source independently confirms them. > > ### 🗒️ Notes > #### ⏱️ [00:30](https://www.youtube.com/watch?v=tivaWTTVRhY&t=30) Penn links the impact of Opus 4 > - Penn links the impact of Opus 4.5 to Claude Code as the product vehicle through which users experienced the model. > #### ⏱️ [15:57](https://www.youtube.com/watch?v=tivaWTTVRhY&t=957) Rapid capability changes require teams to update plans when new evidence appears > - Rapid capability changes require teams to update plans when new evidence appears. > #### ⏱️ [23:57](https://www.youtube.com/watch?v=tivaWTTVRhY&t=1437) Labs is framed as a place for discontinuous bets outside the core roadmap > - Labs is framed as a place for discontinuous bets outside the core roadmap. > #### ⏱️ [40:52](https://www.youtube.com/watch?v=tivaWTTVRhY&t=2452) Research product managers are expected to reason from user value rather than repeat old SaaS playbooks > - Research product managers are expected to reason from user value rather than repeat old SaaS playbooks. > #### ⏱️ [47:00](https://www.youtube.com/watch?v=tivaWTTVRhY&t=2820) The team translates vague user feedback into reproducible failure categories and evals > - The team translates vague user feedback into reproducible failure categories and evals. > #### ⏱️ [83:31](https://www.youtube.com/watch?v=tivaWTTVRhY&t=5011) Penn argues that easier building increases the need for detailed, user-centered product judgment > - Penn argues that easier building increases the need for detailed, user-centered product judgment. > > ## 📚 References > - [Original video](https://www.youtube.com/watch?v=tivaWTTVRhY) > - Not applicable beyond the source video; the transcript doesn't supply a complete, verified bibliography. > > ## 🗣️ Paraphrase > - Anthropic's product identity formed through repeated experiments that exposed model behavior to users. > - A frontier model and a fitting product surface can amplify each other's adoption. > - Fast capability jumps make static roadmaps unreliable. > - Labs explores large bets that may not fit the core roadmap. > - Research product managers embed with researchers where changes directly affect users. > - Vague feedback becomes useful only after a team reproduces and classifies the failure. > - Evals can act like executable requirements while PRDs still hold context and alignment. > - As implementation gets easier, deciding what is worth building becomes harder and more important. > > ## 📰 Newsletter > The striking line from Anthropic's Dianne Penn is that evals are becoming executable requirements for product teams. A model may be capable, but users meet it through a product, a harness, and a context. WordPress's parallel is familiar: shipped behavior beats a proposal, and a test case beats a vague complaint. > > ## ✅ Action Items > #### DevRel > - Turn vague developer feedback into reproducible cases with clear scope and expected behavior. > #### OSPO > - Fund independent evaluation and document decision rights across research, product, and community stakeholders. > #### Community > - Invite domain experts and affected users into evaluation design before a pilot scales. > #### WordPress > #### Core > - Not applicable to this source; it doesn't identify a direct action for this team. > #### Core Performance > - Not applicable to this source; it doesn't identify a direct action for this team. > #### Core AI > - Use task-specific evals with field replay before promoting model-assisted features. > #### Core Program Team > - Track cross-team assumptions that change after model or API updates. > #### Mobile > - Not applicable to this source; it doesn't identify a direct action for this team. > #### Playground > - Provide reproducible browser fixtures for model and harness experiments. > #### Community > - Not applicable to this source; it doesn't identify a direct action for this team. > #### Training > - Teach eval design, user interviews, and failure reproduction together. > #### Marketing > - Not applicable to this source; it doesn't identify a direct action for this team. > #### Support > - Not applicable to this source; it doesn't identify a direct action for this team. > #### Design > - Test whether model behavior and product surface support the same user goal. > #### Documentation > - Publish eval scope, known gaps, and replay steps near feature guidance. > #### Photos > - Not applicable to this source; it doesn't identify a direct action for this team. > #### TV > - Not applicable to this source; it doesn't identify a direct action for this team. > #### Polyglots > - Not applicable to this source; it doesn't identify a direct action for this team. > #### Accessibility > - Include disabled users and assistive-technology workflows in eval coverage. > #### Meta > - Measure real contributor outcomes, not only model or interface activity. > #### CLI > - Not applicable to this source; it doesn't identify a direct action for this team. > #### Hosting > - Not applicable to this source; it doesn't identify a direct action for this team. > #### Tide > - Not applicable to this source; it doesn't identify a direct action for this team. > #### Openverse > - Not applicable to this source; it doesn't identify a direct action for this team. > #### Themes > - Not applicable to this source; it doesn't identify a direct action for this team. > #### Plugins > - Not applicable to this source; it doesn't identify a direct action for this team. > #### The WPOCC > - Support public, provider-neutral eval methods and domain-expert participation. > ### FAIR > - Keep evaluation artifacts portable, inspectable, and governed outside a single vendor's private scoring system. > > ## 🧪 Hypotheses > - Model launches paired with a purpose-built product surface will produce more sustained adoption than model-only launches. > - Structured eval cases will shorten time from user complaint to research action. > - Teams that revisit roadmaps after capability jumps will ship more useful features than teams that preserve prior plans. > - First-principles user research will predict successful vertical applications better than copying established SaaS patterns. > - PMs who prototype directly with models will translate research capabilities into actionable requirements faster. > > ### 📊 Methodology > - Compare adoption cohorts across model-only and model-plus-product launches while controlling for marketing exposure. > - Track feedback items from receipt through reproduction, classification, eval creation, model change, and user outcome. > - Run prospective roadmap studies that record which assumptions changed after each model release. > - Audit eval suites for coverage, construct validity, demographic breadth, and mismatch with field behavior. > - Interview researchers, PMs, and users separately to test whether internal definitions of value agree. > > ### ❓ Questions > - When does a specialized harness create durable vertical value rather than a temporary wrapper? > - How are eval suites protected from Goodhart's law? > - Who decides that a vague complaint represents a large enough user problem? > - What user groups remain absent from internal feedback loops? > - How should product teams expose uncertainty when model behavior changes between releases? > > ### 🔮 Future Research > - Independent replication of claims about model-product adoption effects. > - Cross-company comparisons of research product management structures. > - Long-term study of vertical AI products after general models improve. > - Accessibility and internationalization audits of eval datasets and product surfaces. > - Governance research on capability access concentrated inside frontier labs. > > ### 💡 Implications > - Practice: turn vague feedback into reproducible cases before asking research teams for changes. > - Policy: record eval ownership, coverage limits, and decisions made from each score. > - Security: include misuse, prompt injection, tool boundaries, and unsafe fallback behavior in release gates. > - Education: teach PMs to prototype, inspect model behavior, and reason from user needs. > - Funding: support independent evaluation and domain experts who can challenge lab-centered evidence. > > ### 🧠 Learning Styles Assessment > - Reading level estimate: Flesch–Kincaid grade 10–12 based on conversational syntax and specialized vocabulary; an exact score requires the cleaned transcript. > #### 👀 Visual learners > - Present: a spoken framework and examples, with limited dependence on visuals in the captured transcript. > - Improve: add a labeled model, process diagram, and before/after example with text alternatives. > #### 🔊 Aural learners > - Present: the interview or narration format supports listening and speaker emphasis. > - Improve: add a short chaptered audio summary and identify speakers consistently. > #### 🗣️ Verbal learners > - Present: definitions, analogies, repeated terms, and concrete stories. > - Improve: add a glossary and a plain-language recap after each major idea. > #### 🧩 Kinesthetic learners > - Present: practical prompts or reflection exercises can be attempted during viewing. > - Improve: add a bounded exercise, expected result, and verification checklist. > #### 🧠 Logical learners > - Present: frameworks and causal claims organize the discussion. > - Improve: distinguish assumptions, evidence, mechanism, and measurable outcome in a table. > #### 🧑‍🤝‍🧑 Social learners > - Present: dialogue exposes agreement, challenge, and clarification. > - Improve: provide moderated discussion prompts and a peer-review rubric. > #### 🧘‍♀️ Solitary learners > - Present: the transcript supports self-paced review and reflection. > - Improve: add a worksheet, pause points, and answer key for self-checks. > #### 🌿 Naturalistic learners > - Present: Not applicable to this source; environmental systems aren't a central teaching device. > - Improve: connect resource use and lifecycle effects only where evidence supports the link. > > ### 📊 Accessibility Assessment > #### Screen reader semantics > - Present: the saved transcript provides text access, but source-video heading and landmark quality wasn't verified. > - Improve: use a logical heading hierarchy, identify speakers, and supply descriptive link text. WCAG 2.2: 1.3.1, 2.4.6. > #### Keyboard navigation > - Present: Not verifiable from transcript content alone. > - Improve: test every player and transcript control by keyboard with visible focus. WCAG 2.2: 2.1.1, 2.4.7. > #### Contrast and readability > - Present: Not verifiable from the transcript. > - Improve: test 4.5:1 normal-text contrast, 3:1 large text, line spacing, and readable line length. WCAG 2.2: 1.4.3, 1.4.12. > #### Text resizing and reflow > - Present: transcript text can support reflow, but the source page wasn't audited. > - Improve: test 200% zoom and 320 CSS-pixel reflow without loss. WCAG 2.2: 1.4.4, 1.4.10. > #### Media and captions > - Present: a machine-captured transcript exists. > - Improve: publish corrected captions, speaker labels, and timestamps; add audio description when visuals carry meaning. WCAG 2.2: 1.2.2, 1.2.5. > #### Forms and interactions > - Present: Not applicable to the spoken content; player controls still require testing. > - Improve: ensure accessible names, target size, status messages, and error recovery. WCAG 2.2: 2.5.8, 4.1.2, 4.1.3. > #### Standards reference > - This is a content-based review, not a conformance claim. A WCAG 2.2 AA audit needs the rendered page, player, captions, keyboard flow, and assistive-technology testing. > > ### 🌍 DEIB Assessment > #### Cultural breadth > - Present: examples come mainly from a US/English-speaking professional context. > - Improve: include workers and users across regions, labor systems, languages, and income levels. > #### Inclusive language > - Present: the conversational language is generally approachable but includes field-specific shorthand. > - Improve: define jargon and avoid idioms where they obscure meaning. > #### Neurodiversity support > - Present: chaptered notes and a transcript support review at a chosen pace. > - Improve: add predictable summaries, visual structure, and low-distraction text. > #### Representation > - Present: the source represents named professional experiences, not a representative sample. > - Improve: add voices from affected groups with different roles, seniority, disability, and access to resources. > #### Psychological safety > - Present: the discussion is candid about uncertainty and tradeoffs. > - Improve: separate critique from personal judgment and point people toward practical support when the topic affects livelihood or identity. > > ### ✅ Logical Validity > #### Strong reasoning > The model-plus-vehicle example offers a coherent causal account of adoption, and the feedback-to-eval workflow explains how product work can become actionable for researchers. > > #### Fallacies > The episode risks success-story selection by emphasizing Anthropic launches that worked. Claims about inclusive access and product impact need independent outcome measures. “Evals are the new PRDs” is a memorable compression, but it becomes a false dichotomy if read literally; Penn later clarifies that PRDs still serve a different purpose. > > #### Tone of voice > Reflective, optimistic, technical, and candid about uncertainty and rapid change. > > ### 🛡️ DARVO Analysis > Not applicable to this source. The interview discusses product practice and organizational learning, not an allegation-response pattern. > > ## 🎯 Conclusion > Build the model and product surface together, translate feedback into reproducible failures, and keep first-principles user judgment central as production gets faster. > > Next steps: test the strongest claims against independent evidence, run a bounded real-world case, and record conditions that change the result. > > ## 🔄 Refactor Appendix — Why AI is going vertical (again) | Dianne Penn (Anthropic) > > > Density: 5 Literature atoms and 1 Permanent candidate. > > Literature quotes match the associated master Readwise highlights. > > ### 📚 Literature Atoms (H3) > > ### [H1] Models need products to reach users > > [!quote] ID:rw_1040672517 > > What was magical about Opus 4.5 is we also now not just had a model but a vehicle, a great product experience like Claude Code. > > **Context & use**: > - 00:30 in the source. > - A capability becomes valuable through a product experience that makes it usable. > > **Tags**: #xref > **Highlight**: [view highlight](readwise://open?highlight=1040672517) > > <!-- atomicity: 1 idea ✓ --> > > ### [H2] New evidence should change product plans > > [!quote] ID:rw_1040672518 > > the adaptability of when you're faced with new information: how do you then make better decisions versus keeping the same plan? > > **Context & use**: > - 15:57 in the source. > - Fast model progress rewards roadmap revision rather than plan defense. > > **Tags**: #xref > **Highlight**: [view highlight](readwise://open?highlight=1040672518) > > <!-- atomicity: 1 idea ✓ --> > > ### [H3] First principles reveal actual user value > > [!quote] ID:rw_1040672519 > > rather than pattern matching, what you used to do in consumer product or B2B SaaS, but actually figuring out what is the user value? > > **Context & use**: > - 40:52 in the source. > - Old category patterns can obscure new user behavior. > > **Tags**: #xref > **Highlight**: [view highlight](readwise://open?highlight=1040672519) > > <!-- atomicity: 1 idea ✓ --> > > ### [H4] Evals make feedback executable > > [!quote] ID:rw_1040672520 > > It's basically test-driven development for PMs; you write the test first. > > **Context & use**: > - 47:00 in the source. > - Reproducible cases shorten the distance from complaint to research action. > > **Tags**: #xref > **Highlight**: [view highlight](readwise://open?highlight=1040672520) > > <!-- atomicity: 1 idea ✓ --> > > ### [H5] User-centered work grows more important > > [!quote] ID:rw_1040672521 > > the role of people who are user-centric, who go into the details of understanding what users are trying to accomplish > > **Context & use**: > - 83:31 in the source. > - Cheap implementation raises the premium on judgment about what deserves to exist. > > **Tags**: #xref > **Highlight**: [view highlight](readwise://open?highlight=1040672521) > > <!-- atomicity: 1 idea ✓ --> > > ### 🧠 Permanent Candidates (H4) > > #### [P1] Product surfaces turn general capability into vertical value > **Claim** (one sentence, Courtney's voice): A general model becomes a vertical product when context, tools, feedback, and evaluation organize it around a specific user outcome. > **Evidence pointer**: [[#H1]] > **Scope & conditions**: This holds when domain workflows and data create measurable value; thin branding around the same generic interaction doesn't establish vertical differentiation. > **Links**: > - [[Model-product fit]] — because extends > - [[Evaluation-driven development]] — because applies-in AI product management > - [[User research]] — because evidence-for vertical value > > <!-- atomicity: 1 idea ✓ --> > > ### ✏️ Fleeting Queue > - [F1] Compare eval-driven product work with WordPress release-test gates. https://www.youtube.com/watch?v=tivaWTTVRhY #fleeting #inbox > - [F2] Map which parts of a vertical product remain valuable as base models improve. https://www.youtube.com/watch?v=tivaWTTVRhY #fleeting #inbox > > ### → Things 3 (max 2 actions) > - [ ] Draft one eval from a vague user complaint and state its coverage limit — source follow-up #things > - [ ] List the domain context and tools required for one vertical workflow — source follow-up #things ## Highlights - What was magical about Opus 4.5 is we also now not just had a model but a vehicle, a great product experience like Claude Code. - Note: Model capability reaches users through a product surface designed for the work. - the adaptability of when you're faced with new information: how do you then make better decisions versus keeping the same plan? - Note: Fast model progress makes adaptation a core product-management competency. - rather than pattern matching, what you used to do in consumer product or B2B SaaS, but actually figuring out what is the user value? - Note: First-principles product thinking matters when precedent ages faster than the technology. - It's basically test-driven development for PMs; you write the test first. - Note: Evals turn vague user complaints into reproducible, actionable product requirements. - the role of people who are user-centric, who go into the details of understanding what users are trying to accomplish - Note: Capable models increase the value of detailed user understanding rather than eliminating product work.