Opportunity Solution Trees Built by AI Inside Vistaly's Rebuild

Opportunity Solution Trees Built by AI: Inside Vistaly's Rebuild ![rw-book-cover](https://d34adp677peecb.cloudfront.net/static/images/article0.00998d930354.png) 🔗 **Source:** [Opportunity Solution Trees Built by AI: Inside Vistaly's Rebuild](https://www.youtube.com/watch?v=zverZHvdpPo) > [!note]+ Document Notes > tags: [opportunity-solution-trees, continuous-discovery, ai-evals, agentic-workflows, product-discovery, llm-orchestration] > author: Product Talk (Teresa Torres) with Matt O'Connell, CP Dehli and Steve Klein of Vistaly > source: Product Talk (YouTube) > reference: https://www.youtube.com/watch?v=zverZHvdpPo > date created: 2026-10-02 09:24 > > > [!info] Generated analysis — not the author's words. Sections below are machine-written > > from the captured source. Only text inside `> [!quote]` blocks is verbatim from the author. > > Do not quote or attribute any other line to the named author. > > # 🏷️ Title: Opportunity Solution Trees Built by AI: Inside Vistaly's Rebuild >  ## 🧭 Table of contents - [📝 Summary](#-summary) - [📖 Main content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning styles assessment](#-learning-styles-assessment) - [📊 Accessibility assessment](#-accessibility-assessment) - [🌍 DEIB assessment](#-deib-assessment) - [✅ Logical validity](#-logical-validity) - [🛡️ DARVO analysis](#️-darvo-analysis) - [🎯 Conclusion](#-conclusion) - [🔄 Refactor appendix](#-refactor-appendix--opportunity-solution-trees-built-by-ai-inside-vistalys-rebuild) ## 📝 Summary - **What happened.** Product Talk (channel of Teresa Torres) uploaded a 1:19:59 episode of its *Just Now Possible* series on 2026-10-01. Torres interviews the three Vistaly co-founders, Matt O'Connell (CEO), CP Dehli and Steve Klein, about rebuilding their opportunity solution tree (OST) software from scratch with AI. Torres is also a participant: the host says at 00:00:46 that they have a partnership with Vistaly and are building with the team, and the description names Torres as the builder of the AI synthesis services for Vistaly V2. - **Name garbling.** The auto-captions spell the company "Vistili," "Vistily," "Vistil" and "Vistley," the host "Teresa Taus," one founder "CP Daily," Jira "Jerry," Zencity "Zen City," and evals "emails" (00:53:19, 01:04:52). This analysis uses the spellings from the video description. Quote blocks keep the transcript spelling. - **The claim.** V1 was a manual canvas for drawing trees. V2 takes uploaded customer interviews, turns each into an interview snapshot, then drafts and updates the tree with an agentic workflow. The speakers say the rewrite reached alpha users in 2.5 months, against about three years for V1 (00:33:56). - **What the hour teaches.** The practical content is the failure analysis: chat UIs read as slow, a "house of cards" where a bad snapshot corrupts the tree above it, a seesaw between missing subgroups and badly framed parents that 16 prompt variants didn't fix, unstable IDs, and change sets that a plain diff can't produce. - **Why it matters.** Few vendors describe an LLM product's failure modes in this detail, including a fix that moved from prompt changes to orchestration (a repair loop, 00:54:02) and an eval that became a production guardrail (00:54:47). - **Key stakeholders.** Vistaly (vendor, three co-founders), Torres and Product Talk (host, partner and builder), Vistaly's beta and V1 customers, AWS Bedrock and Anthropic (infrastructure and model providers named), and enterprise buyers who require SOC 2, GDPR and EU data residency. - **Commercial interests.** Two exist and the video states neither as a disclosure beyond the partnership line. Product Talk sells coaching and courses (Torres mentions courses, a community and AI coaches) and now ships a product that carries a "powered by Product Talk" label (00:46:53). Vistaly sells the software. The episode doubles as a launch conversation for a V2 released "about a month ago" (00:10:53). Financial terms of the partnership aren't in the transcript. - **Claims versus evidence.** Statements about how the AI performs are vendor claims with no numbers: the snapshots are "90% right" (00:19:33), change-set bugs are rare (01:00:33), and Sonnet 4.6 scores beat Sonnet 5 on the team's evals (01:15:44). No eval results are shown in the audio. - **Risks and tradeoffs.** Quality of output depends on input quality, and the beta showed sales demos, stakeholder meetings and LLM-generated fake transcripts uploaded as "interviews" (00:44:44). A tree "that looks like truth" from weak evidence is the risk Torres names directly (00:46:53). Rollback and versioning for an agent that edits shared trees are unsolved (01:07:47). - **Bottom line.** Treat the architecture lessons (layers of analysis, repair loops, stable IDs, semantic change logs, evals as guardrails) as transferable and checkable. Treat product-quality and speed statements as unmeasured vendor claims from a partner interview. ## 📖 Main content **The product and the framework.** An opportunity solution tree is a tree diagram. The root is a business outcome. Below it sit opportunities, which Torres defines at 00:08:01 as customer needs, pain points and desires. Each opportunity branches into candidate solutions, and each solution into assumptions to test. Vistaly V1 was a purpose-built canvas for these trees. The founders say they picked it over general whiteboards because a node can link to the interviews that produced it, so a reader can click an opportunity and see the customer evidence (00:10:53 to 00:12:19). **Why AI forced a rebuild.** By the tail of V1, customers were pasting interviews into Claude to shortcut synthesis, and a few people were posting prototypes that generated trees instantly (00:14:28). The founders report churn and feature requests that pointed the same way. Their first AI experiment was a guided chat UI that walked a user through one interview's insights, one at a time. Users called it slow even while saying it beat their old process (00:18:07 to 00:19:33). The speakers gutted it in favor of automatic extraction. **Who the tool competes with.** Torres argues that most teams skip real synthesis: they keep the two or three most memorable points and put those on a tree. The product therefore competes with "shallow synthesis," not with a careful researcher (00:23:48). Torres cites two things that moved them toward AI help: a controlled comparison where they hand-synthesized interviews and then compared Claude's output on coverage and how actionable the opportunities were (00:24:32), and a study Torres describes as showing AI raises the floor for novices and the ceiling for experts (00:25:59). Torres doesn't name the study. **V2 mechanics.** A user uploads three interviews. The system produces an interview snapshot per interview (Torres's template for single-interview synthesis), then drafts a tree. Later uploads update the tree. Early versions forced users to review each snapshot before the tree appeared. Users wanted the tree, so the team removed that gate (00:38:13). **Layers of analysis.** The central engineering idea is a pyramid: raw transcripts, then key moments and opportunities per interview, then consolidated key moments across interviews, then the tree. Torres borrows the shape from a Zencity episode (00:40:24). Torres's thesis is that pipelines versus agents is the easy question; the hard part is the scope of each layer and making each one good enough that the next doesn't "fall apart" (00:43:16). **The eval story.** A customer said she would have to "clean up this branch" (00:41:50). Torres traced that to a measurable error: too many children under one node. Torres added a code assertion (more than N children) as a cheap pre-filter so the LLM judge ran only on flagged nodes, then struggled to calibrate the judge because the labeled examples contained other errors. Fixing parent framing and adding subgroups traded off against each other. Four new evals and 16 experiment variants didn't balance them. Moving the check into a repair loop, where a tool checks the model's work and sends it back, did (00:51:09 to 00:55:29). **Change sets and IDs.** The launched beta "barely understood" change sets (00:59:50). The LLM changed, dropped or misused IDs. Diffing input and output trees produced ambiguous change sets (a merge could read as a delete plus reassignments). The fix was to teach the agent a set of semantic moves, such as merge, move and reframe, and have it log them, then use deterministic code to verify the change set reproduces the output tree. Torres says this took a month of rewriting. **Contested or unverifiable in this source.** - No error rates, sample sizes or customer counts back any quality claim. The only counts are "the first eight customers" (00:32:29), "over a hundred different teams" in beta transcripts (00:44:44), and the eval counts (four evals, 16 variants). - "People just want the AI to do it" (00:37:31) is drawn from a small, early beta and Torres's calls. It conflicts with the earlier stated goal of human-in-the-loop collaboration (00:18:07), and the speakers say so themselves. - The 40-point transcript variation system and the "three of the 40" finding (00:45:28) are described but not shown. The caption text "three of the 40 transcripts" most likely means three of 40 variation points; confirm against Torres's blog post. - Model comparisons (Sonnet 4.6 versus Sonnet 5, Haiku for evals) are reported without scores. Version names are caption renderings ("Sonnet 46," "ha coup. I model"). ## 🗒️ Notes #### ⏱️ [00:00:00](https://www.youtube.com/watch?v=zverZHvdpPo&t=0) Meet the founders - Matt O'Connell (CEO, product engineering background), CP Dehli (technical background) and Steve Klein (B2B SaaS background, product, design and marketing, joined about 2.5 years earlier) introduce themselves. - Torres discloses the partnership and a role at 00:00:46: building with the team as "engineer slash discovery person." #### ⏱️ [00:01:05](https://www.youtube.com/watch?v=zverZHvdpPo&t=65) How Vistaly began - O'Connell and Dehli have worked together about 10 years across agencies, startups and Fortune 500 companies. They saw teams track delivery work but not the problems worth solving. - Early prototypes used Toyota's 5 Whys (captioned "5 Ys," then "five W's") for root-cause analysis. They found Torres's work after starting the company. #### ⏱️ [00:03:10](https://www.youtube.com/watch?v=zverZHvdpPo&t=190) Community-first partnership - Torres gets three to four vendor emails a week asking for advice. The stock reply points to their community, and Torres says only O'Connell acted on it. > [!quote] [00:03:42] > The best way to connect with me is to join my community and to connect with my community. - O'Connell participated for about two years before any partnership talk (00:04:27), and Torres says they featured O'Connell in their customer recruiting course as an example of joining communities without pitching. The relationship is a commercial one that started as community goodwill. #### ⏱️ [00:04:56](https://www.youtube.com/watch?v=zverZHvdpPo&t=296) Why build OST software - An early prototype was a radial graph with a "big hairy problem" at the center. OSTs added the link to product and business outcomes (00:05:55). - Software's advantage: each opportunity can carry detail (the Jira epic-and-tickets analogy, captioned "Jerry") behind a clean visual. An early marketing line was "don't make me drag boxes around." > [!quote] [00:07:19] > don't make me drag boxes around #### ⏱️ [00:07:18](https://www.youtube.com/watch?v=zverZHvdpPo&t=438) OST explained fast - Root is the business outcome. Opportunities are customer needs, pain points and desires. Solutions branch from opportunities, assumptions from solutions. Torres says they'll add a visual to the video. - The original company name was "Aligner" (00:08:43), reflecting the alignment problem. > [!quote] [00:08:01] > So when we talk about an opportunity, think about it as a customer need. #### ⏱️ [00:10:28](https://www.youtube.com/watch?v=zverZHvdpPo&t=628) Evidence inside the tree - Whiteboard trees overload: Torres says they skip drawing lines in Miro (captioned "Muro") because moving a branch means redrawing them, which then hides what's grouped with what (00:09:26). - In V2, clicking an opportunity shows the interview clips behind it, playable if video was uploaded. A founder frames it as a research repository that's actionable. > [!quote] [00:10:53] > the bulk of the way the product works is through adding your interviews and we ultimately create an opportunity solution tree out of those. #### ⏱️ [00:12:56](https://www.youtube.com/watch?v=zverZHvdpPo&t=776) AI changes discovery - Founders say ChatGPT's arrival brought requests, churn and customer prototypes that generate trees instantly (00:14:28). No churn figures are given. - Torres had integrated their interview coach into V1's snapshot flow earlier (00:15:11). > [!quote] [00:14:28] > they were really like throwing their interviews at Claude and trying to shortcut that work #### ⏱️ [00:16:58](https://www.youtube.com/watch?v=zverZHvdpPo&t=1018) Chat UI experiment - V1's chat walked users through each extracted insight, searching tree cards and creating new ones through tools. An insight took about 20 steps, some asynchronous database queries plus streaming (00:21:42). - It processed one interview at a time, so three interviews meant three passes (00:22:24). - Users called it slow but faster than before. Torres links the complaint to chat itself. > [!quote] [00:18:07] > One of the first things we heard was like this is great but it's super slow. > [!quote] [00:19:33] > Yeah, I think some of this is this has come up in a few of our episodes where chat sets a speed expectation. #### ⏱️ [00:23:00](https://www.youtube.com/watch?v=zverZHvdpPo&t=1380) Should AI do synthesis - Torres's reframing: the competitor is shallow synthesis, not good synthesis done slowly. - The evidence is one controlled comparison against Claude on Torres's own interviews (00:24:32) and a paper Torres doesn't name (00:25:59). Both are self-reported; neither is linked. > [!quote] [00:23:48] > And so we're not competing with making good synthesis faster. We're competing with make shallow synthesis better. #### ⏱️ [00:27:10](https://www.youtube.com/watch?v=zverZHvdpPo&t=1630) Rebuilding as V2 - It began as a Claude Code prototype (captioned "cloud code") about a week of play, then a decision to rebuild (00:27:25). The founders cite old tech-stack and architecture baggage as a second reason. - V1 laid every OST side by side in one giant tree. V2 scopes one opportunity space per team. Torres says they tell people not to build a mega tree: use a KPI tree for outcomes, one OST per outcome (00:29:32). > [!quote] [00:27:25] > We really have an opportunity here to burn the boats #### ⏱️ [00:32:08](https://www.youtube.com/watch?v=zverZHvdpPo&t=1928) Turning off V1 signups - Demos pulled prospects toward V2, so Vistaly stopped V1 signups and sent people to a V2 beta waitlist. Torres calls the choice bold and a surprise (00:32:29). - Rationale given: avoid onboarding and customer-success load on a product being replaced, since a startup dies by "a thousand paper cuts." The speakers say AI changes the build-time math. > [!quote] [00:33:56] > I think we had users on in two and a half months. #### ⏱️ [00:34:43](https://www.youtube.com/watch?v=zverZHvdpPo&t=2083) V2 workflow and tensions - Flow: upload three interviews, get a snapshot (user journey, pain points, unmet needs) per interview, press synthesize, get a first-draft tree. Adding interviews updates the tree. - Torres says they framed output as a "first draft" to invite collaboration; customers mostly want the AI to do it (00:37:31). The first V2 flow showed each snapshot before the tree, and the team dropped that gate (00:38:13). > [!quote] [00:36:05] > we're using AI that is doing the work for you. #### ⏱️ [00:39:35](https://www.youtube.com/watch?v=zverZHvdpPo&t=2375) Trusting AI outputs - The "house of cards": the tree depends on each snapshot being correct. A founder describes "right versus fast" as the tension (00:38:57 to 00:39:39). - Claim (not measured here): people now expect AI to be right on the first try, so the product moves toward automatic output with correction tools. > [!quote] [00:39:39] > people are at the point where they really trust and expect AI to get get it right on the on the first try #### ⏱️ [00:39:56](https://www.youtube.com/watch?v=zverZHvdpPo&t=2396) Layered analysis pyramid - Torres recounts the Zencity episode: raw data points at the base, an insights layer above, a hypothesis layer above that, so an agent doesn't search raw data on every question. > [!quote] [00:40:24] > You can't just throw a LLM at those raw data points because it's a crapshoot on whether they return the right thing. #### ⏱️ [00:41:25](https://www.youtube.com/watch?v=zverZHvdpPo&t=2485) Building better trees - Tree steps: per-interview key moments and opportunities, consolidated key moments across interviews, then placement of each interview opportunity. A wrong consolidation leaves opportunities "no right place to live." - A customer's remark during a call became the thread for a blog post and an evals guide Torres had published or was about to publish. > [!quote] [00:41:50] > I guess I'll have to go back and clean up this branch. > [!quote] [00:42:33] > when people talk about AI products, they talk about, oh, you can have pipelines and you can have agents and you can have this and that. And like honestly those pieces feel like the easy parts. #### ⏱️ [00:43:34](https://www.youtube.com/watch?v=zverZHvdpPo&t=2614) Bad inputs and coaching - A founder says many users don't run good interviews (leading questions, "would you like X"). Torres, from beta transcripts of over 100 teams, saw sales demos, stakeholder meetings and team conversations labeled as interviews. - Fake inputs: an LLM-generated transcript set where all three had the same character count. Torres applied their own 40-point variation system, built to make synthetic test transcripts, to flag them (00:45:28). - The open question: should the product push back on weak signal instead of presenting it as evidence? > [!quote] [00:44:44] > Oh, we got AI generated transcripts. > [!quote] [00:46:53] > I don't want to present a tree that looks like truth #### ⏱️ [00:47:20](https://www.youtube.com/watch?v=zverZHvdpPo&t=2840) Teaching through software - A brand-risk thread: one of the speakers worries "Teresa's stamp of approval" implies correctness. Another reports a customer team that wouldn't edit its tree because it was "Teresa's version" (00:47:35). - Torres sees an upside: teach synthesis through opinionated software and nudge toward better questions, customers over stakeholders, more interviews before choosing a target. - Vistaly's solution space and assumption tests exist but have no AI help yet (00:49:00). A founder describes users driving tree edits through MCP and mapping a Jira epic backlog into the opportunity space. > [!quote] [00:47:35] > I think this opens up a way to teach through software #### ⏱️ [00:50:12](https://www.youtube.com/watch?v=zverZHvdpPo&t=3012) Evals and guardrails - Error mode: too many children under a node. Step one was a code assertion (more than N children), which also cut LLM-judge cost by running the judge only on flagged nodes (00:51:09). - LLM-as-judge needs calibration against the author's labels. The labeled set carried other errors, so the judge invented subgroups that weren't there (00:51:53 to 00:52:36). - The seesaw: more subgroups produced badly framed or repetitive parents, strict framing rules made the model add no parents at all (00:53:19). Four new evals, 16 variants, no prompt fix (00:54:02). > [!quote] [00:51:53] > But you can't just say evaluate this and trust it. You have to calibrate the judge against your own judgment. #### ⏱️ [00:55:23](https://www.youtube.com/watch?v=zverZHvdpPo&t=3323) Deterministic change sets - A change set lists what to do to a tree given new interviews, and every card and ID must be exactly right or the update fails. - The beta launched with a prototype that used toy data, manufactured IDs and no input trees. Real data exposed ID changes, dropped IDs and opportunity text pasted into ID fields (00:56:56 to 00:57:41). - Diffing input and output trees fails because several change sets can yield the same output (a merge versus a delete plus reassignment), which is why the agent now logs semantic moves (00:58:24 to 00:59:50). > [!quote] [00:58:24] > And we learned really quickly that the same input output tree pair can generate more than one change set. #### ⏱️ [01:00:54](https://www.youtube.com/watch?v=zverZHvdpPo&t=3654) UX for tree updates - With tree edits solved, the founders say the open problem is how a user comprehends a long list of changes. Small trees show the diff fine; larger customers hit too many changes (01:03:27). - Both the V1 interview walkthrough and the first V2 tree walkthrough were removed after users said "just show me." - Feedback ideas: thumbs up or down, plus customer actions (reframe, move) fed into the eval harness as implicit signals (01:04:52 to 01:05:38). - Dehli or Klein (speaker unlabeled) says customer calls surface more than ratings. > [!quote] [01:03:27] > And I think what we're learning is people just want us to tell them the answer. > [!quote] [01:00:33] > Just more LM calls. LM calls all the way down. #### ⏱️ [01:07:12](https://www.youtube.com/watch?v=zverZHvdpPo&t=4032) Chat agents and versioning - An agent is returning to the canvas, behind a feature flag and not live. Torres cites a Banani episode on canvas-plus-agent design tools. - The agent sometimes hallucinates, so the team is choosing which customer use cases should get tools (01:07:47). Versioning and undo matter because trees "hold a lot of weight" and teammates share them. - Torres's chatbot framing: users see either a useless bot or an expert bot that does everything (01:09:57). > [!quote] [01:07:47] > if an agent is able to go in and make a lot of changes to the tree, how do we allow you to undo those things easily > [!quote] [01:09:57] > You're either a dumb chatbot that can't help me at all or you better be an expert chatbot that lets me do everything. #### ⏱️ [01:11:17](https://www.youtube.com/watch?v=zverZHvdpPo&t=4277) Privacy compliance tradeoffs - A founder asks how AI startups handle privacy. Torres reports an eight-month approval at one client company for their AI coaches (01:12:04) and says they chose not to build HIPAA (captioned "HIPPA"), GDPR and SOC 2 compliance themselves. - Vistaly splits data residency between US and EU, supports deletion, and moved from the direct Claude API to AWS Bedrock for EU inference (01:13:30). ISO work is under way. - Torres calls the burden procedural, not technical. > [!quote] [01:12:47] > It doesn't seem like it's technically hard. It's procedurally bureaucratic. #### ⏱️ [01:15:06](https://www.youtube.com/watch?v=zverZHvdpPo&t=4506) Model selection reality - Bedrock lags the Anthropic API on newer features and caps input and output tokens independently of the model, even for open-source models (01:14:15). - Torres runs a model-switching router for experiments but keeps production on one family. The stated reason is that prompts are model- and version-specific, and their evals score Sonnet 4.6 (captioned "Sonnet 46") above Sonnet 5 until prompts migrate (01:15:44). No scores are given. - Prompt caching differs by provider; Torres hasn't found an open-source success case because of dependence on Anthropic-specific behavior (01:17:11). > [!quote] [01:15:44] > the way you write prompts is very model specific even version specific #### ⏱️ [01:17:32](https://www.youtube.com/watch?v=zverZHvdpPo&t=4652) What comes next - Near-term focus: conversational ways to explain what changed in the tree, which a speaker says is harder than assumed (01:18:00). - Candidates: more interview types, the solution space, and signal-quality coaching. A founder warns against broad footprints before quality is there. > [!quote] [01:18:37] > But Claude doesn't solve the hard problems. #### ⏱️ [01:19:08](https://www.youtube.com/watch?v=zverZHvdpPo&t=4748) Closing and links - Torres points to vistaly.com (captioned "vistally.com") and the Product Talk blog about building the services, and floats a follow-up episode on the agent-plus-canvas UX. - The standard subscribe-and-rate outro closes the episode. ## 📚 References The transcript itself contains no links; the following come from the video description or are primary sources for named entities. **From the video description** - [Vistaly](https://vistaly.com/): the product under discussion. Captions spell it "vistally.com." - [Product Talk](https://www.producttalk.org/): Torres's site; the host says they blog the build there. - [Model Context Protocol](https://modelcontextprotocol.io/): the protocol a founder says customers use to drive tree edits. - [AWS Bedrock](https://aws.amazon.com/bedrock/): the inference service Vistaly moved to for European data residency. - [Just Now Possible episode on Banani](https://www.producttalk.org/building-banani-how-a-canvas-first-ai-designer-is-raising-the-floor-on-product-design/): the canvas-plus-agent design tool Torres mentions at 01:05:38. - [Just Now Possible episode on Zencity](https://www.producttalk.org/powering-government-with-community-voices/): the source of the layered data pyramid (00:40:24). - [All Things Product episode on quality of evidence, with Petra Wille](https://www.producttalk.org/quality-of-evidence-all-things-product-podcast-with-teresa-torres-petra-wille/): referenced at 00:44:44. - [Spotify](https://open.spotify.com/episode/63cEIKic84ba7X7kBawaey) and [Apple Podcasts](https://podcasts.apple.com/us/podcast/generating-opportunity-solution-trees-with-ai-how/id1838832993?i=1000792530309): audio versions listed in the description. **Named in the audio, no URL given** - Torres's book *Continuous Discovery Habits* (the "CDH community" at 00:03:42) and the AI evals guide and blog post on the "clean up this branch" case (00:50:25). Find the live posts on producttalk.org before citing. - The novice-versus-expert AI productivity study (00:25:59). The speaker doesn't name it; no source is given. - Toyota's 5 Whys, the root-cause method behind Vistaly's early prototypes (00:01:31). **Primary sources for named entities** - [Jira](https://www.atlassian.com/software/jira): the tracker the captions render as "Jerry" (00:05:55 and 00:49:43). - [GDPR text](https://eur-lex.europa.eu/eli/reg/2016/679/oj) and [HIPAA overview](https://www.hhs.gov/hipaa/index.html): the regimes named at 01:12:04. SOC 2 and ISO are named as well; the speakers give no specific standard numbers. ## 🗣️ Paraphrase Teresa Torres sits down with the three people who run Vistaly, the company that sells software for drawing opportunity solution trees, and Torres discloses up front that the partnership exists with them. The team explains why they stopped bolting AI onto a canvas and rebuilt the product so that customer interviews go in and a draft tree comes out. Their first AI try, a chat that stepped users through each insight, worked but felt slow, so they dropped it. Torres reframes the goal: most teams skip real synthesis, so the product has to beat a rushed version, not an ideal one. The longer second half is an engineering post-mortem. A customer's remark about cleaning up a messy branch turned into an eval, then into 16 prompt experiments that never balanced two competing errors, then into a repair loop where a checking tool sends the model back to fix its own work. Change sets needed the agent to log what it did as named moves, because a diff can't tell a merge from a delete. IDs drifted until deterministic code pinned them. The conversation ends on open problems: showing a big change set without overwhelming anyone, letting an agent edit shared trees without a way to undo, and meeting enterprise privacy rules with EU hosting on Bedrock. Every statement about how well the AI works is the speakers' own report, and none comes with a number. ## 📰 Newsletter **Your AI feature has a middle layer, and that's where it breaks.** Teresa Torres's *Just Now Possible* put the three Vistaly co-founders on camera for 80 minutes to explain how they rebuilt an opportunity-solution-tree app around AI. Skip the product pitch. Watch for the repair loop, the stable-ID fight and the "clean up this branch" customer, which is a bug report you can reuse as a test case. Why WordPress folks should care: anyone shipping AI-assisted content, block-pattern or plugin-review tools hits the same wall. A model that works on three examples invents IDs on real data, and a prompt tweak that fixes one error brings back another. The episode names two fixes you can try this week: a cheap code check before you pay for an LLM judge, and an orchestration change in place of a prompt change. Disclosure for the road: the host is a Vistaly partner, so treat the quality claims as claims. Links: [the Product Talk site](https://www.producttalk.org/), [Vistaly](https://vistaly.com/), [Model Context Protocol](https://modelcontextprotocol.io/), and the [Zencity episode](https://www.producttalk.org/powering-government-with-community-voices/) behind the layered-analysis idea. ## ✅ Action items #### DevRel - Developer advocacy: write one case study of "a customer quote becomes an eval" from your own product, with the error mode, the measure and the fix named. Torres's branch story at 00:41:50 is the template. - Education: run a workshop on LLM-as-judge calibration: label 20 examples, compare judge to labels, report accuracy and precision (00:51:53). - Developer experience: add a "stable IDs and provenance" section to any docs that let an LLM edit structured data, with the failure modes seen here (changed, dropped and misplaced IDs, 00:57:41). - Developer marketing: when you cite vendor AI quality, attach a number and a method. This episode offers none, so quote the mechanism, not the "90% right." - Community management: when a vendor partner interviews its own partner, add a standing line in link-share channels noting the relationship (00:00:46). - Internal/external/cross-project: ask Vistaly or Product Talk for the published eval results and the novice-versus-expert study citation before reusing either argument. #### OSPO - Policy and governance: write the rule for agents that edit shared artifacts: a visible change log and a rollback path (01:07:47) before the feature ships. - License compliance: if you adopt a "powered by" label for AI output (the Product Talk label at 00:46:53), record who approved the use of that name. - Security management: treat uploaded user documents as untrusted inputs to an LLM; fake and malformed transcripts appeared in the beta (00:44:44). - Software inventory: log model provider, version and region as inventory fields; Sonnet versions change eval scores (01:15:44) and Bedrock caps tokens separately (01:14:15). - Developer support: document the escalation path when AI output is wrong, since the "clean up this branch" moment shows users fix errors silently. - Community engagement: no direct action from this source; nearest relevance is the host's community-first route into a vendor partnership (00:03:10). - Strategic alignment: add a KPI for time to first usable draft separate from draft correctness; the episode blurs them. #### Community - Maintainers: if you review AI-generated patches, add a repair-loop style pre-check (lint, test) before human review; no direct data from this source on maintainers. - Contributors: no direct action from this source; nearest relevance is the point that inputs of mixed quality (labeled "interviews" that weren't) need an intake check. - Sponsors: when a sponsored tool is built with an AI partner, publish what the sponsor gets (here, the terms aren't stated). - Ecosystem: share eval cases across projects the way Torres shares error modes, so one project's "too many children" case becomes another's test. #### WordPress - Core: no direct action from this source. - Core Performance: no direct action; nearest relevance is the cost point that a code assertion cuts LLM-judge calls (00:51:09). - Core AI: use the guardrail pattern as a model for AI features: an eval that also runs in production and triggers a targeted repair (00:54:47). Add an ID-stability requirement for anything an LLM edits. - **Core Program Team:** no direct action from this source; nearest relevance is cross-team ownership of a shared eval set. - Mobile: no direct action from this source. - Playground: build a demo of a change-set review with merge, move and reframe shown as semantic moves, not raw diffs (00:59:06). - Community: no direct action from this source. - Training: add the "competing with shallow synthesis" framing (00:23:48) to any lesson on AI-assisted research. - Marketing: no direct action; nearest relevance is the "powered by" attribution risk of readers assuming an expert endorsed each output (00:47:35). - Support: capture "I'll have to clean this up" statements as defect tickets, as Torres did. - Design: study the canvas-plus-agent pattern and the undo expectation it creates (01:07:12). - Documentation: document what an AI doc generator does with low-quality source input; the pyramid argument says the base layer sets the ceiling (00:41:50). - Photos: no direct action from this source. - TV: no direct action from this source. - Polyglots: no direct action; nearest relevance is that EU data residency drives where translation inference can run (01:13:30). - Accessibility: no direct action from this source. - Meta: no direct action from this source. - CLI: no direct action from this source. - Hosting: record data-residency options (US or EU) as a documented feature if AI services run on the platform. - Tide: no direct action from this source; nearest relevance is calibrating an automated judge against human labels before trusting its scores (00:51:53). - Openverse: no direct action; nearest relevance is that fake or mislabeled inputs appear at intake (00:44:44). - Themes: no direct action from this source. - Plugins: no direct action; nearest relevance is the same input-quality and calibration lessons for automated review. #### The WPOCC - Programs: no direct action from this source. - Projects: no direct action from this source; The WP Open Community Collective could host a shared AI-eval case library. - Funding experiments: if The WPOCC funds AI tooling, require the grantee to publish eval counts and error rates, which this episode doesn't. ### FAIR - Technical independence: the team's move from the direct Claude API to Bedrock for EU hosting (01:13:30) shows provider choice constrains features (01:14:15). Record that dependency in any FAIR-adjacent AI tooling. - Policies and covenants: require disclosure of partner relationships when a project promotes a vendor's AI product. - Productized governance artifacts: add model version, region and fallback fields to operator scorecards. ## 🧪 Hypotheses 1. A code-assertion pre-filter (a node has more than N children) cuts LLM-judge calls by at least 50% without missing nodes a human would flag, on a tree-quality eval set. 2. A repair loop that checks one error class and returns the output for correction produces a lower combined error rate (missing subgroups plus badly framed parents) than the best of 16 prompt-only variants. 3. A change set built from agent-logged semantic moves gets rated "makes sense" more often by users than a change set derived from a post-hoc diff of the same tree pair. 4. Removing the snapshot-review gate speeds time to first tree without raising the share of trees users later correct in the first week. 5. Scoring uploaded transcripts on a variation rubric (the 40-point system) separates LLM-generated transcripts from real interviews at over 90% precision on a labeled sample. 6. Prompts tuned on one model version score lower on the next version until migrated, by a margin large enough to change a ship decision (the Sonnet 4.6 versus Sonnet 5 claim at 01:15:44). ## 📊 Methodology - **H1.** Take 200 tree nodes with human labels for "needs grouping." Run the judge on all nodes and on the over-N subset. Compare cost, recall and precision, and report where the threshold N was set before looking at results. - **H2.** Hold the model and data fixed across arms: 16 prompt variants, one repair-loop variant. Measure both error rates per tree on 50 trees, use a paired test, and keep the seesaw metrics separate so a gain on one can't hide a loss on the other. - **H3.** Show 30 users two change sets for the same tree pair, in random order, and ask which matches what happened. Record accuracy of identified merges and time to decide. - **H4.** A/B test the flow on new signups, with first-week reframe and move events (the product already tracks them, 01:05:38) as the correction signal. Pre-register the success threshold. - **H5.** Build a labeled set of 100 real and 100 synthetic transcripts. Fit the rubric on half, evaluate on the other half, and report precision and recall. A rubric of 40 variation points needs a held-out check to avoid overfitting. - **H6.** Run the team's eval set on two model versions with identical prompts, then after prompt migration. Report the score gap and the migration effort in hours. - **Controls.** Fixed seeds where available, the same input trees, and a blind grader for human ratings. ## ❓ Questions - What error rates do the snapshot and tree steps have on a labeled set, and who labeled it? The "90% right" claim at 00:19:33 has no figure behind it. - What is the study behind "raises the floor for novices, raises the ceiling for experts" (00:25:59), and does it cover interview synthesis? - How was Torres's hand-versus-Claude comparison scored (00:24:32): how many interviews, which metrics, who judged? - How many of the beta transcripts were fake or non-interviews, and what share of trees do they affect (00:44:44)? - What does a user see when the product pushes back on weak evidence, and does it change behavior? - How do versioning and rollback work when an agent and several humans edit the same tree (01:07:47)? - What is the financial arrangement between Vistaly and Product Talk, and how does it shape the "powered by" claim? - Which Sonnet versions and eval scores sit behind the 4.6-versus-5 statement (01:15:44)? ## 🔮 Future research - **Compound error across layers.** The pyramid argument says a base-layer mistake spreads upward. Nobody in this source measures how fast. - **Implicit feedback as an eval signal.** Torres wants reframe and move events in their harness (01:05:38). Whether those events track quality or only habit is untested. - **Collaboration versus automation.** The speakers report users prefer automation (01:03:27) while hoping for collaboration. A study of when each helps learning would settle it. - **Teaching through software.** The claim that using opinionated tools teaches synthesis (00:47:35) needs a pre/post skill measure. - **Judge calibration with contaminated labels.** The calibration failure at 00:51:53 suggests a method for building label sets with one error type at a time. ## 💡 Implications - **Practice.** Build a cheap deterministic check next to every LLM judge. Add a repair loop when two prompt fixes fight each other. Log semantic moves when users must review changes. - **Policy.** Name partner relationships when an interview doubles as a launch, and keep quality claims tied to numbers. - **Security.** Uploaded content is untrusted input. The beta contained LLM-generated and non-interview files, and data residency and deletion are table stakes for enterprise buyers (01:11:17). - **Education.** The "shallow synthesis" framing shows how product design can teach a method, with the risk that users assume the tool's output is the expert's own (00:47:35). - **Funding.** AI rebuilds can cut build time (2.5 months to alpha, per the speakers), which shifts cost toward quality work: evals, compliance, UX for comprehension. Budget for those, not only for generation. ## 🧠 Learning styles assessment - **Reading level estimate.** Flesch-Kincaid Grade Level 7.5 and Flesch Reading Ease 73.7 on the transcript body (15,352 words, 836 sentences, about 18 words per sentence; computed with a heuristic syllable counter, so treat as plus or minus one grade). Conversational English, but terms such as LLM-as-judge, repair loop, change set and Bedrock push effective difficulty to intermediate practitioners. #### 👀 Visual learners - Present: the host says at 00:08:01 that they'll add an OST visual, and the chapter list gives a map. The sections on tree edits and canvases describe screen visuals that the transcript can't confirm. - Improve: add a diagram of the four-layer pyramid, the seesaw between missing subgroups and badly framed parents, and a before/after change set. A tree with the "too many children" node highlighted would anchor the eval story. #### 🔊 Aural learners - Present: full conversational audio, available on Spotify and Apple Podcasts per the description. Speakers repeat key points in different words. - Improve: label speakers audibly at each turn; the transcript has none, and overlapping exchanges blur who is claiming what. #### 🗣️ Verbal learners - Present: the host defines opportunity, LLM-as-judge and repair loop in plain sentences (00:08:01, 00:51:53, 00:54:02). - Improve: define change set, provenance, MCP and Bedrock on first use. A written glossary in the description would help. #### 🧩 Kinesthetic learners - Present: a concrete recipe for a first eval: assert on child count, then judge only the flagged nodes. - Improve: link the evals guide and sample data so viewers can repeat the calibration exercise. #### 🧠 Logical learners - Present: clear causal chains: customer remark, error mode, measure, experiments, orchestration fix. - Improve: show the experiment table (16 variants, four evals) so the logic can be audited. #### 🧑‍🤝‍🧑 Social learners - Present: four voices in dialogue, plus mention of the Continuous Discovery Habits community. - Improve: add a discussion prompt per chapter and a community thread for the eval examples. #### 🧘‍♀️ Solitary learners - Present: 25 chapters let a solo viewer jump to the section that matters. - Improve: end with two or three reflection questions, such as "which of your AI features has an untested base layer?" #### 🌿 Naturalistic learners - Present: the pyramid and house-of-cards images are structural analogies, not nature ones. - Improve: no ecosystem content here. A systems framing (errors propagating through a food-web-like chain) could connect the layered-analysis idea to this group. ## 📊 Accessibility assessment #### Screen reader semantics - Present: the YouTube page supplies landmarks and player controls. The description holds links and a chapter list. - Improve: the OST visual, UI demos and any on-screen text aren't described in the audio or the description. WCAG 1.1.1 (Non-text Content) applies to the visuals the host promises at 00:08:01. #### Keyboard navigation - Present: the player is keyboard operable through the platform, and chapters give jump targets. - Improve: nothing in the content adds a barrier. 2.1.1 passes via the platform, not via the author. #### Contrast and readability - Present: unverifiable from a transcript; slides or UI demos may exist. - Improve: check on-screen text against 1.4.3 (4.5:1) and show UI at readable size when walking through trees. #### Text resizing and reflow - Present: caption size is user-adjustable in the player. - Improve: any burned-in text can't reflow, which engages 1.4.4 and 1.4.10. Prefer captions over baked-in text. #### Media and captions - Present: auto-captions exist, and the description lists 25 chapters. - Improve: auto-captions garble the company name four ways (Vistili, Vistily, Vistil, Vistley), the host and a co-founder (Teresa Taus, CP Daily), Jira (Jerry), Zencity, evals (emails), and model names (Sonnet 46). That fails the accuracy intent of 1.2.2 (Captions, Prerecorded). There's no audio description for visuals (1.2.5) and no speaker labels. A corrected transcript is the fix. #### Forms and interactions - Present: none in the video. The CTA is to subscribe and rate. - Improve: no action; the Vistaly and Product Talk sites would need their own review against 3.3.2. #### Standards reference - Gaps against WCAG 2.2 AA: **1.2.2** (caption accuracy), **1.2.5** (audio description for visuals), **1.1.1** (promised visuals not described), **1.3.1** (no speaker labels in the transcript), **1.4.2** (opening music; confirm it doesn't overlap speech). Chapters help **2.4.5**-style navigation. Highest-impact fix: human-corrected captions with speaker names. ## 🌍 DEIB assessment #### Cultural breadth - Present: the privacy section covers the US and EU and names GDPR, HIPAA, SOC 2 and ISO (01:12:04). Customers are described as product teams in general. - Improve: no non-English interview examples appear, though the product's core input is customer speech. Say whether it handles other languages. #### Inclusive language - Present: no exclusionary terms appear, and the tone is respectful. - Improve: idioms and violent metaphors raise comprehension load for non-native listeners: "burn the boats" and "burn the ships" (00:27:25, 00:31:45), "a little dagger to me" (00:41:50), "death by a thousand paper cuts" (00:33:56), "crapshoot" (00:40:24). Ability-tinged words appear too: "dumb chatbot" (01:09:57), "insane" (01:12:04), "crazy" (00:23:06). #### Neurodiversity support - Present: 25 chapters, a predictable arc from origin to V1, V2, evals and next steps, and repeated summaries. - Improve: 80 minutes of crosstalk without speaker labels is a tracking burden. Add on-screen name captions on speaker changes. #### Representation - Present: four named speakers, with a host who is also an interested party. - Improve: the transcript says nothing about identities, so none are inferred here. Customers speak only second-hand (the "clean up this branch" quote), and no independent critic or customer appears. #### Psychological safety - Present: the speakers correct each other politely ("Fair correction there," 00:49:43), laugh at their own mistakes, and name their fears about the "Teresa's stamp of approval" effect (00:47:35). - Improve: the transcript reports a customer team that wouldn't edit its tree because it believed it was an expert's version. Product copy should state that edits are expected, and the show could say so too. ## ✅ Logical validity #### Strong reasoning - **Causal chain with a measurement step, 00:41:50 to 00:55:29.** A customer remark becomes an error mode (too many children), then a code assertion measures it, an LLM judge is calibrated against human labels, experiments run, and the fix is verified as an eval. The chain is checkable at each link. To strengthen: publish the counts. - **Diagnosis of a failed approach, 00:53:19 to 00:54:47.** Sixteen prompt variants that seesawed show prompts alone can't balance the two errors, which justifies the orchestration change. The inference follows from the described experiments, though the experiments themselves aren't shown. - **Counterexample that kills a simple design, 00:58:24 to 00:59:06.** The same input and output trees map to more than one valid change set, so a diff can't recover the meaning of a merge. This is a clean argument by counterexample. - **Reframing the competitor, 00:23:48.** Comparing against what users do today (shallow synthesis) instead of an ideal is a useful basis for evaluating the product. It rests on Torres's coaching experience, not on a measurement. - **Candor about reversals, 00:18:07 to 00:19:33, 01:02:43.** The speakers say they reverted a chat UI and removed two review gates, which gives the later "people want the answer" claim a trail of prior evidence. #### Fallacies - **Unsupported quantification, 00:19:33.** "that we think is 90% right" has no sample, metric or date. Fix: state what was measured and on which set. - **Anecdote presented as a pattern, 00:37:31.** "What we're learning over and over again is the team just wants the AI to do it" comes from a small early beta and calls. Fix: report how many teams, and the share who edited output later. - **Appeal to an unnamed study, 00:25:59.** The novice-floor, expert-ceiling finding is cited as "this research study" without a title. Fix: name it, and say whether it covers synthesis tasks. - **False dilemma, 01:09:57.** "Either a dumb chatbot... or an expert chatbot" leaves out a scoped bot that states its limits. Torres asks the right follow-up in the same breath (how to avoid looking dumb), so the point is more rhetorical than faulty. Fix: name the middle option. - **Undisclosed interest handled partly, 00:00:46 and 00:46:53.** The partnership is stated, but the commercial terms and the "powered by Product Talk" label aren't examined as a conflict while quality is discussed. Fix: say what each side gains. - **Unmeasured model comparison, 01:15:44.** "My eval scores on Sonnet 46 are way better than on Sonnet 5" lacks scores. Fix: give the deltas. #### Tone of voice - Conversational and candid, with a practitioner's bluntness: "a hot mess" (00:57:41), "that was a little dagger to me" (00:41:50), "LM calls all the way down" (01:00:33). The founders are enthusiastic about their rewrite, and the host shifts between interviewer, partner and builder. The second half is more concrete and self-critical than the first, which carries the origin story and the community-goodwill framing. ## 🛡️ DARVO analysis Not applicable to this source. DARVO (Deny, Attack, Reverse Victim and Offender) describes a response to accusation, and this is a product interview with no accuser. - **Deny:** nothing is denied. The speakers volunteer failures: the slow chat, the beta that "barely understood" change sets, and the fake-transcript inputs (00:59:50, 00:44:44). - **Attack:** no one is attacked. Customers who upload poor interviews are described with mild humor and sympathy ("the customer is judging us," 00:46:10), and the speakers treat that as a design problem. - **Reverse victim and offender:** no such move appears. The closest case is the concern at 00:47:35 that Torres's name makes people trust output, and the speakers treat it as a risk to manage. ## 🎯 Conclusion The lasting content is a set of failure patterns and fixes, each tied to a timestamp: a pyramid where the bottom layer caps quality (00:43:16), a code check ahead of an LLM judge (00:51:09), a repair loop where prompts could not balance two errors (00:54:02), agent-logged moves for change sets (00:59:06), and ID control by deterministic code (00:57:41). Those transfer to any LLM product that edits structured data. The quality and speed claims stay unmeasured. The audio gives no error rates, no eval scores and no customer counts, and the episode is a partner interview in which the host builds the services under discussion. The product's ground-up rebuild in 2.5 months is a report, not a result you can check. **Next steps for readers:** pick one LLM workflow you own, write the cheapest deterministic check for its most common error, and run your judge only on flagged cases. Before citing the Vistaly numbers or the novice-versus-expert study, ask Product Talk for the underlying data. Confirm names and model versions against the video description, because the auto-captions mangle most of them. ## 🔄 Refactor Appendix — Opportunity Solution Trees Built by AI: Inside Vistaly's Rebuild > Density: 2–5 Literature atoms (H3 pad), 0–3 Permanent candidates (H4 pad). > Emitted: 5 Literature, 3 Permanent. Resonance filter applied: the model-selection and > privacy claims were kept in Notes only because they carry no numbers. > > Konik claim-header rules: present tense, declarative, ≤10 words, searchable, unique. > > Atomicity: one declarative sentence per atom. "and"/"also" → split. > > Tag pool: #bmf · #addendum · #xref · #articleseed · #storystem · #share · #aphorisms ### 📚 Literature Split Pad #### [H1] Base-layer quality caps every layer above it > [!quote] [00:43:16] > different scopes that you work at and the different layers of analysis and how they connect to each other. **Context & use**: - 00:43:16, right after Torres says pipelines and agents are the easy part of AI products. - This is the mechanism behind the "house of cards" (00:38:57). Keep it with the caveat that no error-propagation numbers are given. **Tags**: #xref #articleseed  #### [H2] LLMs corrupt IDs unless code constrains them > [!quote] [00:57:41] > Like agents, the LLM would randomly change IDs. It would drop IDs. It would put the string of the opportunity in the ID field. **Context & use**: - 00:57:41, the beta failure after a prototype built on toy data with manufactured IDs. - Self-reported by the builder; the fix (deterministic code and a repair loop) appears at 01:00:33 without error counts. **Tags**: #xref #share  #### [H3] Agents need semantic moves to produce change sets > [!quote] [00:59:06] > I have to get the agent. I need to teach the agent a game. As you build this tree, you're playing a game. **Context & use**: - 00:59:06, after the 00:58:24 finding that one tree pair can yield several change sets. - The design principle: log meaningful moves as the agent works instead of reconstructing intent from a diff. **Tags**: #xref #addendum  #### [H4] The competitor is shallow synthesis > [!quote] [00:23:48] > And so we're not competing with making good synthesis faster. We're competing with make shallow synthesis better. **Context & use**: - 00:23:48, Torres reframing the product goal after the Claude comparison. - Rests on coaching experience; keep it as a positioning claim, not a finding. **Tags**: #xref #aphorisms  #### [H5] An eval becomes a production guardrail > [!quote] [00:54:47] > And so basically my eval became a guardrail. We run it in production. It looks for the error. **Context & use**: - 00:54:47, closing the seesaw story after 16 prompt variants failed. - Pairs with the repair loop at 00:54:02; the broader point is to move a failing check from offline to inline. **Tags**: #xref #articleseed  ### 🧠 Atomic Split Pad #### [P1] Layer quality is a ceiling **Claim** (one sentence, Courtney's voice): In a layered AI pipeline, the weakest base layer sets the quality ceiling for every layer built on it, so I measure the base first. **Evidence pointer**: [[#H1]] **Scope & conditions**: Holds when upper layers consume lower outputs without re-checking raw data. It weakens when each layer re-verifies against the source. The only support here is one team's account, not a measured propagation rate. **Links**: - [[AI evals]] — because extends - [[Opportunity solution trees]] — because applies-in tree generation from interviews - [[Reading vendor claims]] — because contrasts on evidence: mechanism stated, rates absent  #### [P2] Orchestration beats prompt tuning when fixes conflict **Claim** (one sentence, Courtney's voice): When two prompt fixes undo each other, I move one check into orchestration, such as a repair loop, instead of writing a longer prompt. **Evidence pointer**: [[#H5]] **Scope & conditions**: Based on one reported case: four evals and 16 variants. It holds when the error is checkable by a tool. It doesn't hold for errors with no deterministic check. **Links**: - [[Repair loops for LLM output]] — because extends - [[Evals as guardrails]] — because evidence-for - [[LLM-as-judge calibration]] — because applies-in cheap pre-filters before judges  #### [P3] Log the moves, not the diff **Claim** (one sentence, Courtney's voice): When users must review agent edits, I have the agent log named moves, because a diff can't tell a merge from a delete. **Evidence pointer**: [[#H3]] · [[#H2]] **Scope & conditions**: Holds for structured data where edits carry meaning. Needs deterministic code to confirm the moves reproduce the output. Doesn't help if the move vocabulary is too coarse for the domain. **Links**: - [[Change sets and versioning]] — because extends - [[Agent edits on shared artifacts]] — because applies-in rollback and review - [[Structured output with schema validation]] — because contrasts on where meaning is enforced  ### ✏️ Fleeting Queue - [F1] Confirm spellings from the description: Vistaly (captioned Vistili, Vistily, Vistil, Vistley), Teresa Torres (Teresa Taus), CP Dehli (CP Daily), Zencity, Jira (Jerry) — https://www.youtube.com/watch?v=zverZHvdpPo #fleeting #inbox - [F2] Find the novice-floor, expert-ceiling study Torres cites at 00:25:59 and check whether it covers synthesis tasks #fleeting #inbox - [F3] Open question: how does a team measure whether a tree "looks like truth" when input interviews are weak (00:46:53)? #fleeting #inbox - [F4] Read Torres's evals guide and the "clean up this branch" post on producttalk.org and compare to the 00:50:12 telling #fleeting #inbox - [F5] Article seed: "A customer quote is a bug report you can turn into an eval" #fleeting #inbox #articleseed ### → Things 3 (max 2 actions) - [ ] Confirm product, person and model-name spellings against the video description before citing this episode — research, 10 min #things - [ ] Add a code-assertion pre-filter ahead of the LLM judge in one current eval and log the cost change — AI evals #things  📄 **Full text:** [[Full Document Contents/Articles/Opportunity Solution Trees Built by AI Inside Vistaly's Rebuild.md|Opportunity Solution Trees Built by AI: Inside Vistaly's Rebuild by Product Talk (Teresa Torres) with Matt O'Connell, CP Dehli and Steve Klein of Vistaly]] ## Highlights - [00:23:06] they were doing before. But >> that's what I was going to say. What's crazy to me about this is that even just what you described sounds pretty great to me because >> interview synthesis is really hard and you're going to help me do it. Awesome. >> One thing I've started to realize about this problem and the more work we do on this, the more I'm realizing this is true. as a discovery coach like I I want to believe that everybody conducts an interview and then takes the time to do good synthesis and then does that three times and then takes the time to map on the opportunity space well. But I think what happens in reality is people do the interviews. They sit down to do the synthesis. It's really hard and they give up and they just say, "I'm going to pull out the two or three things that were really salient and I'm going to put [00:23:48] those on my tree." And so they're not really doing the synthesis work. They're just taking the most salient things that they remember. And so we're not competing with making good synthesis faster. We're competing with make shallow synthesis better. >> Yep. >> I think that that's a hard thing to compete with, right? So I know I initially framed it as, oh, okay, let's take what a researcher would do and help a product team do that. And I think my takeaway from all of our experiments so far is that a product team is never going to put in that much effort into synthesis and maybe we should just do it for them. >> Yeah. You I think you wrestled with this question for a long time around is this ([View Highlight](https://read.readwise.io/read/01m3yd0sw5rcayrkg89p1h1twc)) - Note: [H4] The competitor is shallow synthesis - Tags: atom/literature/h3 - [00:43:16] >> the part that's hard is the like different scopes that you work at and the different layers of analysis and how they connect to each other. >> Yeah. >> I like to me it doesn't matter if >> an interview snapshot is a pipeline or an agent, >> right? It matters that like you're using the orchestration pattern that gets you the quality that you need so that when you do the next layer of analysis, it doesn't all fall apart. And then when you do the next layer of analysis, that doesn't all fall apart. >> Yeah. I think it's so funny. I think one of the hardest problems is for us is a lot of people just don't run good interviews in the first place. And so it's like people sometimes upload interviews that have leading questions or someone saying, "Would you like XYZ thing?" Right? Classic kind of not great [00:44:01] product management questions. And so when that's the content that we have to go off of, how are we supposed to take that and take and create a a good opportunity solution tree out of it if like the basis layers of input are problematic, but it opens the door for >> an interview coach. >> Yeah, I was say it opens the door for >> interview coach. Yeah, >> this is where I think all these pieces are really interconnected. I on my other podcast all things product we Petra and I did a episode about the quality of evidence and this came out of our work with Vistily like I I've worked with a lot of product teams like I know teams are not great at interviewing but when you get to see like we did in the beta transcripts from over a hundred ([View Highlight](https://read.readwise.io/read/01m3yd0hs42dwzezkjjkzw3ph3)) - Note: [H1] Base-layer quality caps every layer above it - Tags: atom/literature/h1 - [00:54:47] write the prompts to be really targeted towards well-framed parents. So I could keep the side of the seessaw that solved the well-framed parents. And then when the repair loop triggered, it was like a much smaller problem for the agent to just figure out subgroupings and then let a different process reframe the parent. >> And so it really it just required an orchestration change. It's funny to me. This is wasn't the first time I encountered this, but like it's so easy to think you can just fix everything with prompt changes. >> Yeah. >> And then you like bang your head against the wall and you're like, "Okay, I need a different approach than just prompt changes." And so basically my eval became a guardrail. We run it in production. It looks for the error. If [00:55:29] the error is present, we tackle that specifically. >> Yeah. Speaking of guardrails, one one thing that I had written down that I think is a really interesting problem is so much of all of these when you're updating a tree for the listeners on the back end, we create these big chain change sets which basically says, hey, given this is what the tree was when the person wanted to synthesize these new interviews into it, what should we do to update the tree? And so that has to keep track of so many different cards and card IDs and all all this stuff that's very programmatically has to be exactly correct or it'll fail on our end. anything interesting that you've learned ([View Highlight](https://read.readwise.io/read/01m3yd0x57j9cydtyra5y2drx7)) - Note: [H5] An eval becomes a production guardrail - Tags: atom/literature/h3 - [00:57:41] overlooked this like what I would call just like ba basic computer science like data management coding but I just swept under their rug and was like whatever it'll just work. And we learned pretty quickly it didn't just work. Like agents, the LLM would randomly change IDs. It would drop IDs. It would put the string of the opportunity in the ID field. Like it was just a hot mess because LLM's make mistakes. And then [snorts] I think pretty early on we had this conversation about going back to our conversation about getting the user to collaborate with the agent. We had I think we were inspired by like git diff. can we show the user what changed when you updated the tree? >> And Matt was like, we're really thinking you should just give us change sets. [00:58:24] >> I'm like, okay, sure. I'll give you change sets. >> I didn't know what would be required to do that. And I just thought, okay, we'll take the input tree and we'll compare it to the output tree and we'll generate a change set. And this is literally how my code worked in the beginning. Like I just told Claude, I was like, here's the input tree, here's the output tree, let's write a function that generates a change set. And like that's what went into my code. And we learned really quickly that the same input output tree pair can generate more than one change set. There's more than one change set that can produce that output tree. And the problem with this is we're talking about semantic tasks like this opportunity ended up merged with this other ([View Highlight](https://read.readwise.io/read/01m3yd0mh92fs436c619av45c0)) - Note: [H2] LLMs corrupt IDs unless code constrains them - Tags: atom/literature/h2 - [00:59:06] opportunity. There was a semantic reason for that. But it looks like we just removed an opportunity and then moved some sources around. And in the change set, we could present that as a merge, which has semantic meaning, or we could present it as a delete with some sources being reassigned. And if you present that latter one to the user, it doesn't make any sense. Like the semantic merge is what matters. >> And so I realized, oh, I can't just do this as a diff at the end. I have to get the agent. I need to teach the agent a game. As you build this tree, you're playing a game. And these are the rules of the game. You can make these semantic moves that have meaning. And as you make these moves, log them so that I can [00:59:50] produce an accurate change set at the end. And this required like I literally had to rebuild both services from the ground up. like we launched in beta with this prototype that barely understood change sets. And then two months later, I spent a month rewriting everything from the ground up to come up with the rules of the game to teach the agent the rules of the game to teach the agent to show its work so that at the end we could get reliable change sets. And then I also had to like just get way better about how do I use deterministic code to guarantee the change set and to make sure that if I start with the input tree run the change set it generates the output tree and more importantly it generates the output tree in a way that ([View Highlight](https://read.readwise.io/read/01m3yd0q3y2ns7j5k0sbvwasw0)) - Note: [H3] Agents need semantic moves to produce change sets - Tags: atom/literature/h3