The prompting playbook - CourtneyR.dev's 2nd Brain

The prompting playbook ![rw-book-cover](https://readwise-assets.s3.amazonaws.com/static/images/article1.be68295a7e40.png) 🔗 **Source:** [The prompting playbook](https://www.youtube.com/watch?v=G2B0YWuJUgI) > [!note]+ Document Notes > tags: [prompt-engineering, llm-optimization, model-evaluation, agentic-systems, ai-debugging, claude-prompting] > author: Margot Vanlar > source: Anthropic > reference: https://www.youtube.com/watch?v=G2B0YWuJUgI > date created: {{date:YYYY-MM-DD}} {{time:HH:mm}} > > # 🏷️ Title: The prompting playbook >  ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ## 📝 Summary - Anthropic Applied AI Engineer Margot Vanlar presents best practices for prompt engineering across two scenarios: maintaining existing prompts in production and building new agentic systems from scratch. - Demonstrates systematic debugging of a customer support bot prompt through iterative improvements: structural cleanup, removing redundant instructions, adding tools for calculations, and balancing trade-offs. - Shows how different model choices (Sonnet vs. Opus) and architectures (single prompt vs. agentic loops) impact performance, cost, and latency. - Emphasizes evaluations as the foundation for rigorous prompt optimization, using both programmatic checks and LLM judges. - Key stakeholders include AI engineers, DevRel teams building on Claude, and organizations deploying LLM-powered customer service. ## 📖 Main Content This presentation covers practical prompt engineering for production systems, focusing on two common scenarios engineers face when working with large language models. ### Core Framework: Evaluation-Driven Development The foundation of effective prompt engineering is rigorous evaluation. Before making any changes, teams need: - **Control cases**: Unambiguous scenarios the model should always handle correctly - **Edge cases**: Previous failure modes with targeted fixes - **Boundary testing**: Cases where the model should escalate to humans or refuse requests ### Scenario 1: Debugging Production Prompts Using a Meridian Mobile customer support bot as an example, the talk demonstrates systematic prompt debugging: **Initial State**: A complex prompt built by multiple contributors over time, mixing policy, tone, and patches for previous models. Performance degraded after model migration. **Step 1: General Hygiene** - Added XML structure to separate role definition, guidelines, policy, and tone - Removed redundant information (references to "hero images" and cookies copied from websites) - Created clear output contracts with stop sequences **Step 2: Targeted Failure Mode Resolution** *Information Withholding*: The model was deflecting questions about grandfathered plans instead of using available customer data. Root cause was an overly defensive instruction: "Never give customer wrong plan details, instead point them to URL." Solution involved updating the instruction to trust customer-specific data as the authoritative source. *Calculation Accuracy*: Instead of telling the model "always calculate prorated amounts correctly," they provided a calculation tool. This illustrates the principle that "instructions don't add capability." *Over-Optimization*: The model avoided escalating billing errors due to cost-focused instructions ($8 escalation cost). The fix involved presenting both sides of the trade-off (escalation cost vs. refund cost and customer trust loss). ### Scenario 2: Building New Agentic Systems For a retail staff scheduling agent, the presentation compared different approaches: **Single Prompt Approaches**: - Simple prompt + Sonnet 3.5: All test cases failed - Simple prompt + Opus: Fewer violations but still failing - Opus + adaptive thinking: Reliable compliance but 3x token cost and latency - Sonnet + enhanced prompt: Partial success but token/latency inefficient **Agentic Approach**: Generate-Evaluate-Repair loop with three specialized prompts: 1. Generator creates initial schedule 2. Evaluator identifies specific violations with evidence 3. Repairer makes targeted fixes This approach achieved full compliance with lower token usage and latency than single-prompt solutions, plus runtime flexibility for soft constraints. ## 🗒️ Notes #### ⏱️ [00:01:30](https://www.youtube.com/watch?v=G2B0YWuJUgI&t=90) Introduction and Scenarios - Two main scenarios: maintaining existing prompts and building new agentic systems - Prompts in production often accumulate complexity from multiple contributors without clear ownership #### ⏱️ [00:04:15](https://www.youtube.com/watch?v=G2B0YWuJUgI&t=255) Evaluation Framework - Need control cases, edge cases, and boundary testing - Model migration can fail due to behavioral differences (fixable with prompting) or capability gaps (not fixable) - Demo uses 5 test cases for Meridian Mobile customer support bot #### ⏱️ [00:08:45](https://www.youtube.com/watch?v=G2B0YWuJUgI&t=525) Initial Prompt Analysis - Original prompt claimed bot was "human" and contained website copy artifacts - Instructions mixed together without clear structure between policy, guidelines, and tone #### ⏱️ [00:11:20](https://www.youtube.com/watch?v=G2B0YWuJUgI&t=680) Structural Improvements - Added XML tags for role, guidelines, policy, tone separation - Rule of thumb: "If you can't tell guidelines from policy from data, most likely the model can't either" - Immediate performance improvement on prepaid scenario test case #### ⏱️ [00:14:30](https://www.youtube.com/watch?v=G2B0YWuJUgI&t=870) Output Contract Best Practices - Define clear output format using XML tags - Add stop sequences to API calls for consistency - Consider structured outputs for complex schemas #### ⏱️ [00:17:45](https://www.youtube.com/watch?v=G2B0YWuJUgI&t=1065) Hotspot Data Failure Analysis - Model deflected grandfathered plan questions despite having customer data - Root cause: Overly defensive instruction was a "patch" for previous model behavior - Modern models better at instruction following, making some patches redundant #### ⏱️ [00:20:15](https://www.youtube.com/watch?v=G2B0YWuJUgI&t=1215) Version Control for Defensive Changes - Track reasons for defensive prompt changes - Previous patches can cause unwanted effects with newer models - Models can withhold information they have access to, not just hallucinate #### ⏱️ [00:22:30](https://www.youtube.com/watch?v=G2B0YWuJUgI&t=1350) Proration Calculation Problem - Model attempted mental math instead of giving concrete answers - Original instruction "always calculate prorated amounts correctly" insufficient - Solution: Provided calculation tool with proper schema and implementation #### ⏱️ [00:25:45](https://www.youtube.com/watch?v=G2B0YWuJUgI&t=1545) Instructions vs Capability - Key principle: "Instructions don't add capability" - Telling model calculations are "critical" doesn't improve math ability - Correct approach: Give tools for reliable execution, let model reason over harder problems #### ⏱️ [00:28:00](https://www.youtube.com/watch?v=G2B0YWuJUgI&t=1680) Escalation Over-Optimization - Model avoided escalating billing errors due to cost-focused instructions - Only presented one side of trade-off ($8 escalation cost) - Fix: Present both costs (escalation) and benefits (avoiding refunds and customer trust loss) #### ⏱️ [00:30:30](https://www.youtube.com/watch?v=G2B0YWuJUgI&t=1830) New Agent Development - Retail Scheduling - Problem: Create week-long retail schedule with employee availability and constraints - Hard rules allow programmatic validation instead of LLM judges - Test different combinations of prompts, models, and architectures #### ⏱️ [00:33:15](https://www.youtube.com/watch?v=G2B0YWuJUgI&t=1995) Model Comparison Results - Simple prompt + Sonnet 3.5: All cases failed - Simple prompt + Opus: Fewer violations, better reasoning capability - Opus + adaptive thinking: Reliable compliance but 3x tokens and latency #### ⏱️ [00:36:45](https://www.youtube.com/watch?v=G2B0YWuJUgI&t=2205) Enhanced Prompt Approach - Added reasoning instructions and "check your work" directive to Sonnet - Passed 2/5 cases but hit output limits - Higher token usage and latency than other approaches #### ⏱️ [00:38:30](https://www.youtube.com/watch?v=G2B0YWuJUgI&t=2310) Agentic Generate-Evaluate-Repair Loop - Three specialized prompts instead of one complex prompt - Generator creates initial schedule, Evaluator finds violations, Repairer fixes issues - Achieved full compliance with lower tokens and latency - Allows runtime soft constraints without code changes #### ⏱️ [00:41:45](https://www.youtube.com/watch?v=G2B0YWuJUgI&t=2505) Key Takeaways - General hygiene provides immediate uplift - Target failure modes systematically with evaluations - Consider breaking complex tasks into specialized prompt systems - Balance model choice, prompt optimization, and architectural approaches ## 📚 References - [Anthropic Claude API Documentation](https://docs.anthropic.com/) - [Structured Outputs Documentation](https://docs.anthropic.com/en/docs/build-with-claude/tool-use) ## 🗣️ Paraphrase Margot Vanlar from Anthropic demonstrates a systematic approach to prompt engineering through two practical scenarios. The first involves debugging an existing customer support bot where performance degraded after model migration. She shows how structural improvements (XML organization), removing redundant patches from previous models, providing tools instead of just instructions, and balancing trade-offs in guidelines can systematically resolve failure modes. The second scenario covers building a retail scheduling agent from scratch, comparing single-prompt approaches across different models with an agentic generate-evaluate-repair architecture. The key insight is that rigorous evaluations enable systematic optimization, and breaking complex tasks into specialized prompts often outperforms monolithic approaches in both performance and maintainability. ## 📰 Newsletter **The Prompting Playbook: From Debugging to Deployment** Anthropic's Margot Vanlar just dropped some serious knowledge on systematic prompt engineering that every WordPress developer building AI features should bookmark. Her talk breaks down two scenarios we all face: fixing that prompt that mysteriously broke after a model update, and building new AI-powered features from scratch. The debugging approach is pure gold—start with evaluation suites (control cases, edge cases, boundary testing), then apply "general hygiene" by structuring prompts with XML tags and removing redundant instructions. Her rule of thumb? "If you can't tell guidelines from policy from data, most likely the model can't either." The real gem is her insight that "instructions don't add capability." Instead of telling Claude to "calculate correctly," give it a calculation tool. Instead of just saying "escalate when necessary," explain both the costs AND benefits of escalation decisions. For complex tasks like scheduling or content generation workflows, her generate-evaluate-repair pattern is brilliant—break one complex prompt into three specialized ones. Often performs better with lower token costs than trying to do everything in a single prompt. Check out the full presentation for the live debugging session and practical examples that'll level up your Claude integrations. ## ✅ Action Items #### DevRel - Developer education: Create workshop curriculum on systematic prompt debugging techniques and evaluation-driven development - Documentation: Build practical guide showing XML structuring patterns and output contract best practices - Developer experience: Develop templates for common prompt patterns (customer support, content generation, data analysis) - Community management: Host office hours for prompt engineering troubleshooting and peer review #### WordPress - Core AI: Implement evaluation frameworks for WordPress AI features before prompt changes reach production - Training: Develop lesson plans covering the generate-evaluate-repair pattern for complex WordPress workflows - Documentation: Create DevHub examples showing proper tool integration for calculations, data validation, and external API calls - Plugins: Establish review guidelines for AI-powered plugins focusing on prompt structure and capability boundaries #### Community - Contributors: Set up pairing sessions for prompt optimization techniques and failure mode analysis - Maintainers: Create governance templates for tracking prompt changes and their rationale (version control for defensive changes) #### OSPO - Developer Support: Provide consultation hours for teams migrating AI systems to new models or architectures - Training: Establish prompt engineering best practices documentation with real-world debugging examples ## 🧪 Hypotheses - Structural prompt organization using XML tags improves model performance more than content changes alone - Models trained for instruction-following may over-optimize for defensive patches designed for previous model generations - Agentic architectures with specialized prompts achieve better cost-latency-performance trade-offs than monolithic prompts for complex reasoning tasks - Evaluation suites with control/edge/boundary cases enable more systematic prompt optimization than ad hoc testing - Tool-augmented prompts outperform instruction-only approaches for tasks requiring reliable calculations or external data access ### 📊 Methodology **Hypothesis 1**: A/B test prompt variants with and without XML structural tags, measuring task completion rates across control and edge cases. **Hypothesis 2**: Historical analysis of prompt patches across model versions, tracking which defensive instructions become counterproductive with newer models. **Hypothesis 3**: Performance benchmarking comparing single complex prompts vs. generate-evaluate-repair loops across multiple task domains, measuring accuracy, token usage, and latency. **Hypothesis 4**: Longitudinal study tracking prompt optimization velocity and success rates with systematic evaluation frameworks vs. manual testing approaches. **Hypothesis 5**: Controlled experiments comparing instruction-only vs. tool-augmented approaches for mathematical, data retrieval, and validation tasks. ### ❓ Questions - How do we systematically identify which defensive prompt patches become counterproductive as models improve? - What evaluation framework design patterns work best for different types of AI applications (conversational, analytical, creative)? - When should teams choose agentic architectures over single-prompt solutions, and what are the decision criteria? - How can version control practices for prompts integrate with existing software development workflows? - What metrics best capture the trade-offs between prompt complexity, model capability, and system reliability? ### 🔮 Future Research - Develop automated tools for detecting redundant or conflicting instructions within complex prompts - Create standardized evaluation frameworks for common AI use cases (customer support, content generation, data analysis) - Research optimal prompt decomposition strategies for different types of reasoning tasks - Study the evolution of prompt effectiveness across model generations to predict when updates are needed - Investigate meta-learning approaches for prompts that can adapt their structure based on task performance ### 💡 Implications **Practice**: Teams need systematic evaluation frameworks before optimizing prompts, and should regularly audit defensive instructions that may become counterproductive with model improvements. **Policy**: Organizations should establish version control practices for prompt changes, documenting the rationale for defensive modifications to enable future cleanup. **Education**: Prompt engineering training should emphasize structural organization, tool integration, and agentic decomposition as core skills beyond basic instruction writing. **Tooling**: Development platforms need better support for prompt versioning, A/B testing, and evaluation suite management to enable systematic optimization workflows. ### 🧠 Learning Styles Assessment Reading level estimate: Flesch-Kincaid Grade 12 (technical presentation with domain-specific terminology but clear explanations) #### 👀 Visual learners Present: Live demo interface, diff views of prompt changes, performance comparison charts Improve: Architecture diagrams for agentic systems, flowcharts for debugging methodology, before/after prompt structure visuals #### 🔊 Aural learners Present: Live presentation format with clear verbal explanations Improve: Audio summary highlighting key debugging steps, TTS-friendly transcript with pronunciation guides for technical terms #### 🗣️ Verbal learners Present: Clear definitions of terms like "defensive instructions," step-by-step explanations of debugging process Improve: Glossary of prompt engineering terminology, analogy-driven explanations of model behavior #### 🧩 Kinesthetic learners Present: Live coding demonstration, interactive debugging session Improve: Hands-on lab exercises for prompt optimization, step-by-step checklists for debugging workflows #### 🧠 Logical learners Present: Systematic debugging methodology, clear cause-effect relationships for prompt changes Improve: Decision trees for choosing between architectural approaches, structured evaluation criteria #### 🧑‍🤝‍🧑 Social learners Present: Conference presentation format with shared learning experience Improve: Discussion prompts for prompt review sessions, collaborative debugging workflows #### 🧘‍♀️ Solitary learners Present: Self-contained examples and principles Improve: Reflection questions on current prompt practices, self-assessment rubrics for prompt quality #### 🌿 Naturalistic learners Present: Organic evolution of prompts over time, natural language interaction patterns Improve: Ecosystem thinking about prompt interactions, evolution metaphors for model capability growth ### 📊 Accessibility Assessment #### Screen reader semantics Present: Clear headings and structure in transcript Improve: Alt text for demo screenshots, ARIA labels for interactive elements, time-stamped section navigation #### Keyboard navigation Present: Standard video player controls Improve: Keyboard shortcuts for demo sections, tab-accessible transcript navigation #### Contrast and readability Present: High contrast code examples in demo Improve: Verify color choices meet WCAG AA standards, provide high-contrast theme option #### Text resizing and reflow Present: Responsive video player Improve: Ensure demo interface scales properly at 200% zoom, test transcript readability #### Media and captions Present: Full transcript available Improve: Closed captions with technical term spelling, downloadable transcript option #### Forms and interactions Present: Basic video controls Improve: Accessible demo interface for hands-on practice, clear error messages for invalid inputs ### 🌍 DEIB Assessment #### Cultural breadth Present: Technical content applicable across regions Improve: Examples from different industries beyond US telco, consideration of multilingual customer support scenarios #### Inclusive language Present: Professional technical terminology Improve: Define technical jargon consistently, avoid idioms that may not translate across cultures #### Neurodiversity support Present: Clear structure and systematic approach Improve: Consistent heading patterns, predictable section organization, visual breaks between concepts #### Representation Present: Focus on technical content over personal examples Improve: Include diverse use case examples beyond customer support, acknowledge different organizational contexts #### Psychological safety Present: Professional conference environment Improve: Acknowledge that prompt debugging can be challenging, normalize iteration and failure in learning process ### ✅ Logical Validity #### Strong reasoning - **Systematic evaluation approach**: Demonstrates clear cause-effect relationships between prompt changes and performance improvements with specific test cases - **Empirical validation**: Uses both programmatic checks and measurable outcomes to validate each optimization step - **Comparative analysis**: Shows rigorous comparison between different architectural approaches with consistent metrics #### Fallacies - **Hasty generalization**: Presents solutions based on limited test cases (5 for customer support, retail scheduling examples). Should acknowledge need for broader validation across different domains and use cases. - **Post hoc reasoning**: Some prompt improvements showed immediate results, but causation vs. correlation could be better established with more controlled testing. #### Tone of voice Professional and instructional with practical focus. Uses inclusive language ("we'll learn," "let's have a look") and acknowledges real-world complexity while maintaining technical precision. ### 🛡️ DARVO Analysis No DARVO patterns detected. The presentation maintains a collaborative, educational tone focused on technical best practices rather than defensive rhetoric. The speaker acknowledges both model limitations and prompt engineering challenges without deflecting responsibility or attacking critics. ## 🎯 Conclusion This presentation provides a systematic framework for prompt engineering that moves beyond trial-and-error approaches to evaluation-driven optimization. The key insight is that effective prompt maintenance requires treating prompts as living systems that need regular auditing, especially when models evolve. For practitioners, the immediate next steps are establishing evaluation suites for existing AI systems, implementing XML-based prompt structure, and considering agentic decomposition for complex reasoning tasks. The debugging methodology and architectural patterns presented here offer a foundation for building more reliable and maintainable AI systems in production environments. --- ## Full Transcript [00:20] Hello everyone. Um, thank you so much [00:23] for joining me this afternoon in the [00:25] breakout room. The last session today of [00:28] code with Claude. I hope you've all had [00:30] a fantastic day so far. My name is [00:32] Margot Vanlar. I am an applied AI [00:35] engineer at Anthropic here in London. [00:38] And this afternoon, we're going to be [00:39] talking about the prompting playbook. [00:42] And prompting is arguably one of the [00:45] first skills, if not the first skill [00:48] that we had to learn as engineers when [00:50] we first started to work with LLMs. and [00:53] even now it continues to be one of the [00:56] most critical um skills to building [00:59] effective AI systems. So today we're [01:02] going to discuss some best practices [01:05] um in the context of two practical [01:09] scenarios that you're probably [01:10] encountering at work. The first is where [01:13] you have an existing prompt in [01:15] production that you've been maintaining [01:17] for some time um and possibly you're [01:21] migrating it to a new model or making a [01:23] change to the architecture and for some [01:25] reason it's no longer working as well. [01:28] The second scenario is where we're [01:29] building an entirely new agentic use [01:32] case from the ground up and we need to [01:33] build the prompt from zero to one. [01:37] Now, [01:38] in order to illustrate these best [01:41] practices, I don't just want to give you [01:43] a list of dos and don'ts. I want to walk [01:45] through a practical example that's been [01:48] inspired by real prompts that um I've [01:51] seen some of our customers work with who [01:54] are building on Claude. So, the prompt [01:58] that we'll look at today is a [01:59] miniaturized example. the prompts that [02:02] you're working with are probably a lot [02:04] longer and more complex than the one [02:06] we'll see today. Um, but it's [02:08] representative of some common problems [02:11] that you might encounter when [02:13] maintaining a prompt. So, imagine that [02:16] we have a prompt that multiple people [02:19] have been collaborating on, contributing [02:21] to. There's no clear owner. It covers a [02:25] lot of different areas like policy, like [02:28] tone, processes. um we have some patches [02:32] for kind of previous models that we've [02:34] migrated to all mixed together. Um it's [02:37] built up and it's complex. And when [02:40] we're migrating to a new model, we're [02:43] finding that suddenly a lot of our test [02:46] cases are no longer working as well as [02:48] we expected. So what's actually going on [02:51] here? Well, in order to start unpacking [02:53] that question, um we need a starting [02:56] point. And that starting point is [02:58] evaluations. We need evaluations to [03:02] provide that rigor um to understand [03:05] whether a change to our prompt is [03:07] actually correlating to an improvement [03:09] in its performance. [03:12] And we have different models which have [03:14] different capabilities and different [03:17] behaviors. And when you migrate to a [03:20] different model, it could be that your [03:23] system is no longer working as well for [03:24] two reasons. First of all, if um the new [03:28] model might be capable, but it's [03:31] behaving differently and therefore we [03:32] can tune our prompting to fix that [03:35] behavior. The second case is where [03:38] actually the model that we're changing [03:39] to isn't as capable and no amount of [03:42] prompting is going to fix that. So we [03:45] need to have an eval suite to act as a [03:48] way of testing that regression so that [03:51] we can apply our prompting best [03:53] practices to that. [03:56] So in the example that we're going to be [03:59] looking at today, as I said, it's going [04:01] to be a miniaturized example. We'll have [04:03] five test cases in our eval. [04:07] In reality, you'll have a lot more test [04:09] cases in your EVA suite, but the key [04:12] thing here is that it's representative [04:14] of three key cases that we need to [04:16] cover. Those three key cases include [04:19] having a control case, which is a case [04:22] which should always pass. It's something [04:24] that the we know the model handles well. [04:26] It's unambiguous. [04:28] The second is edge cases. And these are [04:32] cases where we've seen the model fail [04:35] before. And by including instructions [04:38] into the prompt, we're making sure that [04:40] same behavior doesn't slip through again [04:42] in the future. And finally, and [04:45] critically, we need to make sure that [04:47] the model has a good understanding of [04:50] the extent of its capabilities, where it [04:53] should be handing off to a human or [04:56] where actually maybe it should be point [04:57] blank refusing to answer a request. [05:02] So, in the example that we're going to [05:04] be looking at today, um we'll be using a [05:08] um prompt for a customer support bot for [05:11] a telco company called Meridian Mobile. [05:14] And these are the five test cases that [05:17] we are going to be looking at today. We [05:19] have a simple control case looking at um [05:24] you know what's the data limit in the [05:25] basic plan. Uh we're also looking at [05:29] edge cases such as its ability to do [05:32] calculations such as calculating [05:35] proration bills. If I switch my bill [05:37] halfway through the month uh or if I [05:39] switch my plan halfway through the [05:40] month, what will my bill look like? We [05:43] want to check that it's accurately [05:45] addressing key questions which are [05:47] covered by our policy. [05:50] um we need to make sure that it's [05:51] escalating to a human whenever there is [05:54] um a billing error. Um and finally, we [05:58] want to make sure that our model isn't [06:00] withholding any information that it has [06:02] access to, which it should be handing [06:04] over to the customer. [06:07] So what we're going to do in this [06:08] process is we'll take our prompt and [06:12] we'll run it on our v0ero um of of the [06:16] eval and we'll see what our failure [06:18] modes are and systematically target [06:20] those failure modes one at a time to see [06:22] if we can resolve those failure modes by [06:24] prompting and along the way we'll learn [06:27] a little bit more about the kind of [06:28] antiatterns uh um and traps to avoid [06:32] and this is representative of how we [06:35] would apply apply these best prompting [06:37] techniques in practice. Right? We are [06:39] rarely writing a prompt from scratch. [06:42] We're often debugging an existing [06:44] prompt. [06:46] And best practice before we start [06:48] targeting those failure modes [06:50] specifically is to kind of apply our [06:52] general prompting 101 best practices, [06:55] applying general hygiene to clean up [06:58] before we do the eval run. So let's have [07:02] a little look at the example that we're [07:04] going to be using. So what we're looking [07:06] at here first of all before we look at [07:08] the prompt is just this vibecoded web [07:10] app that I've made for the presentation [07:12] today so that we can look at how we're [07:14] iterating on the prompt together. Uh um [07:17] in this page here I can easily run my [07:21] evals on all five test cases and inspect [07:24] the results in a little bit more detail. [07:28] So before we have a look at the prompt [07:30] I'm just going to run the evals in the [07:32] background. [07:34] This is a pretty good first pass at a [07:38] prompt. When we look at this, we've [07:40] defined the bot's role at the top. [07:44] When we scroll down, um we've given it [07:47] some data. We've given it some [07:50] information on how to reason over uh um [07:53] the answers that it should be giving to [07:55] the customer. It's giving some critical [07:58] instructions around the tone it should [08:00] use, um how to do calculations, etc. And [08:04] then finally we're passing in our [08:05] customer account context and our user [08:08] message. So let's have a look at how our [08:12] first pass the evals did. So we can see [08:15] as we expect our control case all of our [08:18] test cases have passed. This is what we [08:21] expect for this unambiguous test case. [08:24] But it's performing pretty poorly in [08:27] these other areas. Now, before we zoom [08:31] in on those specific failure modes here, [08:33] let's do some general cleanup of our [08:36] prompt. [08:43] So, as we mentioned, when we look [08:44] through this prompt, there's a couple [08:47] oddities here already. So, for example, [08:50] first one is we're telling the bot that [08:52] it's um a human, which just isn't true, [08:56] right? We can see as we scroll down [08:57] there's clearly some information here [08:59] that has been copied directly from a [09:01] website. So the key giveaway here is a [09:03] reference to a hero image. Um there's [09:06] even some references to cookies um at [09:09] the bottom. [09:10] So we need to remove a bit of redundant [09:12] information. [09:15] When we look at the instructions here, [09:17] they're all grouped into one big [09:19] paragraph. So we've got some reasoning [09:21] here. We've got instructions about the [09:23] role. uh um some critical instructions [09:26] as well without a real way of unpacking [09:30] um policy from guidelines from tone etc. [09:34] So [09:37] let me just [09:41] I've preempted some changes we want to [09:42] make to this prompt and this is just a [09:44] diff view of some of those changes. So [09:47] what we've done is first of all added [09:49] some structure. So you can see that [09:50] we've added XML tags here to define the [09:53] role to separate general guidelines [09:58] to separate policy to separate tone of [10:01] voice [10:04] um etc. [10:08] So if we run that eval then on this new [10:12] updated prompt, we should hopefully see [10:14] an improvement in the output as is. [10:27] So we can see just by clearing up the [10:29] prompt, we've already improved the [10:32] model's performance on this prepaid [10:34] scenario. [10:36] There's an interesting regression there [10:37] in that fifth hotspot case and I don't [10:40] want to worry too much about that now. [10:42] There's going to be some natural level [10:44] of variance in the different runs of the [10:47] eval and we'll come back to that case [10:50] specifically to see if we can make the [10:52] prompt consistently better in that area. [10:54] So what did we learn from this then? Um, [10:58] simply clearing up the prompt with a [11:01] better structure, with a better role [11:03] description has improved the [11:04] performance. And this is a best practice [11:06] that you can return to at any stage of [11:09] writing and maintaining your prompt, [11:11] especially as your prompts get more [11:12] detailed and more complex. A general [11:15] rule of thumb that I like to follow is [11:17] if you're reading a prompt and you can't [11:20] tell guidelines from policy, from data, [11:23] most likely the model isn't able to [11:25] either. [11:27] So before looking at some of those cases [11:30] in more detail, there's a little bit [11:31] more general cleanup we can do. Um [11:34] specifically here looking at creating an [11:38] output contract. This is a key best [11:40] practice to follow if you're struggling [11:43] with your output format consistency. Now [11:46] in this case, we have a customer support [11:47] bot. We want it to reply in a [11:49] conversational tone. So it's unlikely to [11:52] be a big issue in this case, but it's [11:54] something to bear in mind if you you're [11:56] dealing with more complex output [11:58] structures like nested JSONs for [12:00] example. [12:04] So again, if we go back to the prompt [12:07] and see what fixes we can apply here. [12:10] First of all, we've added a section uh [12:14] um at the end where we've defined an [12:16] output format for the model telling it [12:18] to use uh um XML tags to output the [12:22] response. But the prompt is not always [12:26] the most effective way of handling [12:29] issues. We can also change things in the [12:31] harness to ensure consistency to a [12:34] higher degree. So what we've added here [12:37] to the API call is a stop sequence which [12:40] is going to detect that closing XML tag [12:45] and tell the model to stop generating a [12:47] response at that point. Now when I run [12:50] the eval here I don't necessarily expect [12:53] to see any clear improvement in [12:56] performance. Um, but it's a general best [12:58] practice that we should be following and [13:00] as I said is something that we should [13:02] remember in particular when we have more [13:04] complex output schemas. [13:09] One thing to point out here as well if [13:12] you do have a more complex output schema [13:14] something like structured outputs can be [13:15] incredibly helpful to ensure that [13:17] consistency in a more programmatic way. [13:21] Okay. So after the cleanup then we can [13:24] see that we now have two test cases [13:27] which are consistently passing but we [13:29] have three key failure modes the [13:31] proration the billing error and the [13:33] hotspot. So let's isolate these one by [13:37] one um to iterate on the prompt and and [13:40] see the effect of that. [13:50] First of all then the hotspot question. [13:52] So the question is how much hotspot data [13:54] is on my unlimited plan. What we expect [13:58] the model to do is state directly the [14:00] amount of hotspot data that the customer [14:02] has. [14:04] And the reason this is a slightly [14:06] complex case is because the customer [14:08] test case that we're dealing with is on [14:10] a legacy plan. So actually the current [14:13] policy doesn't apply to them. So if we [14:16] see what's going on in the actual test [14:18] case here, the customer data which we're [14:22] feeding uh um to the prompt includes the [14:26] amount of hotspot data that customer [14:27] has. They have 5 gigabytes, right? But [14:29] they also have a grandfathered plan. So [14:33] what we're seeing uh the model is [14:36] actually telling the customer is [14:40] the general um the unlimited plan [14:42] includes 4 GB um but since you're on a [14:45] legacy plan you should go check this out [14:47] yourself. [14:55] So let's have a look at the prompt then [14:57] to see um why the model [15:02] is deflecting this question to the [15:05] customer account URL rather than [15:06] actually giving the information itself. [15:10] Now if we read this prompt originally it [15:12] said we changed our plans recently and [15:14] the policy doc shows the current plan [15:16] data and customers on grandfather's plan [15:18] have different rates. Never give a [15:20] customer the wrong plan details. instead [15:23] point them to the URL. So it's clear [15:27] that this instruction, this latter one, [15:30] never give customer the wrong [15:31] information, is the instruction that the [15:33] bot has been optimizing for. And you [15:37] might recognize this as being very [15:40] similar to a patch that you might have [15:42] introduced in a previous model that you [15:45] were using to avoid where the model was [15:48] giving the customer the wrong [15:50] information about that plan. Now, as our [15:52] models have evolved, they have gotten [15:55] much better at instruction following. [15:56] So, it's likely that instructions like [15:58] these have now become redundant and are [16:00] actually being overfitted to. [16:06] So, what we're going to tell the model [16:07] instead is give this balanced view uh um [16:11] where it says, you know, customers on [16:13] grandfather's plan have different [16:15] allowances, but it's captured in the [16:17] customer information that's given and [16:18] that is the accurate source of truth. [16:26] So running the eval here we should [16:28] hopefully be addressing uh um all of the [16:31] test cases for the hotspot case. Now I [16:34] am running this live so there could be [16:36] some variability here but we see here [16:38] that now clearly all of our test cases [16:40] are are passing. [16:44] So what did we learn from this? Well, we [16:46] worry a lot about hallucinations or the [16:50] invention of facts and numbers, but [16:54] actually the opposite can also happen. [16:57] The model can withhold information that [17:00] it actually has access to. Now, we saw [17:02] here that this is likely a result of a [17:04] patch that we introduced for a previous [17:06] model. And a best practice that we could [17:09] follow here is actually using version [17:11] control where wherever we are making [17:14] defensive changes in the prompt, we are [17:18] tracking the reason why we've introduced [17:20] these. Sometimes they're necessary, but [17:23] in the future these kind of changes can [17:26] produce unwanted effects so that we can [17:28] backtrack on them. [17:31] So the next failing test case then is [17:34] this proration calculation where a [17:37] customer asks what if I upgrade to the [17:40] 30 gigabyte plan? What will my next bill [17:44] be? And what we want the model to do is [17:46] to perform some calculation and return [17:49] exactly uh um what their next bill would [17:52] be rather than giving some sort of vague [17:55] output which is what we can see it's [17:57] doing right now. Uh um if we look at [18:00] what the model is returning, [18:02] it's clearly reasoning through it. It's [18:05] doing a little bit of mental maths here [18:07] and there, but it's not really giving [18:08] the customer a concrete answer. And I [18:10] wouldn't rely on this as being able to [18:12] accurately give the customer a response. [18:20] So if we look at the prompt then to see [18:23] how we can fix this [18:29] in the original prompt [18:33] we can see that all the instructions [18:35] that were given to it is telling it [18:38] don't ever give a customer a vague [18:39] answer. Uh um critical always calculates [18:43] any pr-rated amounts correctly. Now, [18:47] telling the model to do a good job isn't [18:49] particularly helpful when we don't give [18:52] the model the capability to actually do [18:55] a good job. We want to avoid the model [18:57] doing mental math. So, what we're going [18:59] to introduce is give the model a tool. [19:01] So, we're saying in the prompt whenever [19:03] you're doing any calculations, please [19:05] use the calculate proration tool to do [19:08] so. [19:10] In order to introduce that tool, we need [19:12] to introduce it into the API to tell the [19:16] model you have access to this tool. We [19:19] need to define um the tool schema which [19:22] tells the model what this tool does and [19:24] when to use it. And then finally we need [19:27] to actually implement the tool which is [19:29] the maths behind how it should be doing [19:32] that calculation. [19:38] So running that eval then for another [19:41] pass [19:48] we can see that all the test cases are [19:50] now passing. It's clearly done uh um the [19:54] maths using the tool in the background [19:56] and returning the correct response. [20:00] So the key lesson to take away here is [20:03] instructions don't add capability. [20:06] Telling the model it's critical to do a [20:08] calculation right doesn't make it better [20:12] at mental math. So the correct approach [20:14] was to give it a tool. Overall giving it [20:16] the ability to reason over harder [20:19] problems and using tools to actually [20:22] execute them reliably. [20:25] So now we have one final failing test [20:28] case which we need to address which is [20:30] this billing error here. [20:34] In this scenario there is a billing [20:37] conflict and what we really want is the [20:40] agent to escalate this to a human. And [20:44] what we're seeing it doing instead is [20:46] it's trying to [20:48] explain to the customer what the reason [20:51] behind it might be. [20:54] and it's trying to kind of diagnose the [20:57] problem itself. [21:04] So in order to fix this behavior, let's [21:09] again have a look what it was told in [21:11] the prompt. [21:15] We see in the initial instructions it [21:17] was giving, it says, "Avoid escalating [21:19] or transferring to a care specialist [21:21] unless absolutely necessary as it cost [21:24] approximately $8 and it counts against [21:27] our team's fast contract resolution." [21:29] Now, this is only giving one side of the [21:32] story, right? We're telling it what the [21:34] cost is to escalating, but not the [21:36] benefit, which means it's going to [21:39] overfit again to not escalating this [21:43] scenario. And second of all, we've got [21:45] this clear conflict between what we've [21:47] defined [21:49] in the eval in terms of what we want the [21:51] model to do to do this escalation versus [21:54] what we're actually telling it to do. [21:56] And the fix that's relevant here is to [21:58] give it both sides of the story by [22:00] saying it costs $8 uh um to escalate a [22:05] case, but actually if you get this [22:08] wrong, [22:10] then it's going to cost you a refund as [22:14] well as customer trust. [22:16] Again, here we observed how the model [22:20] optimizes for a goal. And this kind of [22:22] instruction is a common instruction to [22:25] give. It's quite similar to the one we [22:27] saw earlier where we didn't want it to [22:29] overfitit to a certain type of behavior. [22:31] But it's the kind of instruction that [22:33] can be followed quite differently by [22:35] different generations of models. And [22:38] specifically, as models become more [22:40] intelligent, we need to remember to [22:42] state both sides of the trade-offs [22:44] because our models are becoming better [22:47] themselves at making those tradeoffs [22:50] themselves. [22:57] So [22:59] if we just go back to our eval then and [23:02] um run our final test case, [23:06] we should see that all of our evals are [23:09] now passing correctly. So overall, we [23:13] looked at applying general hygiene [23:15] principles, how that can provide an [23:16] initial uplift to the prompt, making [23:19] sure we're removing any redundant [23:21] instructions which were initially [23:23] intended as patches for previous models [23:26] behavior, making sure we're giving it [23:28] tools to do certain tasks reliably. [23:34] Now there's one other scenario that we [23:38] uh introduced at the start which is one [23:40] that you might also encounter in your [23:42] work which is where we're building a new [23:45] agent from scratch. And the example that [23:48] we'll look at here is um an agent whose [23:52] purpose it is to create a week-long [23:54] retail staff schedule based on employee [23:57] availability and other constraints. [24:00] And when we're building a new agent from [24:03] scratch, [24:05] we need to consider not just the prompt, [24:08] but also the model that we're using and [24:10] the harness that we're using. So in this [24:14] next example, we're going to compare a [24:16] number of approaches to explore the [24:18] impact of those three different areas. [24:24] So again, I've just vcoded up this web [24:27] app so that we can walk through this [24:29] problem. Uh um in this demo here I've [24:32] just laid out what the problem is that [24:34] we're addressing. We have our eight [24:36] employees. Um on the right we have this [24:39] schedule that we need to staff with the [24:41] headcount and we have our constraints [24:45] that must be satisfied in every [24:47] scenario. [24:50] Now, because we have these hard rules, [24:53] rather than using an LLM judge like we [24:56] did in the previous case to do the [24:57] grading, we can actually use a just a [25:01] Python function which programmatically [25:03] checks for every schedule that's [25:06] generated how many violations were made. [25:13] So to begin with in a we want to start [25:17] simple. We're going to use a simple [25:19] prompt. We're going to use the bare [25:20] bones that we think we'll need with a [25:22] model set 46 to see how it performs and [25:25] how we're going to hill climb against [25:27] that. Um, so here is our baseline [25:29] prompt. We've already applied some of [25:32] that general hygiene and those best [25:34] practices that we saw earlier on using [25:36] XML tags to structure the prompt. We've [25:39] given it an output format as well. Now [25:41] that we're giving a schedule, uh we're [25:43] asking it to [25:45] output a JSON which if we don't give [25:49] that output structure might lead to [25:51] passing errors uh downstream. [25:59] When we run the [26:02] simple model on a first iteration of the [26:05] evals, all cases fail. Now, just what [26:09] we're looking at here is in our test [26:11] set, we're essentially repeating uh um [26:15] we're doing five trials here. Uh and [26:18] these numbers are showing how many [26:20] violations were made in each trial. [26:25] In the outputs, we can see that it's [26:28] made a decent attempt at reasoning [26:31] through the problem, [26:33] but it's burning a lot of tokens. Uh and [26:38] it's clearly not checking its work um as [26:40] it's not getting to the right impact. So [26:44] let's try a larger model uh a model [26:47] which we know is better at reasoning. So [26:49] we're going to run it through [26:52] uh Opus 4.7 instead [26:57] keeping everything else the same. Now, [27:00] interestingly, whilst all test cases are [27:04] still failing, you can see that the [27:06] overall number of violations that Opus [27:08] has made has reduced significantly from [27:12] Sonnet 4.6. So, we're possibly on to [27:16] something here, right? This isn't good [27:19] enough to ship because it's still [27:21] failing, but clearly giving it more [27:23] reasoning capability is helping drive it [27:26] towards a better result. So what we're [27:28] going to try next is using opus with [27:32] adaptive thinking instead. So it can [27:35] decide for itself [27:38] how much thinking it needs how much [27:40] reasoning it needs to use to solve this [27:44] issue. So no change to the prompt really [27:47] just uh a change to the API here. [27:54] So this now seems to reliably generate [27:58] compliance schedules, but it requires a [28:01] lot more tokens. [28:04] We're tripling essentially in the number [28:06] of tokens that we're using here, and [28:08] we're tripling the latency. So we want [28:10] to try and see if we can optimize that [28:13] cost latency trade-off a little bit [28:15] more. Um, this is latency 100 seconds. [28:18] Obviously, I'm running this one async. [28:21] uh for the purposes of time, Opus 47 [28:24] hasn't magically gotten much faster [28:25] since the last time uh you used it. Um [28:31] so [28:33] let's see if we can optimize a little [28:35] bit more for the token latency tradeoff. [28:39] What we haven't tried yet is using [28:41] Sonnet 46. So a smaller model but with a [28:44] better prompt. We looked a lot at the [28:45] prompt optimization uh um in that last [28:48] section. So I've added a couple details [28:51] to the prompt here. We're talk we're [28:54] in particular how to uh reason through [28:58] this problem and most critically telling [29:01] it to check its work before outputting [29:04] it. [29:10] So when I ran that EAL we see that it [29:13] passes in two out of the five cases. Now [29:17] the failure modes that we're seeing is [29:20] actually not violations of [29:25] the scheduling requirements but the [29:27] model hasn't been able to finish the [29:30] tasks within the output limit that we [29:34] set. So whilst we could increase the max [29:39] tokens that this model is able to use to [29:42] get all five test cases passing, [29:45] we see here that we're using even more [29:47] tokens and this run has an even higher [29:50] latency. So this is probably not the [29:52] route that we want to go down. [29:54] Now as a final pass, then we want to [29:58] look at doing this a little bit more [30:01] agentically. [30:03] So we're going to use this generate [30:05] evaluate repair loop where essentially [30:09] the generator now creates a first draft [30:12] of the schedule and then we have a [30:14] separate prompt [30:16] which reports any specific violations [30:19] that it made. So not programmatically [30:21] checking it but checking it with an LLM. [30:24] So we're checking for every rule and [30:27] we're providing evidence of every [30:29] violation. [30:30] And we then have a third repair prompt [30:35] which receives uh any violations that [30:38] were made and tries to make targeted [30:42] fixes to it. So we have three very [30:46] simple prompts but they're now running [30:47] independently rather than trying to do [30:50] everything in one large prompt. [30:57] So we can see in this case our [31:01] agentic approach has solved all of our [31:04] test cases um with a much lower number [31:09] of tokens and with a lower latency than [31:11] trying Sonet 46 [31:14] with a better prompt. So going forward, [31:16] it seems like there's two appropriate [31:20] approaches to take here. Using Opus 47 [31:23] with adaptive thinking or using this [31:25] agentic loop. Now moving forward, we'd [31:28] probably want to do a little bit more [31:30] optimization on this loop to try and get [31:33] it to be more efficient. But there's one [31:35] key benefit as well from using this [31:37] generate evaluate repair loop. And that [31:40] is that you can put in soft requirements [31:44] at runtime. So in the evaluation prompt, [31:47] we can say Harry doesn't like working [31:49] with Sally. So as much as possible, try [31:52] and separate them from working together [31:54] or we need a third shift uh um on [31:57] Wednesday, for example. So it means that [31:59] you're not having to make changes to the [32:02] um Python function which is doing the [32:04] evaluation in the back end every time to [32:07] satisfy for any soft constraints which [32:09] might depend just on a case-byase basis. [32:16] So to wrap up then [32:19] pulling all of those learnings together, [32:21] what did we see? Well, we looked at two [32:23] scenarios. Two scenarios which I, as an [32:26] engineer, see most in my day-to-day, [32:29] which is where we're maintaining a [32:30] prompt. We're migrating to a new model [32:32] which has some different behaviors and [32:35] we're and building a new use case from [32:37] scratch. We saw that general hygiene [32:42] principles following those can and [32:44] immediately uplift the performance [32:47] against um a set of evals and that we [32:50] need those eval to be able to rigorously [32:52] see any impacts of changing our prompt [32:55] on the output. Then we saw this process [32:58] of targeting failure modes one by one, [33:02] adding structure, avoiding long band [33:04] lists, etc. were all things that helped [33:08] push our model to the correct behavior. [33:10] And then finally with our new agentic [33:15] bot that we were building, we saw the [33:17] impact of splitting [33:20] into three separate prompt systems. So [33:23] rather than using one prompt to address [33:26] everything, we're actually isolating [33:28] different tasks where it's easy and [33:31] repeatable to separate out the steps [33:33] that it needs to take every time. [33:36] Thank you so much for attending this [33:38] afternoon. I hope you have a fantastic [33:40] rest of your day.  ## Highlights - Presents solutions based on limited test cases (5 for customer support, retail scheduling examples). Should acknowledge need for broader validation across different domains and use cases. - Note: 🔍 Fallacy: Hasty Generalization #fallacy/hasty-generalization - Some prompt improvements showed immediate results, but causation vs. correlation could be better established with more controlled testing. - Note: 🔍 Fallacy: Post Hoc #fallacy/post-hoc