Stop Building AI Like Traditional Software with Aishwarya Naresh Reganti

Stop Building AI Like Traditional Software with Aishwarya Naresh Reganti 🔗 **Source:** [Stop Building AI Like Traditional Software with Aishwarya Naresh Reganti](https://www.youtube.com/watch?v=GAF_ychy32k) > [!note]+ Document Notes > tags: [ai-development, continuous-calibration, agentic-systems] > author: Aishwarya Naresh Reganti > source: YouTube > reference: https://www.youtube.com/watch?v=GAF_ychy32k > date created: 2025-01-02 10:30 > > # 🏷️ Title: Stop Building AI Like Traditional Software >  ## 🧰 Action Inbox - [H1] Implement CCCD framework for next AI project — start with low agency, high control #ai-architecture - [H2] Design custom evaluation metrics beyond traditional unit tests — include LLM judges for natural language outputs #evaluation - [H3] Plan incremental agency increases across product versions — avoid full automation on day one #product-strategy ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [Original Video](https://www.youtube.com/watch?v=GAF_ychy32k) - [Lenny's Newsletter Post](https://www.lennysnewsletter.com) (referenced) ## 📝 Summary - Former AWS AI researcher shares hard lessons from building 30+ enterprise AI solutions - Traditional software and ML practices fail when building AI agents that interact with users through natural language - Introduces "Continuous Calibration, Continuous Development" (CCCD) framework as alternative to CI/CD - Key insight: Start with high control/low agency, gradually increase autonomy as AI system proves reliability - Real case study of customer support agent that failed in 5 hours demonstrates high stakes of automation ## 📖 Main Content Aishwarya Reganti, an AI researcher with 35+ papers and former AWS tech lead, argues that building AI systems requires fundamentally different approaches than traditional software development. Drawing from experience building 30+ enterprise AI solutions at AWS's GenAI Innovation Center, she reveals why standard CI/CD practices fail for AI systems. The core problem: AI systems interact through unpredictable natural language rather than deterministic UI elements, and they generate free-form responses that can't be evaluated with traditional unit tests or ML classification metrics. This creates an "agency-control tradeoff" where giving AI more autonomy reduces human oversight. Her solution is Continuous Calibration, Continuous Development (CCCD), which starts AI systems with minimal agency and human oversight, then gradually increases autonomy as the system proves reliable through real user interactions. This prevents the trust breakdowns that occur when fully autonomous AI systems fail in production. ## 🗒️ Notes #### ⏱️ [00:00](https://youtu.be/GAF_ychy32k?t=0) — [H1] Introduction to CCCD Framework - Key idea: Traditional software development approaches fail for AI systems due to natural language unpredictability - Evidence: "Stop building AI like traditional software" - need new frameworks for AI development #### ⏱️ [03:24](https://youtu.be/GAF_ychy32k?t=204) — [H2] Speaker Background and Context - Key idea: Speaker has both research (35+ papers) and enterprise experience building AI solutions - Evidence: "Early member at Alexa research team at Amazon" and "tech lead for GenAI innovation center" #### ⏱️ [05:12](https://youtu.be/GAF_ychy32k?t=312) — [H2] Common Enterprise AI Use Cases 2023 - Key idea: Two dominant patterns emerged - "find anything fast" (RAG) and "messy data into insights" (text-to-SQL) - Evidence: "Companies wanted a ChatGPT over their ecosystem" and "text to SQL or text to code" #### ⏱️ [07:45](https://youtu.be/GAF_ychy32k?t=465) — [H2] Team Backgrounds Building AI Solutions - Key idea: Teams combined software engineers (CI/CD, APIs, scaling) with ML/data science experts (evaluation, feature engineering) - Evidence: "No AI engineers or AIPMs... came from two backgrounds: software and machine learning" #### ⏱️ [09:18](https://youtu.be/GAF_ychy32k?t=558) — [H1] First Critical Lesson: Unpredictable User Interactions - Key idea: Natural language interactions are fundamentally different from predictable UI clicks and forms - Evidence: "Users will interact in surprising ways that you cannot predict up front" #### ⏱️ [11:30](https://youtu.be/GAF_ychy32k?t=690) — [H1] Second Critical Lesson: AI Evaluations Need Redesign - Key idea: Traditional unit tests and ML metrics (precision/recall) don't work for natural language generation - Evidence: "You cannot rely on traditional software tests... neither traditional ML metrics like precision recall" #### ⏱️ [13:42](https://youtu.be/GAF_ychy32k?t=822) — [H2] Real Example: Loan Officer Assistant Surprises - Key idea: Underwriters asked unexpected questions requiring historical data and cross-referencing capabilities - Evidence: "What exceptions did regulators flag for large loans... we did not have historical view" #### ⏱️ [15:25](https://youtu.be/GAF_ychy32k?t=925) — [H2] 2024 Mindset Shifts: Iterations and Smart Metrics - Key idea: Build smaller iterations to understand user behavior quickly, create custom evaluation metrics - Evidence: "Run small measurable iterations... define clear success metric whether user feedback, task completion" #### ⏱️ [18:10](https://youtu.be/GAF_ychy32k?t=1090) — [H2] Key Mental Model: AI System as Intern - Key idea: AI won't be perfect day one regardless of groundwork - needs training and feedback loops - Evidence: "Think of your AI system as an intern. It will not be perfect on day one" #### ⏱️ [20:33](https://youtu.be/GAF_ychy32k?t=1233) — [H1] Evolution to AI Agents in 2024 - Key idea: Enterprise demand shifted from chat systems to end-to-end workflow automation - Evidence: "Can you automate entire back office workflows... instead of just answering questions" #### ⏱️ [23:47](https://youtu.be/GAF_ychy32k?t=1427) — [H1] Customer Support Agent Failure Case Study - Key idea: Well-built agent shut down in 5 hours due to error cascades and user frustration - Evidence: "AI agent was shut down in 5 hours... wrong routing... agent would spiral into nonsense" #### ⏱️ [26:15](https://youtu.be/GAF_ychy32k?t=1575) — [H1] Agency-Control Tradeoff Discovery - Key idea: Giving AI more autonomy reduces human control - must build trust before granting agency - Evidence: "Every time you give AI agent more agency... you as developer are losing control" #### ⏱️ [28:40](https://youtu.be/GAF_ychy32k?t=1720) — [H1] CCCD Framework Introduction - Key idea: Start with high control/low agency, gradually increase autonomy through proven performance - Evidence: "Start building solutions that are very low agency first and give higher agency over time" #### ⏱️ [31:22](https://youtu.be/GAF_ychy32k?t=1882) — [H2] CCCD Development Loop Details - Key idea: Three steps - collect reference data, simple architecture, design custom evaluations - Evidence: "Start with small high control and low agency capability... design eval with custom metrics" #### ⏱️ [33:55](https://youtu.be/GAF_ychy32k?t=2035) — [H2] CCCD Calibration Loop Details - Key idea: Deploy to small cohorts, run evals on real interactions, analyze failure patterns iteratively - Evidence: "Run evals for subset of real user interaction... analyze recurring failure modes" #### ⏱️ [36:12](https://youtu.be/GAF_ychy32k?t=2172) — [H2] Marketing Assistant Example: Agency Progression - Key idea: V1 draft content for human review, V2 multi-step campaigns, V3 autonomous optimization - Evidence: "Draft email... human review... multi-step campaign... AB testing and auto-optimize" #### ⏱️ [38:30](https://youtu.be/GAF_ychy32k?t=2310) — [H2] GitHub Copilot Evolution Pattern - Key idea: Coding assistants followed same pattern - inline suggestions to autonomous pull requests - Evidence: "Inline completions... larger code blocks... completely scoped changes... submit pull requests" #### ⏱️ [40:45](https://youtu.be/GAF_ychy32k?t=2445) — [H2] Customer Support Success Case: Three Versions - Key idea: V1 routing assistant, V2 co-pilot with SOPs, V3 autonomous resolution with escalation - Evidence: "Route ticket to right department... draft reply for human... auto-resolve common tickets" #### ⏱️ [43:18](https://youtu.be/GAF_ychy32k?t=2598) — [H1] Data Flywheel Creation - Key idea: Each version generates insights for next iteration - routing accuracy, retrieval gaps, escalation needs - Evidence: "How users describe problems... what human agents use or ignore... where autonomy causes trust breakdowns" ## 📚 References - [Stop Building AI Like Traditional Software](https://www.youtube.com/watch?v=GAF_ychy32k) - [Lenny's Newsletter - CCCD Framework](https://www.lennysnewsletter.com) - [Alan Black - CMU Professor](https://www.cs.cmu.edu/~awb/) ## 🗣️ Paraphrase - AI systems require different development approaches than traditional software because users interact through unpredictable natural language rather than deterministic interfaces - The CCCD framework starts AI systems with minimal autonomy and human oversight, gradually increasing agency as reliability is proven through real user interactions - Real-world case studies show that fully autonomous AI can fail catastrophically in production, requiring careful progression from co-pilot to autonomous modes ## 📰 Newsletter **Game-Changing Framework for AI Development: Stop Building Like Traditional Software** A former AWS AI researcher just shared a framework that could save your next AI project from catastrophic failure. After building 30+ enterprise AI solutions, Aishwarya Reganti discovered why traditional CI/CD approaches fail for AI systems. **The Problem**: AI systems interact through unpredictable natural language, not deterministic UI clicks. When a customer support AI agent failed within 5 hours of deployment, it revealed the critical "agency-control tradeoff." **The Solution**: Continuous Calibration, Continuous Development (CCCD) - start with high control/low agency, gradually increase autonomy as the system proves reliable. **Practical Application**: Instead of launching a fully autonomous AI agent, build V1 as routing assistant, V2 as co-pilot, V3 as autonomous system with escalation paths. Resources: [AI Development Best Practices](https://learn.wordpress.org/), [System Architecture](https://developer.wordpress.org/), [Performance Monitoring](https://wp-cli.org/) ## ✅ Action Items #### DevRel - Developer advocacy: Create workshops on CCCD framework vs traditional CI/CD for AI systems - Education: Build curriculum on AI evaluation metrics beyond unit tests and ML classification - Developer experience: Document agency-control tradeoff patterns in AI system design - Developer marketing: Case studies showing graduated autonomy in successful AI products - Community management: Forums for sharing AI system calibration experiences #### OSPO - Policy & Governance: Guidelines for AI system deployment with human oversight requirements - License Compliance: Review AI model licenses for enterprise automation use cases - Security Management: Escalation procedures when AI systems require human intervention - Software Inventory: SBOM tracking for AI agent dependencies and evaluation frameworks - Developer Support: Training on CCCD implementation and natural language evaluation - Community Engagement: Fund research on AI system reliability and trust metrics - Strategic Alignment: KPIs for measuring AI agency progression and user trust #### Community - Maintainers: Templates for AI system evaluation and calibration workflows - Contributors: Guidelines for contributing to AI evaluation frameworks and metrics - Sponsors: Fund development of open-source CCCD implementation tools - Ecosystem: Cross-project standards for AI system agency progression #### WordPress - Core AI: Implement CCCD principles for WordPress AI features, graduated autonomy rollout - Core Performance: Monitor performance impact of AI evaluation systems - Community: Workshop materials on building AI features for WordPress plugins/themes - Training: Curriculum on AI system design patterns and evaluation methods - Documentation: Guidelines for plugin developers building AI features - Plugins: Review process updates for AI-powered plugins requiring user interaction - Accessibility: Ensure AI system outputs meet WCAG standards for generated content ## 🧪 Hypotheses - AI systems that start with low agency and gradually increase autonomy will have higher user trust and retention than fully autonomous systems deployed immediately - Natural language evaluation metrics (LLM judges) will provide more accurate assessment of AI system performance than traditional software testing approaches - The agency-control tradeoff applies universally across AI applications, requiring calibration periods regardless of domain - Data flywheels created through CCCD iterations will produce higher quality AI systems than traditional ML training approaches ### 📊 Methodology - Study design: Compare user satisfaction and system reliability between CCCD-developed AI systems vs traditional deployment - Data sources: User interaction logs, escalation rates, task completion metrics across multiple iterations - Experiments: A/B test graduated autonomy rollouts vs full automation in controlled enterprise environments ### ❓ Questions - How long should calibration periods be before increasing AI system agency across different domains? - What are the optimal evaluation metrics for natural language AI system outputs beyond human feedback? - How can the agency-control tradeoff be quantified and measured systematically? ### 🔮 Future Research - Standardized frameworks for measuring AI system readiness for increased autonomy - Cross-industry benchmarks for AI agent calibration periods and success metrics - Automated tools for implementing CCCD workflows in existing development pipelines ### 💡 Implications - Practice: Development teams need new skills in AI evaluation design and natural language system testing - Policy: Regulatory frameworks may need to account for graduated AI autonomy rather than binary automation decisions - Security: AI systems require different security models that account for unpredictable natural language interactions - Education: Computer science curricula should include AI-specific development methodologies beyond traditional software engineering - Funding: Investment in AI startups should account for longer calibration periods and iterative deployment strategies ### 🧠 Learning Styles Assessment Reading level estimate: 12-14 (Graduate level due to technical AI concepts) #### 👀 Visual learners - Present: Agency-control tradeoff diagram, CCCD loop illustrations - Improve: Architecture diagrams showing system evolution, user interaction flowcharts #### 🔊 Aural learners - Present: Video presentation with clear narration - Improve: Podcast-style discussion of case studies, audio summaries #### 🗣️ Verbal learners - Present: Clear definitions of CCCD, agency-control tradeoff, calibration concepts - Improve: Analogies comparing AI development to traditional software, glossary #### 🧩 Kinesthetic learners - Present: Step-by-step CCCD implementation examples - Improve: Interactive workshops, hands-on AI evaluation exercises #### 🧠 Logical learners - Present: Clear framework structure, case study analysis, failure mode breakdown - Improve: Decision trees for agency increases, quantified metrics #### 🧑‍🤝‍🧑 Social learners - Present: Team collaboration examples, enterprise case studies - Improve: Discussion forums, peer review processes #### 🧘‍♀️ Solitary learners - Present: Individual --- ## Full Transcript [00:00] Thanks everyone for making it to the [00:02] session today. Um my talk is titled stop [00:06] building AI like traditional software [00:08] and you'll kind of see why I'm getting [00:10] to that point. Um let's get started. [00:15] Um so today I'll be introducing [00:18] something called continuous calibration, [00:21] continuous development. And um before I [00:24] kind of introduce that concept which I [00:26] have picked up over the past few years [00:28] of building AI products, I will also [00:31] introduce you to why that mindset came [00:33] around and why a lot of people think of [00:36] building AI products as traditional [00:37] software and then they forget the key [00:39] differences that AI systems have. And [00:42] I'll kind of build um the story from [00:45] there and also give you a lot of backing [00:46] as to why we came up with this concept [00:48] of continuous calibration, continuous [00:50] development. [00:52] A little bit about me um thanks for the [00:54] wonderful introduction but um a little [00:56] more about my background is I was an AI [01:00] researcher in my past life. I have about [01:02] 35 plus research papers in top AI [01:05] conferences and um my adviser was Alan [01:08] Black and if you come from the machine [01:10] learning world from the NLP world before [01:12] LLM's happened um Alan Black was a [01:15] popular professor um anyway I was also [01:17] an early member at the Alexa research [01:20] team at Amazon and also worked on search [01:22] and recommendations at Amazon using [01:24] graph neural networks. Uh over the past [01:27] two years um I worked at um AWS as a [01:31] tech lead for a org that we call Genaii [01:34] innovation center where we build AI [01:36] solutions using our existing um um uh [01:40] bedrock which is our um AI offering and [01:43] just about 4 months ago I left AWS and [01:46] now I run an AI consulting firm and the [01:49] whole idea is to build AI solutions for [01:52] companies and also do it with a very [01:54] forward deployed mindset. [01:56] um which means we enter your company, we [01:58] start um understanding your ecosystem [02:00] and also build solutions for you. Um [02:03] more about it you can find on my [02:04] LinkedIn but that's kind of my [02:06] background. I I have both of a research [02:08] as well as an enterprise background. [02:10] [snorts] [02:11] Um I will walk you through a bunch of [02:14] lessons from 2023 to 2024 and then 2024 [02:18] to later and which will help build up [02:20] why we came up with continuous [02:22] calibration and continuous development, [02:24] right? Um and remember in 2023 to early [02:28] 2025 I was part of AWS. Um and I was a [02:31] tech lead at Genai innovation center and [02:33] these are all the lessons I learned from [02:34] building about 30 plus use cases while [02:37] at AWS. [02:39] Um 2022 November was when Chptd came [02:42] out. Um people were going gaga about it. [02:45] Um a lot of um consumer use cases came [02:47] into picture. People were chatting with [02:49] Chad GPT and it was seeming super fun. [02:51] But then enterprises also started [02:53] thinking how can we use this for our own [02:55] use cases. Um having something like a [02:57] charge is good but then it doesn't [02:59] really serve our data sets or our [03:02] ecosystem. So what can we do about it? [03:04] And that's when um we saw a lot of um [03:08] common use cases. Two big use cases that [03:10] we saw for enterprises were um find [03:13] anything fast, right? Which essentially [03:16] companies wanted had a lot of [03:18] unstructured data sets and structured [03:20] data as well and they wanted a chat GPT [03:23] over their ecosystem, right? And that's [03:26] kind of the very common use case that we [03:28] saw in 2023. Um the fancy term for it, [03:31] you probably heard of it, it's called [03:32] retrieval augmented generation or rag. [03:34] um essentially means pull up information [03:36] from our knowledge base, summarize it [03:38] for us and answer questions. And the [03:40] second common use case that we saw was [03:43] um on turning messy data into insights, [03:45] right? Companies have a lot of SQL [03:47] databases, relational databases and they [03:50] wanted methods to extract data or to [03:53] transform that data and build analytics. [03:55] It was previously done through code, but [03:58] now um folks wanted things like text to [04:00] SQL or text to code so that they could [04:03] have um a lesser uh bar for people who [04:07] could actually you know talk with their [04:09] data and not really have um you know [04:11] domain experts or data scientists work [04:13] on this. So these were very um common [04:15] use cases that we saw in 2023. Uh there [04:19] are a bunch of public examples that I [04:21] worked on. Uh we worked with a um um ISV [04:25] called imply data where we built a chat [04:27] with your data system on druid SQL which [04:30] is their version of SQL. Uh we also [04:32] built a geoscience co-pilot for [04:33] Hallebertton and also built a loan [04:35] officer assistant for another global [04:37] bank. So these are the kind of use cases [04:38] that we kept seeing. Um now remember [04:42] that in 2023 there were no AI engineers [04:45] or AIPMs or all of those AI roles that [04:47] we talk about today. A lot of the people [04:50] who started building, a lot of the teams [04:52] that started building these solutions [04:54] essentially came from two backgrounds, [04:55] right? One of them was software. They [04:57] were software engineers who understood, [04:59] you know, system architectures. Well, [05:00] they knew how to build CI/CD pipelines. [05:02] They understood how to design APIs, [05:04] scaling, DevOps and all of the software [05:06] practices, right? And then there was [05:08] folks like me who came from the machine [05:10] learning or data science uh background. [05:13] And what we knew was how do you play [05:14] around with your data? Do you know if [05:16] your data is ready? what kind of [05:17] valuation metrics should you be [05:18] building? How do you build [05:19] observability? How do you do feature [05:21] engineering and things like that, right? [05:23] And both of um these backgrounds kind of [05:27] our team also consisted of uh folks in [05:30] the um these backgrounds and we came [05:31] together to you know actually build [05:33] generative AI solutions and we figured [05:35] out that the extra thing that you need [05:37] to be learning um was to you know [05:39] understand prompt engineering because [05:41] that was a new field that um you know [05:43] both ML and software folks had um hadn't [05:46] really played around with a lot and then [05:48] more around you know as rag became more [05:50] mature there were vector DBs and stuff [05:52] like [05:53] This was pretty cool because we had both [05:56] the software and machine learning [05:57] background and all we had to do is [05:59] augment that with the latest generative [06:01] AI principles, right? Um uh and then we [06:05] did about 40 use cases that year from [06:07] 2023 to 2024. Um and having come from [06:12] both of these backgrounds, we learned [06:14] two important lessons, right? And one of [06:16] those lessons was um one is that users [06:20] will interact in surprising ways that [06:22] you cannot predict up front and that AI [06:25] evaluations need deeper design right [06:27] remember that if you're a software [06:28] engineer or even you work with machine [06:31] learning a lot of user behavior is very [06:33] um on a high level it's kind of [06:35] predictable because you have your users [06:37] you probably have a UI which has you [06:39] know clicks buttons options forms and [06:41] things like that and you have a backend [06:42] connected to that right which is [06:44] probably connected to servers databases [06:46] and stuff like that. Now with AI [06:48] systems, your interaction layer becomes [06:51] a lot more um ambiguous in the sense [06:54] that users are not um dealing with, you [06:57] know, menu buttons or clicks and things [06:59] like that. They're dealing with um [07:02] natural language interaction. You can [07:03] see the um figure on the right where [07:06] you're probably having voice or chat or [07:08] video and things like that which makes [07:10] things a lot more harder to predict um [07:13] and kind of design for, right? You're [07:15] not designing for guaranteed behavior. [07:17] You're designing for anticipated [07:19] behavior. And that was a huge learning [07:23] for us when we started building these [07:24] systems because you really need to think [07:27] spend the time to think about how users [07:30] will interact with your system. what [07:31] kind of sentences they might come with [07:33] and just the fact that each of us [07:35] communicate very differently. So there [07:37] has to be that uh understanding of how [07:40] users would communicate with your [07:41] product. And the second lesson we [07:43] learned is that um evaluations need [07:45] deeper design. Again remember even the [07:49] uh even what the AI model predicts can [07:52] be is natural language or it's probably [07:54] voice and things like that. So you [07:56] really need much smarter ways of [07:58] evaluating if a use case was successful. [08:01] You cannot rely on traditional software [08:04] tests like unit tests or integration [08:05] tests and things like that. Neither can [08:08] you rely on very traditional machine [08:09] learning metrics like precision recall [08:11] and stuff like that because those were [08:13] designed with classification or [08:15] regression in mind. Right now you have [08:17] both users interacting with your systems [08:20] in very unpredictable ways and the AI [08:22] model is also generating um you know [08:25] responses that are much more free flow [08:27] than before. So these were two important [08:29] lessons that we learned right. Um one [08:33] example is when I said we were building [08:34] for a global bank we were building a [08:36] loan officer assistant. Turns out that [08:39] the underwriters who were using this [08:40] assistant um started asking questions [08:44] that did we did not anticipate, right? [08:46] For instance, they said, "What [08:47] exceptions did regulators flag for large [08:49] loans in this section?" Um and we just [08:51] had a rag system that was built on top [08:53] of policies and we did not have a [08:55] historical view of the data and they [08:57] were also trying to cross-check proceed [08:59] and style cases like lawyers and things [09:01] like that, right? And we figured out [09:02] that there was data in incompleteness in [09:05] order to serve these features. And even [09:07] as just a chat GPT user, you will kind [09:09] of see that maybe from 2023 to now the [09:12] kind of questions or the kind of queries [09:13] that you've asked Chad GT Chad GPT or [09:16] you know Claude or Gemini or whatever [09:18] this is has evolved significantly, [09:21] right? The patterns of interaction have [09:23] evolved and we need to as builders [09:25] account for that evolution. [09:28] Um so in 2024 we thought okay um that [09:32] makes sense those are some important [09:34] lessons and we said um from now on we [09:37] will be building solutions with quicker [09:39] iterations so that we understand user [09:41] behavior as quickly as possible and we [09:42] can evolve our system to do that and [09:44] we'll also think of much smarter [09:46] evaluation metrics because older metrics [09:48] no longer work right these were like two [09:50] mindset shifts we had which is you know [09:52] run small measurable iterations in order [09:54] to gather user behavior quickly don't [09:56] wait for your final product to be out [09:58] there. Build smaller iterations so that [10:00] you know how users are interacting with [10:02] your system. And the other thing was [10:04] again define a very clear success metric [10:06] whether that is user feedback, task [10:08] completion and along the way check for [10:10] intermediate signals as well. Right? If [10:12] you're building a text to SQL system, [10:14] you could have a check in between to [10:16] just make sure that the syntax of the [10:17] SQL makes sense. You could also um you [10:20] know if you have tool calling agents and [10:22] things like that you could have [10:23] intermediate check make sure that the [10:24] right tool was called um then looking at [10:27] the entire you know final um response of [10:30] whether it was successful or not try to [10:32] have intermediate metrics as well. If [10:33] you're building a rack pipeline can you [10:35] think about how good the retrieval was [10:36] and stuff like that. So think of ways to [10:38] evaluate your pipeline um break it down [10:41] instead of looking at the final action [10:44] right and then you know test all of [10:46] these components separately and stuff [10:47] and um we also figured that since these [10:50] metrics had to be so open-ended as [10:53] compared to traditional machine learning [10:54] metrics or software metrics um we also [10:57] started building uh LLM judges or AI [10:59] judges um that could replace human [11:02] judgment and could you know scale human [11:04] judgment right um and if if I To [11:07] summarize this in a single statement, it [11:09] would um just be that um the biggest [11:12] mindset shift for us was that think of [11:14] your AI system as an intern. It will not [11:16] be perfect on day one. Regardless of how [11:18] much groundwork you do to build this [11:20] system, you will see that users will [11:23] always come up with behavior patterns [11:25] that you never expected. So it won't be [11:27] perfect on day one, but think of ways to [11:29] build iterations and to also collect [11:31] feedback so that you can iterate quickly [11:32] and get to a point where it starts [11:34] performing well, right? Um and even as [11:38] we build these solutions for customers [11:40] um in early stages when it went to [11:42] production we would always iterate you [11:44] know weekly we would look at the data [11:45] weekly we try to improve our systems [11:47] weekly and as we started re reaching a [11:50] point of convergence where we knew a lot [11:53] of the user patterns that's when we [11:55] extended that iteration to you know [11:57] maybe monthly or bimonthly cycle so that [12:00] um you know um the the product was [12:02] stable enough right so that was one huge [12:04] mindset shift that we made we were in [12:07] the middle of 2024 and we picked up [12:09] these two new you know concepts which [12:12] was AI evaluations and also shorter [12:14] iterations and we we were doing pretty [12:16] well and that's when um the whole world [12:18] started talking about this new concept [12:20] right which is um AI agents um from 2023 [12:26] to 2024 a lot of the enterprises their [12:28] mindset around AI was very chat like [12:31] chat system like or you know rag and [12:33] things like that but And in late not [12:36] even late maybe early 2024 people [12:38] started thinking a lot more about not [12:41] just chat systems but also agents that [12:44] could perform tasks end to end. Now [12:46] there's a lot of debate on what agents [12:47] mean. Um but um um at the high level it [12:52] just refers to systems that can perform [12:54] tasks end to end and um basically [12:57] complete workflows right um and then we [13:00] saw that the kind of use cases that we [13:01] started getting from enterprises looked [13:03] a lot more different in 2024 right [13:05] instead of them coming up with things [13:07] like can you build an internal search [13:09] system for us can you build an insights [13:10] dash I mean can you build a text to um [13:13] code kind of system all of their [13:15] requests were more of can you automate [13:17] entire back office kind of workflows for [13:19] us because now you have agents, you have [13:22] the mindset of agents that could call [13:24] other tools that could have memory and [13:25] things like that, right? And some of the [13:27] common use cases that we saw back in um [13:30] you know late 2024 to 2025 was um things [13:34] like customer support automation or um [13:36] for the global bank that we worked on, [13:38] they wanted a more mature use case where [13:40] we were um also doing loan underwriting [13:43] assistant. instead of just answering [13:44] questions, it could also help you do the [13:46] underwriting and could kind of fill in [13:48] some parts so that um uh underwriters [13:51] could spend much less time on doing [13:52] those laborious things, right? [13:55] Um so we were like, cool. I mean, we've [13:57] already done this. Um there's not a huge [14:00] difference. It's pretty much the same [14:01] pipeline, just that the scope of work [14:03] increases. So we're like, okay, we're [14:05] the best team that exists. All of you [14:06] will get agents. and uh enterprises were [14:09] you know kind of super enthusiastic [14:11] about the concept and were like we we've [14:13] got this right [gasps] but then um and [14:16] remember I also went through these extra [14:18] things that we had picked up in that one [14:20] year uh we were all from a machine [14:22] learning or software background and we [14:23] had picked up all of these things about [14:24] evaluations about iterations prompt [14:26] engineering vector DBs and we like okay [14:28] we're experts at this right um and then [14:31] we built a customer support agent for a [14:33] company um unfortunately I won't be able [14:36] to diverge the name but we built it for [14:37] a retail company and the idea was to [14:39] classify tickets to the right department [14:41] and to also retrieve resolution [14:43] documents or standard operating [14:44] procedures or SOPs and provide an answer [14:47] and follow up on user versions typically [14:48] what your customer support agent would [14:50] do but then automating that entire [14:51] workflow right and we kept those two [14:54] principles that we kept in mind like [14:55] were you know build iterations deploy [14:57] early and gather user data and also have [15:00] good eval so that you know where your [15:01] system is going wrong. So we were like [15:03] okay this is this is pretty good and we [15:04] put it into production and then boom we [15:07] deployed it to a very small cohort of [15:09] course not to um you know everybody all [15:11] the customers that these folks had but [15:13] the AI agent was shut down in 5 hours [15:15] and we had to spend a lot of time doing [15:17] damage control right um we had the dream [15:20] team we built a wonderful solution but [15:22] why did it fail um there were three [15:24] reasons one was we had limited initial [15:27] data on how users interacted what kind [15:29] of questions that they would ask the [15:30] system but this was expected right we [15:32] kind knew that it is impossible to get [15:34] all that data up front and we had in our [15:36] mind that we're going to build [15:37] iterations. Um but then another thing [15:40] that happened is we the error started [15:42] rippling because of a wrong routing it [15:44] would pick up the wrong um SOP and then [15:47] agent would spiral into nonsense and [15:49] users were frustrated right remember [15:51] that when you're building your [15:52] iterations you want to build them in a [15:54] way such that you don't frustrate your [15:56] users we didn't think of that we thought [15:58] okay we're going to do error analysis we [16:00] have eval in place and we're going to [16:01] build a better version each time but [16:03] what happens during that time when your [16:05] agent is not up to the mark yet you [16:08] cannot really put it in front of [16:09] customers and that's kind of the issue [16:11] that we saw which is users were getting [16:12] super frustrated and they were demanding [16:14] that real human agents come in and um [16:17] these human agents who were there faced [16:19] a lot of backlash from angry customers [16:21] because there were no humans in the loop [16:22] right and that's why it had to shut it [16:24] down and that's kind of um where where I [16:28] come to this point of the blind spots we [16:30] actually ended up missing right um from [16:32] 2024 or 2023 to 2025 the huge mindset [16:37] shift that people had um or something [16:39] that we didn't really you know take into [16:41] account was that use cases were moving [16:43] from augmentation to automation which [16:45] means there were much higher stakes just [16:47] having the right set of eval or building [16:49] the right iterations were not enough and [16:51] full automation could quickly break user [16:53] trust right as soon as you put these AI [16:55] systems in front of customers you want [16:57] to make sure that they have a seamless [16:59] experience as before and we did not [17:01] think about that and that's where we [17:03] came up with this concept of agency [17:05] control tradeoff which means that every [17:07] time you give your AI agent more agency [17:11] or like more autonomy to do something to [17:13] take decisions you as a human or you as [17:17] you know the developer are losing [17:19] control right and you want to make sure [17:21] that when you're losing that control [17:23] you've your AI agent has built that [17:26] trust has built enough trust to do it [17:28] right um and we also learned that domain [17:31] experts remain super critical until the [17:33] intern or the AI agent matures and you [17:35] know decisions must be handed off to [17:37] real people if things go wrong, right? [17:40] Um, so on paper, agentic use cases could [17:43] be built in a similar way as rag or as [17:45] other um, you know, chat kind of [17:46] systems, but then remember that people [17:49] are not super frustrated when they ask [17:51] chat GPT for a question and it comes up [17:53] with the wrong answer because they're [17:54] like maybe doesn't know about about [17:56] something, right? I'll ask it in a [17:58] different way. But if you go to a [18:00] customer support system and you ask a [18:02] question and you're trying to get a [18:03] resolution and the support bot starts [18:05] saying, you know, some nonsense, you get [18:07] a lot more frustrated because we expect [18:10] that support systems work a certain way. [18:12] We are not ready to um, you know, take [18:15] any um, you know, mishaps the same way [18:18] we would do with chat GPT because that's [18:19] just a chat system. With a support [18:21] system, you actually want uh, you're [18:23] working towards an outcome. You want a [18:24] resolution as quickly as possible. And [18:26] that's also the mindset shift that we [18:27] kind of missed. [18:29] And that's where we came up with two new [18:31] learnings from 2025. Um and um a lot of [18:36] this builds on top of what we learned in [18:38] 2024. But essentially AI systems are [18:40] non-deterministic and you're designing [18:42] for likely behavior, not guaranteed [18:44] behavior. And that's where you know we [18:46] had all of these things around quicker [18:47] iterations, evalu [18:52] agency control tradeoff, right? which is [18:54] if you let AI systems make decisions for [18:57] you, then be sure they're well tested [18:59] and do not automate fully on day one. [19:02] Try to make sure that there are humans [19:03] in the loop. There are points where [19:05] seamless switching can happen so that [19:07] your user experience doesn't get [19:09] affected. This is one of the biggest um [19:12] uh reason of coming up with the CCCD [19:14] pipeline and that's where we um put [19:16] together this framework um uh which is [19:20] called continuous calibration and [19:21] continuous development and I built this [19:24] framework from the learnings that we had [19:26] both from consulting as well as working [19:28] for AWS and we use a lot of this when we [19:31] build for customers as well. The idea is [19:33] pretty simple. It's to come out of your [19:37] mindset of a CI/CD kind of an approach, [19:39] but think of an AI system as something [19:42] that needs continuous calibration. Um, [19:44] and you need you're continuously [19:46] sampling behavior in order to calibrate, [19:48] right? Um and the iterations concept [19:51] still remains the same but we say that [19:54] when you're building solutions AI [19:55] solutions start building solutions that [19:58] are very low agency first and give [20:01] higher agency to your system over time [20:03] right you kind of see this diagram where [20:05] you say as you go through it increase [20:07] the agency and lower the control don't [20:09] give super high autonomy or agency to [20:11] your system on day one right and it has [20:14] two parts of the loop uh one of it is [20:16] continuous calibration and the other is [20:18] continuous is development and I'll just [20:20] walk you through each of these. It's [20:22] pretty simple at the face um but starts [20:24] to make a lot of sense when you start [20:26] building AI systems, right? [20:28] Um the continuous development side of [20:31] the loop basically says start with a [20:34] small high control and low agency [20:36] capability for your AI product whatever [20:38] that is and plan to increase the agency [20:40] over versions right and in order to [20:42] understand what the capability should be [20:45] collect a reference data set that shows [20:47] what good behavior could look like and [20:49] then set up the application um and you [20:51] set it up with the simplest architecture [20:53] that's needed to support the capability [20:55] and three is to design eval right create [20:57] custom evaluation metrics which could [20:59] help you track performance and [21:01] accurately catch failure points. That's [21:03] these three things are essentially what [21:05] most people would do which is the [21:06] development process. The fun part [21:09] happens after your deployment, right? [21:10] Like once you've thought about the [21:12] continuous development or the [21:14] development part, you can go out there [21:16] and deploy your product to a small user [21:18] cohort. But what you do after that is a [21:20] lot more important, which is the [21:22] calibration, right? You think about how [21:24] you could run your evals for a subset of [21:26] the data for a subset of the real user [21:29] interaction so that you can kind of walk [21:30] backwards and understand what to [21:32] improve. Then you analyze recurring [21:34] failure modes and behavior patterns in [21:36] those logs and make targeted changes [21:39] that evolve into your next version, [21:40] right? And kind of do this until you get [21:43] to a point where a particular iteration [21:46] completely functions well and you're [21:48] also scoping features in a way that you [21:50] increase agency over time. Right? If all [21:52] of this feels a little theoretical, I [21:54] can give you a few examples of what this [21:56] could mean. Right? Let's say you're [21:58] trying to build a marketing assistant [22:00] for, you know, your enterprise. Your [22:03] different iterations moving from, you [22:05] know, high control to higher agency [22:08] would be your Wii1 could be something [22:09] like just draft an email ad or social [22:11] copy from prompts and then have a human [22:13] review them and publish them, right? And [22:16] your Wii 2 could be to build a [22:17] multi-step campaign and run them. Now, [22:20] you're giving your system a little more [22:22] agency. and your W3 could be to launch, [22:24] do AB testing and autooptimize campaigns [22:26] across channels. So you you can kind of [22:28] see how you're slowly increasing the [22:30] autonomy and all through the process [22:32] having um humans so that they could [22:34] guide the process and also provide [22:35] feedback and if um folks have used [22:38] coding assistants like GitHub copilot or [22:41] even cursor clot code and all of these [22:43] cloud code um and codeex are probably [22:46] new um and they came up with a lot of [22:47] loaded features but if you've been in [22:49] this space for a while this is exactly [22:52] how GitHub copilot evolved right um [22:54] maybe two years ago you would see coding [22:56] assistants just suggest inline [22:57] completions and boilerplate snippets for [22:59] you. And then slowly they started [23:01] building up to generate larger blocks of [23:03] codes um for human review. And then now [23:06] you can see coding assistants that can [23:08] completely applied scope changes for [23:09] you. They can pull um they can submit [23:11] pull requests. They can build [23:12] autonomously. Right? This is exactly the [23:14] playbook that a lot of large companies [23:16] or coding assistants used in order to [23:17] build their products. Right? [23:20] So um so essentially think of those [23:23] versions that could help you slowly [23:25] increase autonomy so that you're [23:27] observing the patterns that exist. Um I [23:31] have an example of a fully autonomous [23:33] customer support and how you could build [23:35] it. I will just go through um a few [23:39] slides since we're running out of time. [23:41] Um this this is a real example that we [23:44] built. uh we have anonymized the [23:46] customer that we built it for but [23:48] essentially we built three versions of [23:50] the customer support agent. Remember I [23:52] said one of those times when we tried it [23:54] we had to shut it down in 5 hours. This [23:56] was our second attempt to it right. We [23:58] first built just a routing assistant [24:00] which would route the ticket into the [24:02] right department and we would have logs [24:05] of whether it was routed to the right [24:06] department. Did a human actually change [24:09] change the department and things like [24:10] that. Right? that was very high control [24:12] and low agency. And this gave us a lot [24:14] of information about is the agent able [24:16] to understand where a particular query [24:18] should be routed. And then we built a [24:21] co-pilot which would route and look up [24:23] relevant standard operating procedures [24:25] and pass replies and draft a reply for a [24:28] real human agent in order to answer the [24:29] question. Right? It would still have a [24:31] human in the loop but it would only make [24:33] it easier for the human that's um [24:36] answering these questions. This was like [24:37] medium control and medium agency. And [24:39] then we graduated to the fully um um [24:43] functioning resolution assistant which [24:45] would draft and autoresolve common [24:47] tickets using retrieve context right and [24:49] all through this we had a good um set of [24:52] evaluation metrics and also logs to [24:54] understand what changes were happening [24:56] and we built into different parts of [24:58] those um CCCD loop in order to improve [25:00] the system. Um I will show you a quick [25:08] final product. So essentially we ended [25:11] up building a data flywheel where the W1 [25:14] could basically um determine if the [25:17] system can classify and route tickets [25:19] accurately. And what this helps you [25:21] learn for the next iteration is how do [25:24] users describe their problems? Which [25:26] departments are hard to distinguish? [25:27] What metadata is actually useful? So it [25:30] helps you do a lot of that context [25:31] engineering into your system and you [25:33] feed that into your next loop which is [25:35] you build that out and then think about [25:36] your next feature. Um you essentially [25:40] have prompt clarity for routing [25:41] decisions. You have clean routing data [25:44] and then you enter your co-pilot phase, [25:46] right? Where you think about can the [25:48] system actually retrieve context and [25:50] help you know draft replies and then you [25:53] also log information about what human [25:55] agents actually use or ignore, where [25:56] retrieval fails, what gaps or [25:58] inconsistencies are there in the [26:00] standard operating procedures and you [26:01] can use that to then build into your [26:03] third iteration. Right? The third [26:04] iteration is fully autonomous and then [26:06] you could still have learnings on where [26:08] autonomy causes trust breakdowns, what [26:10] tasks still require human escalation. [26:13] Remember although customer support seems [26:15] like a very ripe use case for complete [26:17] automation, there are still a lot of use [26:20] I mean scenarios where you want to [26:22] escalate to a human. So we also you know [26:25] bake that into our system and things [26:26] like that. So essentially once you build [26:28] a system that gradually increases [26:30] autonomy and log everything that happens [26:33] you you're building a data flywheel for [26:34] yourself so that your agent or the [26:36] intern is slowly graduating and you're [26:39] not losing the trust of your users. Um I [26:42] will stop there and um these slides are [26:44] there for your review and we also have a [26:46] post um a complete um a longer post on [26:50] this on Lenny's newsletter that is also [26:52] available for you to check out. So the [26:55] take the takeaway here is let your AI [26:57] system earn its agency through constant [26:59] calibration. Do not grant it prematurely [27:01] and build your iteration such that the [27:03] user experience is not um impacted while [27:06] you you know improve your AI system.  ## 📺 Video ![](https://www.youtube.com/watch?v=GAF_ychy32k) --- # 📝 Full Notes (Deep-Read Analysis) # 🏷️ Title: Stop Building AI Like Traditional Software ## 🧰 Action Inbox - [H1] Implement CCCD framework for next AI project — start with low agency, high control #ai-architecture - [H2] Design custom evaluation metrics beyond traditional unit tests — include LLM judges for natural language outputs #evaluation - [H3] Plan incremental agency increases across product versions — avoid full automation on day one #product-strategy ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [Original Video](https://www.youtube.com/watch?v=GAF_ychy32k) - [Lenny's Newsletter Post](https://www.lennysnewsletter.com) (referenced) ## 📝 Summary - Former AWS AI researcher shares hard lessons from building 30+ enterprise AI solutions - Traditional software and ML practices fail when building AI agents that interact with users through natural language - Introduces "Continuous Calibration, Continuous Development" (CCCD) framework as alternative to CI/CD - Key insight: Start with high control/low agency, gradually increase autonomy as AI system proves reliability - Real case study of customer support agent that failed in 5 hours demonstrates high stakes of automation ## 📖 Main Content Aishwarya Reganti, an AI researcher with 35+ papers and former AWS tech lead, argues that building AI systems requires fundamentally different approaches than traditional software development. Drawing from experience building 30+ enterprise AI solutions at AWS's GenAI Innovation Center, she reveals why standard CI/CD practices fail for AI systems. The core problem: AI systems interact through unpredictable natural language rather than deterministic UI elements, and they generate free-form responses that can't be evaluated with traditional unit tests or ML classification metrics. This creates an "agency-control tradeoff" where giving AI more autonomy reduces human oversight. Her solution is Continuous Calibration, Continuous Development (CCCD), which starts AI systems with minimal agency and human oversight, then gradually increases autonomy as the system proves reliable through real user interactions. This prevents the trust breakdowns that occur when fully autonomous AI systems fail in production. ## 🗒️ Notes #### ⏱️ [00:00](https://youtu.be/GAF_ychy32k?t=0) — [H1] Introduction to CCCD Framework - Key idea: Traditional software development approaches fail for AI systems due to natural language unpredictability - Evidence: "Stop building AI like traditional software" - need new frameworks for AI development #### ⏱️ [03:24](https://youtu.be/GAF_ychy32k?t=204) — [H2] Speaker Background and Context - Key idea: Speaker has both research (35+ papers) and enterprise experience building AI solutions - Evidence: "Early member at Alexa research team at Amazon" and "tech lead for GenAI innovation center" #### ⏱️ [05:12](https://youtu.be/GAF_ychy32k?t=312) — [H2] Common Enterprise AI Use Cases 2023 - Key idea: Two dominant patterns emerged - "find anything fast" (RAG) and "messy data into insights" (text-to-SQL) - Evidence: "Companies wanted a ChatGPT over their ecosystem" and "text to SQL or text to code" #### ⏱️ [07:45](https://youtu.be/GAF_ychy32k?t=465) — [H2] Team Backgrounds Building AI Solutions - Key idea: Teams combined software engineers (CI/CD, APIs, scaling) with ML/data science experts (evaluation, feature engineering) - Evidence: "No AI engineers or AIPMs... came from two backgrounds: software and machine learning" #### ⏱️ [09:18](https://youtu.be/GAF_ychy32k?t=558) — [H1] First Critical Lesson: Unpredictable User Interactions - Key idea: Natural language interactions are fundamentally different from predictable UI clicks and forms - Evidence: "Users will interact in surprising ways that you cannot predict up front" #### ⏱️ [11:30](https://youtu.be/GAF_ychy32k?t=690) — [H1] Second Critical Lesson: AI Evaluations Need Redesign - Key idea: Traditional unit tests and ML metrics (precision/recall) don't work for natural language generation - Evidence: "You cannot rely on traditional software tests... neither traditional ML metrics like precision recall" #### ⏱️ [13:42](https://youtu.be/GAF_ychy32k?t=822) — [H2] Real Example: Loan Officer Assistant Surprises - Key idea: Underwriters asked unexpected questions requiring historical data and cross-referencing capabilities - Evidence: "What exceptions did regulators flag for large loans... we did not have historical view" #### ⏱️ [15:25](https://youtu.be/GAF_ychy32k?t=925) — [H2] 2024 Mindset Shifts: Iterations and Smart Metrics - Key idea: Build smaller iterations to understand user behavior quickly, create custom evaluation metrics - Evidence: "Run small measurable iterations... define clear success metric whether user feedback, task completion" #### ⏱️ [18:10](https://youtu.be/GAF_ychy32k?t=1090) — [H2] Key Mental Model: AI System as Intern - Key idea: AI won't be perfect day one regardless of groundwork - needs training and feedback loops - Evidence: "Think of your AI system as an intern. It will not be perfect on day one" #### ⏱️ [20:33](https://youtu.be/GAF_ychy32k?t=1233) — [H1] Evolution to AI Agents in 2024 - Key idea: Enterprise demand shifted from chat systems to end-to-end workflow automation - Evidence: "Can you automate entire back office workflows... instead of just answering questions" #### ⏱️ [23:47](https://youtu.be/GAF_ychy32k?t=1427) — [H1] Customer Support Agent Failure Case Study - Key idea: Well-built agent shut down in 5 hours due to error cascades and user frustration - Evidence: "AI agent was shut down in 5 hours... wrong routing... agent would spiral into nonsense" #### ⏱️ [26:15](https://youtu.be/GAF_ychy32k?t=1575) — [H1] Agency-Control Tradeoff Discovery - Key idea: Giving AI more autonomy reduces human control - must build trust before granting agency - Evidence: "Every time you give AI agent more agency... you as developer are losing control" #### ⏱️ [28:40](https://youtu.be/GAF_ychy32k?t=1720) — [H1] CCCD Framework Introduction - Key idea: Start with high control/low agency, gradually increase autonomy through proven performance - Evidence: "Start building solutions that are very low agency first and give higher agency over time" #### ⏱️ [31:22](https://youtu.be/GAF_ychy32k?t=1882) — [H2] CCCD Development Loop Details - Key idea: Three steps - collect reference data, simple architecture, design custom evaluations - Evidence: "Start with small high control and low agency capability... design eval with custom metrics" #### ⏱️ [33:55](https://youtu.be/GAF_ychy32k?t=2035) — [H2] CCCD Calibration Loop Details - Key idea: Deploy to small cohorts, run evals on real interactions, analyze failure patterns iteratively - Evidence: "Run evals for subset of real user interaction... analyze recurring failure modes" #### ⏱️ [36:12](https://youtu.be/GAF_ychy32k?t=2172) — [H2] Marketing Assistant Example: Agency Progression - Key idea: V1 draft content for human review, V2 multi-step campaigns, V3 autonomous optimization - Evidence: "Draft email... human review... multi-step campaign... AB testing and auto-optimize" #### ⏱️ [38:30](https://youtu.be/GAF_ychy32k?t=2310) — [H2] GitHub Copilot Evolution Pattern - Key idea: Coding assistants followed same pattern - inline suggestions to autonomous pull requests - Evidence: "Inline completions... larger code blocks... completely scoped changes... submit pull requests" #### ⏱️ [40:45](https://youtu.be/GAF_ychy32k?t=2445) — [H2] Customer Support Success Case: Three Versions - Key idea: V1 routing assistant, V2 co-pilot with SOPs, V3 autonomous resolution with escalation - Evidence: "Route ticket to right department... draft reply for human... auto-resolve common tickets" #### ⏱️ [43:18](https://youtu.be/GAF_ychy32k?t=2598) — [H1] Data Flywheel Creation - Key idea: Each version generates insights for next iteration - routing accuracy, retrieval gaps, escalation needs - Evidence: "How users describe problems... what human agents use or ignore... where autonomy causes trust breakdowns" ## 📚 References - [Stop Building AI Like Traditional Software](https://www.youtube.com/watch?v=GAF_ychy32k) - [Lenny's Newsletter - CCCD Framework](https://www.lennysnewsletter.com) - [Alan Black - CMU Professor](https://www.cs.cmu.edu/~awb/) ## 🗣️ Paraphrase - AI systems require different development approaches than traditional software because users interact through unpredictable natural language rather than deterministic interfaces - The CCCD framework starts AI systems with minimal autonomy and human oversight, gradually increasing agency as reliability is proven through real user interactions - Real-world case studies show that fully autonomous AI can fail catastrophically in production, requiring careful progression from co-pilot to autonomous modes ## 📰 Newsletter **Game-Changing Framework for AI Development: Stop Building Like Traditional Software** A former AWS AI researcher just shared a framework that could save your next AI project from catastrophic failure. After building 30+ enterprise AI solutions, Aishwarya Reganti discovered why traditional CI/CD approaches fail for AI systems. **The Problem**: AI systems interact through unpredictable natural language, not deterministic UI clicks. When a customer support AI agent failed within 5 hours of deployment, it revealed the critical "agency-control tradeoff." **The Solution**: Continuous Calibration, Continuous Development (CCCD) - start with high control/low agency, gradually increase autonomy as the system proves reliable. **Practical Application**: Instead of launching a fully autonomous AI agent, build V1 as routing assistant, V2 as co-pilot, V3 as autonomous system with escalation paths. Resources: [AI Development Best Practices](https://learn.wordpress.org/), [System Architecture](https://developer.wordpress.org/), [Performance Monitoring](https://wp-cli.org/) ## ✅ Action Items #### DevRel - Developer advocacy: Create workshops on CCCD framework vs traditional CI/CD for AI systems - Education: Build curriculum on AI evaluation metrics beyond unit tests and ML classification - Developer experience: Document agency-control tradeoff patterns in AI system design - Developer marketing: Case studies showing graduated autonomy in successful AI products - Community management: Forums for sharing AI system calibration experiences #### OSPO - Policy & Governance: Guidelines for AI system deployment with human oversight requirements - License Compliance: Review AI model licenses for enterprise automation use cases - Security Management: Escalation procedures when AI systems require human intervention - Software Inventory: SBOM tracking for AI agent dependencies and evaluation frameworks - Developer Support: Training on CCCD implementation and natural language evaluation - Community Engagement: Fund research on AI system reliability and trust metrics - Strategic Alignment: KPIs for measuring AI agency progression and user trust #### Community - Maintainers: Templates for AI system evaluation and calibration workflows - Contributors: Guidelines for contributing to AI evaluation frameworks and metrics - Sponsors: Fund development of open-source CCCD implementation tools - Ecosystem: Cross-project standards for AI system agency progression #### WordPress - Core AI: Implement CCCD principles for WordPress AI features, graduated autonomy rollout - Core Performance: Monitor performance impact of AI evaluation systems - Community: Workshop materials on building AI features for WordPress plugins/themes - Training: Curriculum on AI system design patterns and evaluation methods - Documentation: Guidelines for plugin developers building AI features - Plugins: Review process updates for AI-powered plugins requiring user interaction - Accessibility: Ensure AI system outputs meet WCAG standards for generated content ## 🧪 Hypotheses - AI systems that start with low agency and gradually increase autonomy will have higher user trust and retention than fully autonomous systems deployed immediately - Natural language evaluation metrics (LLM judges) will provide more accurate assessment of AI system performance than traditional software testing approaches - The agency-control tradeoff applies universally across AI applications, requiring calibration periods regardless of domain - Data flywheels created through CCCD iterations will produce higher quality AI systems than traditional ML training approaches ### 📊 Methodology - Study design: Compare user satisfaction and system reliability between CCCD-developed AI systems vs traditional deployment - Data sources: User interaction logs, escalation rates, task completion metrics across multiple iterations - Experiments: A/B test graduated autonomy rollouts vs full automation in controlled enterprise environments ### ❓ Questions - How long should calibration periods be before increasing AI system agency across different domains? - What are the optimal evaluation metrics for natural language AI system outputs beyond human feedback? - How can the agency-control tradeoff be quantified and measured systematically? ### 🔮 Future Research - Standardized frameworks for measuring AI system readiness for increased autonomy - Cross-industry benchmarks for AI agent calibration periods and success metrics - Automated tools for implementing CCCD workflows in existing development pipelines ### 💡 Implications - Practice: Development teams need new skills in AI evaluation design and natural language system testing - Policy: Regulatory frameworks may need to account for graduated AI autonomy rather than binary automation decisions - Security: AI systems require different security models that account for unpredictable natural language interactions - Education: Computer science curricula should include AI-specific development methodologies beyond traditional software engineering - Funding: Investment in AI startups should account for longer calibration periods and iterative deployment strategies ### 🧠 Learning Styles Assessment Reading level estimate: 12-14 (Graduate level due to technical AI concepts) #### 👀 Visual learners - Present: Agency-control tradeoff diagram, CCCD loop illustrations - Improve: Architecture diagrams showing system evolution, user interaction flowcharts #### 🔊 Aural learners - Present: Video presentation with clear narration - Improve: Podcast-style discussion of case studies, audio summaries #### 🗣️ Verbal learners - Present: Clear definitions of CCCD, agency-control tradeoff, calibration concepts - Improve: Analogies comparing AI development to traditional software, glossary #### 🧩 Kinesthetic learners - Present: Step-by-step CCCD implementation examples - Improve: Interactive workshops, hands-on AI evaluation exercises #### 🧠 Logical learners - Present: Clear framework structure, case study analysis, failure mode breakdown - Improve: Decision trees for agency increases, quantified metrics #### 🧑‍🤝‍🧑 Social learners - Present: Team collaboration examples, enterprise case studies - Improve: Discussion forums, peer review processes #### 🧘‍♀️ Solitary learners - Present: Individual --- # 📄 Full Transcript *Auto-generated from YouTube captions · timestamps roughly every 30 seconds · [watch on YouTube](https://www.youtube.com/watch?v=GAF_ychy32k)* **[00:00]** Thanks everyone for making it to the session today. Um my talk is titled stop building AI like traditional software and you'll kind of see why I'm getting to that point. Um let's get started. Um so today I'll be introducing something called continuous calibration, continuous development. And um before I kind of introduce that concept which I have picked up over the past few years of building AI products, I will also **[00:31]** introduce you to why that mindset came around and why a lot of people think of building AI products as traditional software and then they forget the key differences that AI systems have. And I'll kind of build um the story from there and also give you a lot of backing as to why we came up with this concept of continuous calibration, continuous development. A little bit about me um thanks for the wonderful introduction but um a little more about my background is I was an AI researcher in my past life. I have about **[01:02]** 35 plus research papers in top AI conferences and um my adviser was Alan Black and if you come from the machine learning world from the NLP world before LLM's happened um Alan Black was a popular professor um anyway I was also an early member at the Alexa research team at Amazon and also worked on search and recommendations at Amazon using graph neural networks. Uh over the past two years um I worked at um AWS as a tech lead for a org that we call Genaii **[01:34]** innovation center where we build AI solutions using our existing um um uh bedrock which is our um AI offering and just about 4 months ago I left AWS and now I run an AI consulting firm and the whole idea is to build AI solutions for companies and also do it with a very forward deployed mindset. um which means we enter your company, we start um understanding your ecosystem and also build solutions for you. Um more about it you can find on my **[02:04]** LinkedIn but that's kind of my background. I I have both of a research as well as an enterprise background. [snorts] Um I will walk you through a bunch of lessons from 2023 to 2024 and then 2024 to later and which will help build up why we came up with continuous calibration and continuous development, right? Um and remember in 2023 to early 2025 I was part of AWS. Um and I was a tech lead at Genai innovation center and these are all the lessons I learned from **[02:34]** building about 30 plus use cases while at AWS. Um 2022 November was when Chptd came out. Um people were going gaga about it. Um a lot of um consumer use cases came into picture. People were chatting with Chad GPT and it was seeming super fun. But then enterprises also started thinking how can we use this for our own use cases. Um having something like a charge is good but then it doesn't really serve our data sets or our ecosystem. So what can we do about it? And that's when um we saw a lot of um **[03:08]** common use cases. Two big use cases that we saw for enterprises were um find anything fast, right? Which essentially companies wanted had a lot of unstructured data sets and structured data as well and they wanted a chat GPT over their ecosystem, right? And that's kind of the very common use case that we saw in 2023. Um the fancy term for it, you probably heard of it, it's called retrieval augmented generation or rag. um essentially means pull up information from our knowledge base, summarize it for us and answer questions. And the **[03:40]** second common use case that we saw was um on turning messy data into insights, right? Companies have a lot of SQL databases, relational databases and they wanted methods to extract data or to transform that data and build analytics. It was previously done through code, but now um folks wanted things like text to SQL or text to code so that they could have um a lesser uh bar for people who could actually you know talk with their data and not really have um you know **[04:11]** domain experts or data scientists work on this. So these were very um common use cases that we saw in 2023. Uh there are a bunch of public examples that I worked on. Uh we worked with a um um ISV called imply data where we built a chat with your data system on druid SQL which is their version of SQL. Uh we also built a geoscience co-pilot for Hallebertton and also built a loan officer assistant for another global bank. So these are the kind of use cases that we kept seeing. Um now remember **[04:42]** that in 2023 there were no AI engineers or AIPMs or all of those AI roles that we talk about today. A lot of the people who started building, a lot of the teams that started building these solutions essentially came from two backgrounds, right? One of them was software. They were software engineers who understood, you know, system architectures. Well, they knew how to build CI/CD pipelines. They understood how to design APIs, scaling, DevOps and all of the software practices, right? And then there was folks like me who came from the machine learning or data science uh background. **[05:13]** And what we knew was how do you play around with your data? Do you know if your data is ready? what kind of valuation metrics should you be building? How do you build observability? How do you do feature engineering and things like that, right? And both of um these backgrounds kind of our team also consisted of uh folks in the um these backgrounds and we came together to you know actually build generative AI solutions and we figured out that the extra thing that you need to be learning um was to you know understand prompt engineering because that was a new field that um you know **[05:43]** both ML and software folks had um hadn't really played around with a lot and then more around you know as rag became more mature there were vector DBs and stuff like This was pretty cool because we had both the software and machine learning background and all we had to do is augment that with the latest generative AI principles, right? Um uh and then we did about 40 use cases that year from 2023 to 2024. Um and having come from both of these backgrounds, we learned **[06:14]** two important lessons, right? And one of those lessons was um one is that users will interact in surprising ways that you cannot predict up front and that AI evaluations need deeper design right remember that if you're a software engineer or even you work with machine learning a lot of user behavior is very um on a high level it's kind of predictable because you have your users you probably have a UI which has you know clicks buttons options forms and things like that and you have a backend connected to that right which is **[06:44]** probably connected to servers databases and stuff like that. Now with AI systems, your interaction layer becomes a lot more um ambiguous in the sense that users are not um dealing with, you know, menu buttons or clicks and things like that. They're dealing with um natural language interaction. You can see the um figure on the right where you're probably having voice or chat or video and things like that which makes things a lot more harder to predict um and kind of design for, right? You're **[07:15]** not designing for guaranteed behavior. You're designing for anticipated behavior. And that was a huge learning for us when we started building these systems because you really need to think spend the time to think about how users will interact with your system. what kind of sentences they might come with and just the fact that each of us communicate very differently. So there has to be that uh understanding of how users would communicate with your product. And the second lesson we learned is that um evaluations need **[07:45]** deeper design. Again remember even the uh even what the AI model predicts can be is natural language or it's probably voice and things like that. So you really need much smarter ways of evaluating if a use case was successful. You cannot rely on traditional software tests like unit tests or integration tests and things like that. Neither can you rely on very traditional machine learning metrics like precision recall and stuff like that because those were designed with classification or regression in mind. Right now you have **[08:17]** both users interacting with your systems in very unpredictable ways and the AI model is also generating um you know responses that are much more free flow than before. So these were two important lessons that we learned right. Um one example is when I said we were building for a global bank we were building a loan officer assistant. Turns out that the underwriters who were using this assistant um started asking questions that did we did not anticipate, right? For instance, they said, "What **[08:47]** exceptions did regulators flag for large loans in this section?" Um and we just had a rag system that was built on top of policies and we did not have a historical view of the data and they were also trying to cross-check proceed and style cases like lawyers and things like that, right? And we figured out that there was data in incompleteness in order to serve these features. And even as just a chat GPT user, you will kind of see that maybe from 2023 to now the kind of questions or the kind of queries that you've asked Chad GT Chad GPT or you know Claude or Gemini or whatever **[09:18]** this is has evolved significantly, right? The patterns of interaction have evolved and we need to as builders account for that evolution. Um so in 2024 we thought okay um that makes sense those are some important lessons and we said um from now on we will be building solutions with quicker iterations so that we understand user behavior as quickly as possible and we can evolve our system to do that and we'll also think of much smarter evaluation metrics because older metrics no longer work right these were like two **[09:50]** mindset shifts we had which is you know run small measurable iterations in order to gather user behavior quickly don't wait for your final product to be out there. Build smaller iterations so that you know how users are interacting with your system. And the other thing was again define a very clear success metric whether that is user feedback, task completion and along the way check for intermediate signals as well. Right? If you're building a text to SQL system, you could have a check in between to just make sure that the syntax of the SQL makes sense. You could also um you know if you have tool calling agents and **[10:22]** things like that you could have intermediate check make sure that the right tool was called um then looking at the entire you know final um response of whether it was successful or not try to have intermediate metrics as well. If you're building a rack pipeline can you think about how good the retrieval was and stuff like that. So think of ways to evaluate your pipeline um break it down instead of looking at the final action right and then you know test all of these components separately and stuff and um we also figured that since these metrics had to be so open-ended as **[10:53]** compared to traditional machine learning metrics or software metrics um we also started building uh LLM judges or AI judges um that could replace human judgment and could you know scale human judgment right um and if if I To summarize this in a single statement, it would um just be that um the biggest mindset shift for us was that think of your AI system as an intern. It will not be perfect on day one. Regardless of how much groundwork you do to build this system, you will see that users will **[11:23]** always come up with behavior patterns that you never expected. So it won't be perfect on day one, but think of ways to build iterations and to also collect feedback so that you can iterate quickly and get to a point where it starts performing well, right? Um and even as we build these solutions for customers um in early stages when it went to production we would always iterate you know weekly we would look at the data weekly we try to improve our systems weekly and as we started re reaching a point of convergence where we knew a lot of the user patterns that's when we **[11:55]** extended that iteration to you know maybe monthly or bimonthly cycle so that um you know um the the product was stable enough right so that was one huge mindset shift that we made we were in the middle of 2024 and we picked up these two new you know concepts which was AI evaluations and also shorter iterations and we we were doing pretty well and that's when um the whole world started talking about this new concept right which is um AI agents um from 2023 **[12:26]** to 2024 a lot of the enterprises their mindset around AI was very chat like chat system like or you know rag and things like that but And in late not even late maybe early 2024 people started thinking a lot more about not just chat systems but also agents that could perform tasks end to end. Now there's a lot of debate on what agents mean. Um but um um at the high level it just refers to systems that can perform tasks end to end and um basically **[12:57]** complete workflows right um and then we saw that the kind of use cases that we started getting from enterprises looked a lot more different in 2024 right instead of them coming up with things like can you build an internal search system for us can you build an insights dash I mean can you build a text to um code kind of system all of their requests were more of can you automate entire back office kind of workflows for us because now you have agents, you have the mindset of agents that could call other tools that could have memory and things like that, right? And some of the **[13:27]** common use cases that we saw back in um you know late 2024 to 2025 was um things like customer support automation or um for the global bank that we worked on, they wanted a more mature use case where we were um also doing loan underwriting assistant. instead of just answering questions, it could also help you do the underwriting and could kind of fill in some parts so that um uh underwriters could spend much less time on doing those laborious things, right? Um so we were like, cool. I mean, we've **[13:57]** already done this. Um there's not a huge difference. It's pretty much the same pipeline, just that the scope of work increases. So we're like, okay, we're the best team that exists. All of you will get agents. and uh enterprises were you know kind of super enthusiastic about the concept and were like we we've got this right [gasps] but then um and remember I also went through these extra things that we had picked up in that one year uh we were all from a machine learning or software background and we had picked up all of these things about evaluations about iterations prompt engineering vector DBs and we like okay **[14:28]** we're experts at this right um and then we built a customer support agent for a company um unfortunately I won't be able to diverge the name but we built it for a retail company and the idea was to classify tickets to the right department and to also retrieve resolution documents or standard operating procedures or SOPs and provide an answer and follow up on user versions typically what your customer support agent would do but then automating that entire workflow right and we kept those two principles that we kept in mind like were you know build iterations deploy early and gather user data and also have **[15:00]** good eval so that you know where your system is going wrong. So we were like okay this is this is pretty good and we put it into production and then boom we deployed it to a very small cohort of course not to um you know everybody all the customers that these folks had but the AI agent was shut down in 5 hours and we had to spend a lot of time doing damage control right um we had the dream team we built a wonderful solution but why did it fail um there were three reasons one was we had limited initial data on how users interacted what kind of questions that they would ask the **[15:30]** system but this was expected right we kind knew that it is impossible to get all that data up front and we had in our mind that we're going to build iterations. Um but then another thing that happened is we the error started rippling because of a wrong routing it would pick up the wrong um SOP and then agent would spiral into nonsense and users were frustrated right remember that when you're building your iterations you want to build them in a way such that you don't frustrate your users we didn't think of that we thought okay we're going to do error analysis we have eval in place and we're going to **[16:01]** build a better version each time but what happens during that time when your agent is not up to the mark yet you cannot really put it in front of customers and that's kind of the issue that we saw which is users were getting super frustrated and they were demanding that real human agents come in and um these human agents who were there faced a lot of backlash from angry customers because there were no humans in the loop right and that's why it had to shut it down and that's kind of um where where I come to this point of the blind spots we actually ended up missing right um from **[16:32]** 2024 or 2023 to 2025 the huge mindset shift that people had um or something that we didn't really you know take into account was that use cases were moving from augmentation to automation which means there were much higher stakes just having the right set of eval or building the right iterations were not enough and full automation could quickly break user trust right as soon as you put these AI systems in front of customers you want to make sure that they have a seamless experience as before and we did not think about that and that's where we **[17:03]** came up with this concept of agency control tradeoff which means that every time you give your AI agent more agency or like more autonomy to do something to take decisions you as a human or you as you know the developer are losing control right and you want to make sure that when you're losing that control you've your AI agent has built that trust has built enough trust to do it right um and we also learned that domain experts remain super critical until the **[17:33]** intern or the AI agent matures and you know decisions must be handed off to real people if things go wrong, right? Um, so on paper, agentic use cases could be built in a similar way as rag or as other um, you know, chat kind of systems, but then remember that people are not super frustrated when they ask chat GPT for a question and it comes up with the wrong answer because they're like maybe doesn't know about about something, right? I'll ask it in a different way. But if you go to a customer support system and you ask a question and you're trying to get a **[18:03]** resolution and the support bot starts saying, you know, some nonsense, you get a lot more frustrated because we expect that support systems work a certain way. We are not ready to um, you know, take any um, you know, mishaps the same way we would do with chat GPT because that's just a chat system. With a support system, you actually want uh, you're working towards an outcome. You want a resolution as quickly as possible. And that's also the mindset shift that we kind of missed. And that's where we came up with two new learnings from 2025. Um and um a lot of **[18:36]** this builds on top of what we learned in 2024. But essentially AI systems are non-deterministic and you're designing for likely behavior, not guaranteed behavior. And that's where you know we had all of these things around quicker iterations, evalu agency control tradeoff, right? which is if you let AI systems make decisions for you, then be sure they're well tested and do not automate fully on day one. Try to make sure that there are humans in the loop. There are points where seamless switching can happen so that **[19:07]** your user experience doesn't get affected. This is one of the biggest um uh reason of coming up with the CCCD pipeline and that's where we um put together this framework um uh which is called continuous calibration and continuous development and I built this framework from the learnings that we had both from consulting as well as working for AWS and we use a lot of this when we build for customers as well. The idea is pretty simple. It's to come out of your mindset of a CI/CD kind of an approach, **[19:39]** but think of an AI system as something that needs continuous calibration. Um, and you need you're continuously sampling behavior in order to calibrate, right? Um and the iterations concept still remains the same but we say that when you're building solutions AI solutions start building solutions that are very low agency first and give higher agency to your system over time right you kind of see this diagram where you say as you go through it increase the agency and lower the control don't **[20:09]** give super high autonomy or agency to your system on day one right and it has two parts of the loop uh one of it is continuous calibration and the other is continuous is development and I'll just walk you through each of these. It's pretty simple at the face um but starts to make a lot of sense when you start building AI systems, right? Um the continuous development side of the loop basically says start with a small high control and low agency capability for your AI product whatever that is and plan to increase the agency **[20:40]** over versions right and in order to understand what the capability should be collect a reference data set that shows what good behavior could look like and then set up the application um and you set it up with the simplest architecture that's needed to support the capability and three is to design eval right create custom evaluation metrics which could help you track performance and accurately catch failure points. That's these three things are essentially what most people would do which is the development process. The fun part happens after your deployment, right? **[21:10]** Like once you've thought about the continuous development or the development part, you can go out there and deploy your product to a small user cohort. But what you do after that is a lot more important, which is the calibration, right? You think about how you could run your evals for a subset of the data for a subset of the real user interaction so that you can kind of walk backwards and understand what to improve. Then you analyze recurring failure modes and behavior patterns in those logs and make targeted changes that evolve into your next version, right? And kind of do this until you get **[21:43]** to a point where a particular iteration completely functions well and you're also scoping features in a way that you increase agency over time. Right? If all of this feels a little theoretical, I can give you a few examples of what this could mean. Right? Let's say you're trying to build a marketing assistant for, you know, your enterprise. Your different iterations moving from, you know, high control to higher agency would be your Wii1 could be something like just draft an email ad or social copy from prompts and then have a human **[22:13]** review them and publish them, right? And your Wii 2 could be to build a multi-step campaign and run them. Now, you're giving your system a little more agency. and your W3 could be to launch, do AB testing and autooptimize campaigns across channels. So you you can kind of see how you're slowly increasing the autonomy and all through the process having um humans so that they could guide the process and also provide feedback and if um folks have used coding assistants like GitHub copilot or even cursor clot code and all of these cloud code um and codeex are probably **[22:46]** new um and they came up with a lot of loaded features but if you've been in this space for a while this is exactly how GitHub copilot evolved right um maybe two years ago you would see coding assistants just suggest inline completions and boilerplate snippets for you. And then slowly they started building up to generate larger blocks of codes um for human review. And then now you can see coding assistants that can completely applied scope changes for you. They can pull um they can submit pull requests. They can build autonomously. Right? This is exactly the playbook that a lot of large companies **[23:16]** or coding assistants used in order to build their products. Right? So um so essentially think of those versions that could help you slowly increase autonomy so that you're observing the patterns that exist. Um I have an example of a fully autonomous customer support and how you could build it. I will just go through um a few slides since we're running out of time. Um this this is a real example that we built. uh we have anonymized the **[23:46]** customer that we built it for but essentially we built three versions of the customer support agent. Remember I said one of those times when we tried it we had to shut it down in 5 hours. This was our second attempt to it right. We first built just a routing assistant which would route the ticket into the right department and we would have logs of whether it was routed to the right department. Did a human actually change change the department and things like that. Right? that was very high control and low agency. And this gave us a lot of information about is the agent able to understand where a particular query **[24:18]** should be routed. And then we built a co-pilot which would route and look up relevant standard operating procedures and pass replies and draft a reply for a real human agent in order to answer the question. Right? It would still have a human in the loop but it would only make it easier for the human that's um answering these questions. This was like medium control and medium agency. And then we graduated to the fully um um functioning resolution assistant which would draft and autoresolve common tickets using retrieve context right and **[24:49]** all through this we had a good um set of evaluation metrics and also logs to understand what changes were happening and we built into different parts of those um CCCD loop in order to improve the system. Um I will show you a quick final product. So essentially we ended up building a data flywheel where the W1 could basically um determine if the system can classify and route tickets accurately. And what this helps you **[25:21]** learn for the next iteration is how do users describe their problems? Which departments are hard to distinguish? What metadata is actually useful? So it helps you do a lot of that context engineering into your system and you feed that into your next loop which is you build that out and then think about your next feature. Um you essentially have prompt clarity for routing decisions. You have clean routing data and then you enter your co-pilot phase, right? Where you think about can the system actually retrieve context and help you know draft replies and then you **[25:53]** also log information about what human agents actually use or ignore, where retrieval fails, what gaps or inconsistencies are there in the standard operating procedures and you can use that to then build into your third iteration. Right? The third iteration is fully autonomous and then you could still have learnings on where autonomy causes trust breakdowns, what tasks still require human escalation. Remember although customer support seems like a very ripe use case for complete automation, there are still a lot of use I mean scenarios where you want to escalate to a human. So we also you know **[26:25]** bake that into our system and things like that. So essentially once you build a system that gradually increases autonomy and log everything that happens you you're building a data flywheel for yourself so that your agent or the intern is slowly graduating and you're not losing the trust of your users. Um I will stop there and um these slides are there for your review and we also have a post um a complete um a longer post on this on Lenny's newsletter that is also available for you to check out. So the **[26:55]** take the takeaway here is let your AI system earn its agency through constant calibration. Do not grant it prematurely and build your iteration such that the user experience is not um impacted while you you know improve your AI system.