Building Autonomously Improving Systems with AI Agents with Ryan Sweet

Building Autonomously Improving Systems with AI Agents with Ryan Sweet 🔗 **Source:** [Building Autonomously Improving Systems with AI Agents with Ryan Sweet](https://www.youtube.com/watch?v=BcB15V9NMuw) > [!note]+ Document Notes > tags: [agentic-engineering, ai-agents, self-improving-systems] > author: Ryan Sweet > source: Microsoft Security AI Research Team > reference: https://www.youtube.com/watch?v=BcB15V9NMuw > date created: 2024-12-28 10:30 > > # 🏷️ Title: Building Autonomously Improving Systems with AI Agents >  ## 🧰 Action Inbox - [H1] Start building self-improving agents to stay ahead of adversaries — security motivation #security - [H2] Focus on workflow patterns over individual prompts — development efficiency #development - [H3] Design guardrails for autonomous improvement boundaries — safety controls #safety ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [Original Video](https://www.youtube.com/watch?v=BcB15V9NMuw) - [Microsoft Security Research](https://www.microsoft.com/en-us/research/group/security-research/) ## 📝 Summary - Microsoft security researcher Ryan Sweet presents a vision for building AI agents that can autonomously improve their own code and functionality - Motivation stems from the need to counter threat actors who will soon deploy self-improving attack agents - Key shift from traditional coding to "agentic engineering" where intelligence becomes both tool and design element - Evolution from individual prompts to reusable workflows to self-improving program architectures - Major challenges include testing/evaluation, UI patterns for human-AI collaboration, and defining improvement metrics ## 📖 Main Content Ryan Sweet from Microsoft's Security AI Research team argues that we're entering an era where AI agents can autonomously improve their own code. This shift is driven by security concerns - threat actors will soon deploy self-improving agents to maintain persistent network presence, creating an "arms race" that requires defensive innovation. The approach involves "agentic software engineering" where intelligence serves as both a development tool and a core design element of the systems being built. This represents a fundamental change from traditional programming, where asking the right questions becomes more important than writing code directly. The evolution follows a pattern: individual prompts become reusable commands, which become standardized workflows, which eventually become self-improving program architectures. Sweet emphasizes that this is similar to building reliable cloud systems from unreliable components - composing unreliable LLMs into more reliable complex systems. Key technical challenges include developing effective testing and evaluation methods for AI-generated code, creating UI patterns for human-AI collaboration, and establishing metrics for measuring autonomous improvement. The societal implications are significant, potentially disrupting traditional SaaS business models when large cloud providers can rapidly replicate software using self-improving agents. ## 🗒️ Notes #### ⏱️ [00:00](https://youtu.be/BcB15V9NMuw?t=0) — [H1] Security motivation for self-improving systems - Key idea: Threat actors will soon build agents that improve their own code for persistent network presence - Evidence: "My conclusion is that they very soon will be building agents that can improve their own code in order to maintain that position" #### ⏱️ [02:15](https://youtu.be/BcB15V9NMuw?t=135) — [H2] Childhood inspiration for learning computers - Key idea: Early expectation that computers would learn alongside humans drives current passion - Evidence: "My expectation as a five or six year old was that that thing would be on a learning journey with me" #### ⏱️ [03:30](https://youtu.be/BcB15V9NMuw?t=210) — [H1] Intelligence as tool and design element - Key idea: Intelligence serves dual role in both building and designing agents - Evidence: "Intelligence is both a tool for building our agents and our programs and it's also a design element of those agents and programs" #### ⏱️ [04:45](https://youtu.be/BcB15V9NMuw?t=285) — [H2] Questions over code writing - Key idea: Asking questions becomes more valuable than writing code directly - Evidence: "Writing the code itself is a lot less interesting than asking questions" #### ⏱️ [06:00](https://youtu.be/BcB15V9NMuw?t=360) — [H1] Lessons from complex project management - Key idea: Patterns from managing large human expert teams will guide AI system development - Evidence: "Those are the patterns that are going to help us build ever more sophisticated AI systems" #### ⏱️ [08:30](https://youtu.be/BcB15V9NMuw?t=510) — [H1] Building reliable systems from unreliable components - Key idea: Similar to cloud architecture, compose unreliable LLMs into reliable complex systems - Evidence: "How do you compose that LLM as an element into a more complex system and get more reliability out of the resulting system" #### ⏱️ [10:00](https://youtu.be/BcB15V9NMuw?t=600) — [H2] Evolution from prompts to workflows - Key idea: Successful prompts naturally evolve into reusable commands and workflows - Evidence: "Eventually over time the prompts that are good become commands or workflows that you can use repeatedly" #### ⏱️ [12:15](https://youtu.be/BcB15V9NMuw?t=735) — [H2] Development workflow pattern example - Key idea: Complete development cycles become automated patterns agents can execute - Evidence: "Starting a new code file with a specification and then defining the API and then writing the code file and then running your pre-commit hooks" #### ⏱️ [14:30](https://youtu.be/BcB15V9NMuw?t=870) — [H1] Self-improvement vectors and approaches - Key idea: Multiple paths exist for agents to improve including zero bugs, goal seeking, and design thinking - Evidence: "There's a lot of different ways to think about what self-improvement means in this context" #### ⏱️ [17:00](https://youtu.be/BcB15V9NMuw?t=1020) — [H1] Major unsolved problems in self-improving systems - Key idea: Testing, UI patterns, improvement measurement, and economic disruption remain challenging - Evidence: "There's a lot of big problems to solve and I think this is a really interesting space" #### ⏱️ [19:30](https://youtu.be/BcB15V9NMuw?t=1170) — [H1] Economic disruption of SaaS industry - Key idea: Self-improving programs will enable rapid replication of existing software - Evidence: "I would not want to be running a traditional SaaS company at that point" #### ⏱️ [21:45](https://youtu.be/BcB15V9NMuw?t=1305) — [H2] Ship of Theseus problem for evolving programs - Key idea: Self-modifying programs raise questions about identity and canonical versions - Evidence: "Once a thing has been rebuilt completely at what point is it no longer the same thing" #### ⏱️ [23:00](https://youtu.be/BcB15V9NMuw?t=1380) — [H2] Shipping intent rather than solutions - Key idea: Programs become representations of problem-solving approaches rather than fixed solutions - Evidence: "What you're shipping is intent, it's a representation of a set of solutions or tools for solving a problem" #### ⏱️ [25:30](https://youtu.be/BcB15V9NMuw?t=1530) — [H2] Embracing code disposal in agentic development - Key idea: AI-driven coding makes throwing away and restarting code beneficial rather than costly - Evidence: "I've never regretted throwing out the code I have and starting over" #### ⏱️ [27:00](https://youtu.be/BcB15V9NMuw?t=1620) — [H1] Current state as waypoint to self-improving systems - Key idea: Today's agent-assisted coding is just a stepping stone to truly autonomous improvement - Evidence: "Frame that as a waypoint, right? That's not the destination" ## 📚 References - [Original Video](https://www.youtube.com/watch?v=BcB15V9NMuw) - [Microsoft Security Research](https://www.microsoft.com/en-us/research/group/security-research/) - [Agentic AI Research](https://arxiv.org/search/?query=agentic+ai&searchtype=all) ## 🗣️ Paraphrase - Security researchers need to build self-improving AI agents because threat actors will soon use similar technology to maintain persistent network access, creating a defensive imperative - The field is shifting from traditional programming to "agentic engineering" where AI intelligence becomes both the tool for development and a core component of the systems being built - Successful prompts naturally evolve into reusable workflows and eventually into self-improving program architectures that can autonomously enhance their capabilities ## 📰 Newsletter Ryan Sweet from Microsoft Security Research shared fascinating insights about building AI agents that can improve their own code. This isn't just theoretical - it's driven by the reality that threat actors will soon deploy self-improving attack agents. The key shift is from writing code to asking the right questions and building workflows that agents can execute repeatedly. Think of it like teaching a junior developer not just what to do, but how to think about problems and improve their own processes. What's particularly exciting is how this mirrors WordPress's evolution - from static sites to dynamic content management to block-based editing. We're now looking at systems that can evolve their own functionality based on user needs and performance feedback. For WordPress developers, this suggests a future where themes and plugins could self-optimize for performance, accessibility, and user experience based on real usage data. Resources: [Microsoft AI Research](https://www.microsoft.com/en-us/research/group/security-research/), [Agentic AI Papers](https://arxiv.org/search/?query=agentic+ai), [WordPress AI Experiments](https://make.wordpress.org/core/tag/ai/) ## ✅ Action Items #### DevRel - Developer advocacy: Create demos of self-improving WordPress plugins, present at WordCamps about agentic development - Education: Develop workshops on AI-assisted WordPress development, office hours for plugin authors - Developer experience: Document patterns for AI-generated WordPress code, create SDK for agentic plugin development - Developer marketing: Blog series on "Future of WordPress Development with AI", social campaigns - Community management: Moderate discussions about AI ethics in WordPress development, gather feedback #### OSPO - Policy & Governance: Develop policies for AI-generated code contributions to open source WordPress projects - License Compliance: Review implications of AI-generated code on GPL compatibility and attribution - Security Management: Create guidelines for security review of AI-generated WordPress code - Software Inventory: Track AI tools used in WordPress core and plugin development - Developer Support: Training on responsible AI use in open source development - Community Engagement: Fund research into AI safety for content management systems #### Community - Maintainers: Roadmap for AI integration in WordPress core, governance for AI-generated contributions - Contributors: Onboarding materials for AI-assisted contribution workflows - Sponsors: Budget for AI safety research and responsible development initiatives - Ecosystem: Coordinate AI development standards across WordPress hosting providers #### WordPress - Core: Evaluate AI integration opportunities for automated testing and code quality - Core Performance: Use AI agents for performance profiling and optimization suggestions - Core AI: Establish guardrails for AI features, create evaluation sets for WordPress-specific AI - Playground: Create sandboxes for testing AI-generated WordPress modifications - Community: Develop meetup content about AI in WordPress development - Training: Create learning paths for AI-assisted WordPress development - Documentation: Guidelines for documenting AI-generated code and features ## 🧪 Hypotheses - Self-improving WordPress plugins will emerge within 18 months as developers apply agentic engineering principles - Traditional WordPress hosting companies will be disrupted by providers offering AI-optimized, self-healing WordPress installations - WordPress core development velocity will increase 3-5x through effective human-AI collaboration workflows - Plugin security will improve through AI agents that continuously audit and patch vulnerabilities ### 📊 Methodology - Study design to test hypothesis 1: Track plugin repository for self-modifying or AI-enhanced plugins - Data sources and analysis for hypothesis 2: Monitor hosting market share and feature announcements - Experiments to validate hypothesis 3: Measure core contribution rates and complexity before/after AI tool adoption ### ❓ Questions - How can WordPress ensure AI-generated contributions maintain the platform's commitment to open source values? - What governance structures are needed when WordPress sites can autonomously modify their own code? - How do we preserve human agency and creativity in a world of self-improving content management systems? ### 🔮 Future Research - Impact of self-improving systems on WordPress's democratic publishing mission - Technical frameworks for safe AI integration in content management platforms - Economic effects of AI-powered website creation on web development industry ### 💡 Implications - Practice: WordPress developers need to learn agentic engineering patterns alongside traditional PHP development - Policy: WordPress.org may need new review processes for AI-generated themes and plugins - Security: Self-modifying WordPress installations require new security models and monitoring approaches - Education: WordPress training programs must incorporate AI collaboration skills - Funding: Investment in AI safety research becomes critical for WordPress ecosystem health ### 🧠 Learning Styles Assessment Reading level estimate: 12th grade (technical content with specialized terminology) #### 👀 Visual learners - Present: Speaker slides with clear diagrams of agentic development progression - Improve: Architecture diagrams showing LLM composition patterns, workflow visualizations #### 🔊 Aural learners - Present: Full video presentation with clear narration - Improve: Podcast version with extended discussion, audio summary of key concepts #### 🗣️ Verbal learners - Present: Well-defined terms like "agentic engineering" with analogies to cloud architecture - Improve: Glossary of AI agent terminology, comparison tables between traditional vs agentic development #### 🧩 Kinesthetic learners - Present: Practical examples of prompt evolution to workflow patterns - Improve: Hands-on lab for building simple self-improving agent, step-by-step coding exercises #### 🧠 Logical learners - Present: Clear progression from security motivation through technical implementation to societal impact - Improve: Decision trees for when to apply self-improvement patterns, metrics for measuring agent effectiveness #### 🧑‍🤝‍🧑 Social learners - Present: References to collaborative patterns from complex human projects - Improve: Discussion prompts for team retrospectives on AI tool adoption, peer review guidelines #### 🧘‍♀️ Solitary learners - Present: Individual reflection prompts about personal development approach changes - Improve: Self-assessment worksheet for agentic engineering readiness, journaling questions #### 🌿 Naturalistic learners - Present: Analogies to evolutionary improvement and adaptation - Improve: Connections to natural system self-organization, ecosystem thinking about AI agents ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Well-structured presentation with clear heading hierarchy - Improve: Detailed alt text for any diagrams, ARIA labels for interactive elements #### Keyboard navigation - Present: Standard video controls accessible via keyboard - Improve: Transcript navigation with keyboard shortcuts, skip links for sections #### Contrast and readability - Present: Clear slide design with sufficient contrast - Improve: High contrast mode option, adjustable font sizes for transcript #### Text resizing and reflow - Present: Responsive transcript layout - Improve: Ensure 200% zoom compatibility, prevent horizontal scrolling #### Media and captions - Present: Full video with clear audio - Improve: Professional captions with speaker identification, downloadable transcript #### Forms and interactions - Present: Basic video interaction controls - Improve: Accessible feedback forms for questions, keyboard-navigable timestamps ### 🌍 DEIB Assessment #### Cultural breadth - Present: Focus on universal technical challenges affecting global developer community - Improve: Examples from diverse tech --- ## Full Transcript [00:00] So I uh came today to talk about uh [00:03] building systems that can improve uh [00:06] their own code and their own [00:08] functioning. Um I got into thinking [00:10] about this uh because I work in the [00:12] security AI research team along with [00:14] Michael and um you know uh I started [00:17] thinking about the work that our threat [00:18] adversaries are doing as they try to [00:20] build a persistent presence in [00:22] Microsoft's networks. And uh if they [00:24] aren't already uh my conclusion is that [00:26] they very soon will be uh building [00:28] agents that can improve their own code [00:30] in order to maintain that position. So [00:32] we in Microsoft spend a lot of time [00:34] thinking about how we can build agents [00:35] that are more autonomous uh that can uh [00:38] sort of evolve and adapt to the [00:41] situations they encounter. Uh and for me [00:43] that's a super depressing arm race. Uh [00:46] so I got to thinking is there a better [00:47] way? Can we do something different? [00:48] Right? Uh and and for me this idea of [00:52] programs that can learn and and uh [00:54] evolve along with you, it goes all the [00:56] way back like my first computer that my [00:57] parents ever got me which is like a [00:59] Radio Shack TR80. I remember my [01:02] expectation as a five or six year old [01:03] was that that thing would would be on a [01:06] learning journey with me. Um and turns [01:09] out that that the computer, you know, [01:11] learning a similar on its own wasn't [01:13] going to happen in the early 80s. But uh [01:16] I've kind of pursued that passion ever [01:19] since. And and I think about it this you [01:22] know the the possibility with LLMs and [01:26] agents uh is so tantalizing that you [01:28] know it's uh sort of we're on the tip of [01:31] being able to uh treat our uh work [01:37] output uh the agents that we build etc. [01:40] uh as if we're bringing them up, raising [01:43] them to go out into the world and do [01:44] work and uh preparing them to be able to [01:47] do that work well and to continue [01:48] learning as they go. For me, that's [01:50] super exciting and that's that's the [01:52] compiler that I want to build, right? Uh [01:56] and so this has become called agentic [01:58] software engineering or agentic [02:00] engineering. Uh and you know it's uh [02:04] very similar to how we have designed [02:07] software for the last four years. Uh but [02:10] there's a few things that are changing [02:11] about the flow and about how we do it. [02:14] Uh and you know part of that is that uh [02:17] intelligence is is both uh a tool for [02:22] building our our agents and our programs [02:24] and it's also a design element of those [02:27] agents and programs and and these are [02:28] two pieces that you know have not been [02:30] true for most of my career as as a [02:33] programmer. Um, and so this yields some [02:36] some interesting changes in in how you [02:38] approach what you build, right? Uh, [02:41] writing the code itself is a lot less [02:42] interesting than than asking questions. [02:45] And that leads you on a journey for [02:47] exploring uh all kinds of different uh [02:50] paths towards the solution you want to [02:52] get to, right? And your job as the [02:56] agentic engineer is to leverage the [02:59] agents you're building, the LLMs you're [03:01] working on on that journey of [03:03] exploration, right? And and to put them [03:06] together into new patterns, new ways of [03:08] doing things. That kind of recursive [03:11] process of involving the agents that [03:13] you're using in the design of themselves [03:17] uh is actually a really important part [03:18] of how you start to accelerate the work [03:20] that you do. [03:22] Um and so at the end of the day like [03:25] this becomes thinking about thinking [03:27] like and and so a lot of the science for [03:30] how we will end up doing this well at [03:33] scale I think is going to come from uh [03:36] what we've already learned as a society [03:38] about how to structure complex projects [03:41] right uh whether it's you know uh [03:43] designing the Apollo program or uh [03:46] building complex public policy uh [03:48] negotiating peace around the world etc [03:51] right? Uh where we get large groups of [03:54] human experts to think uh and [03:56] collaborate on complex work, those are [03:58] the patterns that are going to help us [04:00] build ever more sophisticated AI [04:02] systems, right? And so within the scope [04:06] of the agency you're trying to design, [04:08] the the the problems the questions that [04:10] a PM is going to ask in a software [04:12] development team, those are really [04:13] important to everybody on that team now, [04:15] right? What problems trying to solve? [04:17] What outcome do I want? How do I get to [04:20] a quality solution? Right? Uh and as [04:23] you're doing that, is there any part [04:25] about this process that you can turn [04:26] into tools that your agents can use? [04:29] Right? And we'll talk more about that. [04:31] Like part of this uh structuring how [04:34] you're thinking uh as you move through [04:36] agentic design uh is going to help you [04:39] figure out how to structure the agent in [04:41] such a way that it can do that kind of [04:44] thinking and ask those kinds of [04:45] questions as it builds new components [04:48] for itself. Right? At the end of the day [04:51] uh this really reminds me of a lot of [04:54] the challenges that were involved in [04:56] building cloud in the first place. [04:58] Right? uh cloudscale global systems are [05:01] possible because we dove deep into how [05:05] to build reliable systems out of [05:07] unreliable components whether it's [05:09] storage, network, compute uh you know [05:12] and in in this case it's the unreliable [05:14] component is the LLM which is subject to [05:16] hallucination which uh is trained on a [05:19] limited data set etc. Right. And how do [05:22] you compose that LLM as an element uh [05:27] into a more complex system and and get [05:30] more reliability out of the resulting [05:32] system than all the pieces that it's [05:33] built from. [05:36] One of the things that I've learned in [05:38] doing this repeatedly is that, you know, [05:41] you may start with a prompt and use that [05:44] prompt a few times, refine the prompt, [05:46] change it, uh build it into your [05:48] program. Uh but eventually over time the [05:51] prompts that are good become commands uh [05:54] or workflows that you can use repeatedly [05:57] right and uh they end up being things [06:00] that the LLM can call itself to then uh [06:04] dive off on a particular aspect of the [06:06] workflow. Um once you start doing that [06:09] then you'll have patterns for workflows [06:11] that you use again and again. Here a [06:14] sample pattern would be the uh workflow [06:18] of starting a new uh code file with a [06:21] specification and then defining the API [06:24] and then writing the code file and then [06:27] running your pre-commit hooks on that [06:30] code file and running all the tests and [06:32] then going and fixing everything you [06:33] found and then checking it in then [06:36] pushing to GitHub right and so that [06:38] starts to be you know one piece of the [06:40] development workflow of the whole thing [06:42] right and that becomes then uh a pattern [06:45] that can be used again and again by your [06:47] agents as you're building code if [06:49] building code is your objective, right? [06:51] Uh and so this kind of thing going in a [06:54] loop, right? Uh yields building more and [06:58] more tools that are built on one [06:59] another, right? And so as an example, [07:03] one of the hardest parts in evaluating [07:06] uh AI uh written code, right, is doing [07:11] that end toend test the way that a user [07:12] would. So what does it mean then to [07:15] build up an Gentic system that can do [07:17] that kind of outside in testing? And [07:19] that's sort of like the next more [07:21] complex workflow and the next piece [07:22] that's further out from kind of that [07:24] inner loop, right? Um, and where I lead [07:29] with this is that the the advent of [07:31] agentic coding is sort of a wayp point. [07:33] It's a it's a a stepping stone on the [07:36] way to building self-improving programs, [07:39] right? Um, we suddenly have all these [07:42] new capabilities that we didn't even [07:44] just a year ago uh in terms of of [07:47] producing vast amounts of relatively [07:50] sophisticated high quality code. uh and [07:54] there's a lot of problems with it. The [07:56] problems lead us to start to put a bunch [07:59] of pieces for solutions in place and and [08:02] in part that's sort of what are all the [08:04] guardrails we need as we generate code [08:07] with agents uh in order to try and [08:10] ensure quality. What are the structured [08:13] reflection loops where uh the agents can [08:16] be instructed to look at the work that [08:18] they just did uh with a different point [08:20] of view or multiple different points of [08:22] view uh and then try to reconcile that [08:25] with where you want to go and the [08:26] quality gates that you establish, right? [08:28] What are the playgrounds where you can [08:30] run those agentic end to-end uh tests [08:33] that sort of evaluate the work, give [08:36] feedback to the agents, let them go and [08:39] then structure how they approach making [08:41] a new solution, right? Um by turning to [08:46] recursion and feeding all of these [08:49] questions back into the coding systems [08:51] that you're building, you will come up [08:53] with the structures that are on the [08:54] right hand side of the slide, right? And [08:56] it will start to help you build [08:58] uh the pieces and the tools that are [09:00] necessary to to make a coding uh [09:04] accelerate itself. [09:07] Um there's a lot of different ways to [09:09] think about uh what self-improvement [09:11] means in this context or what the [09:13] sources of self-improvement are. It can [09:16] come from trying to ensure zero bugs, [09:19] right? Uh it can come from creating goal [09:22] seeking agents that you know their job [09:24] is to achieve a particular objective. [09:26] Like I built some agents the other day [09:28] where what I wanted was for them to to [09:30] get to a particular Azure setup uh with [09:34] you know uh five or six resources that [09:36] were related to each other in a certain [09:38] way and I didn't really care about how [09:40] they got there. I just gave them a goal [09:42] and then told them to iterate until they [09:43] got there. And you know that's one way [09:46] of kind of improving the system overall. [09:49] Um I'm particularly interested in design [09:51] thinking as a vector for [09:53] self-improvement and uh bringing you [09:56] know the kind of design thinking [09:57] techniques that we use in the classroom [09:58] or in product design into the agents [10:01] themselves. Uh and so my colleagues at [10:04] Microsoft have been doing some [10:05] interesting experiments around that. [10:07] Right. Um, the AI world has for a long [10:11] time experimented with genetic [10:13] algorithms as a way to mutate a program [10:16] in order to try and identify [10:18] improvements. I think that's a lot more [10:20] fraught uh kind of difficult in this uh [10:23] LLM world, but it happens from time to [10:25] time and it it's one tool in the toolbox [10:27] you could turn to. Similarly, there's [10:29] lots of other vectors here, right? [10:31] whether it's uh user customization or uh [10:34] you know finding things that were [10:35] working well in other programs and [10:37] bringing them into the program that [10:38] you're working on uh etc. There are are [10:41] techniques that can be used to pursue uh [10:44] an objective for self-improvement and [10:46] and I'm sure this is not a comprehensive [10:47] list. I think other folks may come up [10:49] with uh with ones that that are [10:52] interesting to them, right? Um to get to [10:56] a world of self-improving programs, [10:58] there's a lot of big problems to solve. [11:00] And so I think this is a really [11:02] interesting space where all of you that [11:04] are uh agentic engineers can be working [11:07] uh innovating playing and we don't have [11:09] all the answers and I think it's really [11:11] fun to be in that space. That's why I [11:14] have done computer science all my career [11:16] and and why I'm more excited about it [11:18] now than any day since I opened that [11:21] first computer I was telling you about [11:22] you know 40 plus years ago. um you know [11:27] uh I think the te the testing and [11:28] evaluation is one one huge problem we [11:31] can we can generate so much more code [11:33] now uh that we have to write agents to [11:36] sort of pretend to be users to help us [11:38] test that code right uh I think we still [11:40] don't know all the UI patterns uh or uh [11:43] ways of interacting with agents that are [11:45] necessary to to really help them be able [11:47] to learn from us or with us uh and so [11:50] there's some exciting things to answer [11:52] there right uh how do you know whether [11:54] something is actually improving green or [11:55] not is also pretty interesting uh [11:58] especially when it's something that's [11:59] relatively autonomous uh with somewhat [12:02] nebulous goals and even framing for the [12:05] agent what the right way to think about [12:06] that is uh can be really challenging [12:09] right um and then you know I think [12:14] there's there's an interesting spot too [12:16] is uh when the biggest companies in the [12:19] world achieve uh the ability to build [12:22] programs that can improve themselves [12:24] with user feed back with quality at [12:26] scale. Uh I think it really changes uh [12:29] the economics of everything that we're [12:31] doing, right? Uh and uh I would not want [12:35] to be running uh a traditional SAS [12:38] company at the point when that happens [12:40] because I think any large cloud player [12:43] at that point is suddenly able to [12:45] replicate your SAS very very quickly. Uh [12:48] and uh so that's that's a really [12:50] interesting kind of inflection point. As [12:52] soon as the the ability to really do [12:55] Zelvin brewing programs is achieved uh I [12:58] think it's going to be very disruptive [12:59] for the whole industry. uh you know I [13:03] think there's a lot of interesting [13:04] implications for how we design human in [13:06] the loop here and and where you know if [13:09] we bring together agents to help [13:11] facilitate [13:12] uh workflows where experts collaborate [13:15] you know uh what does it look like to [13:17] have humans in the loop to help guide [13:19] and uh have those agents work on their [13:22] behalf collaborate with colleagues on [13:24] their behalf uh and and there's [13:27] interesting challenges there my my [13:28] mental model for it is that this ought [13:30] to be a spiral [13:31] where initially uh most of the [13:35] collaboration it happens between AI [13:37] agents and as it gets further and [13:38] further from the initial idea to more [13:42] detail and more resolution around the [13:43] problem the human experts with the deep [13:46] expertise and the tools uh that are [13:49] outside of the AI agent system start to [13:51] interact with each other and with the [13:52] agents more the wider that spiral gets [13:55] from from the middle uh and the more [13:56] defined the problem becomes uh I think [13:59] at the end of the day there's [14:00] interesting societal impacts for [14:02] self-improving programs and uh you know [14:06] this is uh really exciting for me when I [14:08] think about you know the potential [14:10] impacts for product design for education [14:13] for you know uh building effective [14:16] government policy all kinds of different [14:18] things that we're u giving the systems [14:21] that we build the ability to improve [14:23] themselves and measure that improvement [14:26] uh will will unlock the next big stages [14:30] is in society. Um and then you know so [14:33] so as the builders of these systems then [14:36] how do we work on leveraging that [14:39] technology and adapting to all this [14:40] change? Fascinating questions here. Um [14:44] and there's one more that I want to pose [14:45] which is if you're a developer u [14:49] building a program that can go out into [14:51] the world and uh add to its [14:53] functionality right uh it brings up you [14:56] know the the sort of old ship of thesis [14:59] problem which is you know once a thing [15:00] has been rebuilt completely at what [15:02] point is it no longer the same thing [15:05] right uh and you know whether whether [15:08] it's a service that you've deployed into [15:10] the cloud that's now changing its [15:13] topology to adapt to attacks, right? Or [15:17] if it's a a user program that has [15:19] customized its interface to adapt to the [15:21] user's preferences so thoroughly, uh [15:24] that is no longer effectively what what [15:26] it was that you sent them, right? Uh [15:28] then the representation of what's you [15:30] know the canonical thing you're shipping [15:33] is really something different, right? [15:34] What you're shipping is intent. uh it's [15:38] uh a representation of a set of [15:40] solutions or tools for solving a problem [15:43] rather than a specific solution to a [15:45] given problem. Uh and this is a new way [15:48] for a lot of us to think about the [15:50] programs that we're building, right? Uh [15:54] and so you know I've been pushing the [15:56] boundaries of what we can do with [15:58] agentic coding uh sort of continuously [16:01] for the last three years. Uh and one of [16:04] the things that I want you to take away [16:06] from this talk is uh you know as [16:08] developers we've been taught to uh that [16:12] that the code we build is important [16:14] right uh and and once we've written code [16:17] it's really difficult to to throw that [16:19] away uh but I think AIdriven coding [16:23] especially agentic driven coding changes [16:25] all of that and across you know all the [16:27] time I've been working on this um I've [16:30] never regretted throwing out the code I [16:32] have and starting over uh which is a [16:34] newish feeling as a developer right uh [16:37] and and so you know that this is [16:40] something I want you to take away from [16:42] the session is that when you are [16:43] building agents uh whose job it is to uh [16:47] sort of get better and learn with you as [16:49] you go uh it's important to to be able [16:52] to throw that code down u and the other [16:56] the other thing that I threw in here is [16:58] uh you know at Microsoft especially uh [17:02] uh there's no shortage of people who [17:04] will tell you that what you're doing is [17:05] too ambitious or or that you know uh [17:07] doesn't fit with where we're trying to [17:09] get to right now. Um and for me [17:12] personally, I I often take that as a [17:14] sign that I'm I'm on the right track and [17:16] I'm going the right direction. Uh but as [17:18] with all things uh in this world of of [17:22] uh AIdriven progress uh it is important [17:25] to to exercise our skepticism and to uh [17:29] hold both ourselves and our work output [17:32] to account. Uh and and you'll develop [17:35] over time the more agentic coding that [17:37] you do a strong intuition for when the [17:39] model's lying to you or when you need to [17:41] ask more questions or when you need to [17:43] go deeper. Uh and it's just a reminder [17:45] about that. [17:47] Um yeah, and so lastly, I think, you [17:50] know, we're we're at a really uh [17:53] interesting transitional stage here [17:56] where uh you know, vibe coding is all [17:58] the rage and spectrum and development is [18:00] what everyone talks about and I I [18:03] encourage you to uh frame that as as a [18:06] waypoint, right? Uh that's not the [18:08] destination. I think that moving towards [18:13] um systems that once we bring them into [18:16] being can continue to improve themselves [18:19] uh is is the right way to think about [18:22] where all of this is going. Um and so as [18:26] product builders, as engineers [18:28] uh sort of then it's our job to start [18:31] thinking about how to embody the intent [18:35] uh for the systems that we're building [18:37] into the program itself. And uh those [18:40] those requirements become the guard [18:43] rails, the structure, the frame for what [18:45] the program is, right? Uh you can [18:47] imagine for instance uh you don't want a [18:51] self-improving program to modify the [18:53] building system, right? So you need to [18:55] put some guardrails in place around that [18:57] so that that part of the code is never [18:59] touched by the coding agents. Uh except [19:01] when explicit explicitly instructed to, [19:04] right? Um and and there's plenty of [19:07] things like that where uh putting [19:09] boundaries around these things uh is is [19:11] now a really important part uh of what [19:14] you're doing. Um the other thing I I [19:16] would would love you to think about is [19:18] is you know I raised a lot of questions [19:20] here and uh sort of called out a bunch [19:24] of um you know benefits and a bunch of [19:27] problems. My point is I think all the [19:30] pieces we need in order to build uh [19:33] really exciting self-improving systems [19:35] are right here. They're on the table. Uh [19:37] and it's our job to just start putting [19:39] them together piece by piece. Uh and I [19:41] think that's really interesting.  ## 📺 Video ![](https://www.youtube.com/watch?v=BcB15V9NMuw) --- # 📝 Full Notes (Deep-Read Analysis) # 🏷️ Title: Building Autonomously Improving Systems with AI Agents ## 🧰 Action Inbox - [H1] Start building self-improving agents to stay ahead of adversaries — security motivation #security - [H2] Focus on workflow patterns over individual prompts — development efficiency #development - [H3] Design guardrails for autonomous improvement boundaries — safety controls #safety ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [Original Video](https://www.youtube.com/watch?v=BcB15V9NMuw) - [Microsoft Security Research](https://www.microsoft.com/en-us/research/group/security-research/) ## 📝 Summary - Microsoft security researcher Ryan Sweet presents a vision for building AI agents that can autonomously improve their own code and functionality - Motivation stems from the need to counter threat actors who will soon deploy self-improving attack agents - Key shift from traditional coding to "agentic engineering" where intelligence becomes both tool and design element - Evolution from individual prompts to reusable workflows to self-improving program architectures - Major challenges include testing/evaluation, UI patterns for human-AI collaboration, and defining improvement metrics ## 📖 Main Content Ryan Sweet from Microsoft's Security AI Research team argues that we're entering an era where AI agents can autonomously improve their own code. This shift is driven by security concerns - threat actors will soon deploy self-improving agents to maintain persistent network presence, creating an "arms race" that requires defensive innovation. The approach involves "agentic software engineering" where intelligence serves as both a development tool and a core design element of the systems being built. This represents a fundamental change from traditional programming, where asking the right questions becomes more important than writing code directly. The evolution follows a pattern: individual prompts become reusable commands, which become standardized workflows, which eventually become self-improving program architectures. Sweet emphasizes that this is similar to building reliable cloud systems from unreliable components - composing unreliable LLMs into more reliable complex systems. Key technical challenges include developing effective testing and evaluation methods for AI-generated code, creating UI patterns for human-AI collaboration, and establishing metrics for measuring autonomous improvement. The societal implications are significant, potentially disrupting traditional SaaS business models when large cloud providers can rapidly replicate software using self-improving agents. ## 🗒️ Notes #### ⏱️ [00:00](https://youtu.be/BcB15V9NMuw?t=0) — [H1] Security motivation for self-improving systems - Key idea: Threat actors will soon build agents that improve their own code for persistent network presence - Evidence: "My conclusion is that they very soon will be building agents that can improve their own code in order to maintain that position" #### ⏱️ [02:15](https://youtu.be/BcB15V9NMuw?t=135) — [H2] Childhood inspiration for learning computers - Key idea: Early expectation that computers would learn alongside humans drives current passion - Evidence: "My expectation as a five or six year old was that that thing would be on a learning journey with me" #### ⏱️ [03:30](https://youtu.be/BcB15V9NMuw?t=210) — [H1] Intelligence as tool and design element - Key idea: Intelligence serves dual role in both building and designing agents - Evidence: "Intelligence is both a tool for building our agents and our programs and it's also a design element of those agents and programs" #### ⏱️ [04:45](https://youtu.be/BcB15V9NMuw?t=285) — [H2] Questions over code writing - Key idea: Asking questions becomes more valuable than writing code directly - Evidence: "Writing the code itself is a lot less interesting than asking questions" #### ⏱️ [06:00](https://youtu.be/BcB15V9NMuw?t=360) — [H1] Lessons from complex project management - Key idea: Patterns from managing large human expert teams will guide AI system development - Evidence: "Those are the patterns that are going to help us build ever more sophisticated AI systems" #### ⏱️ [08:30](https://youtu.be/BcB15V9NMuw?t=510) — [H1] Building reliable systems from unreliable components - Key idea: Similar to cloud architecture, compose unreliable LLMs into reliable complex systems - Evidence: "How do you compose that LLM as an element into a more complex system and get more reliability out of the resulting system" #### ⏱️ [10:00](https://youtu.be/BcB15V9NMuw?t=600) — [H2] Evolution from prompts to workflows - Key idea: Successful prompts naturally evolve into reusable commands and workflows - Evidence: "Eventually over time the prompts that are good become commands or workflows that you can use repeatedly" #### ⏱️ [12:15](https://youtu.be/BcB15V9NMuw?t=735) — [H2] Development workflow pattern example - Key idea: Complete development cycles become automated patterns agents can execute - Evidence: "Starting a new code file with a specification and then defining the API and then writing the code file and then running your pre-commit hooks" #### ⏱️ [14:30](https://youtu.be/BcB15V9NMuw?t=870) — [H1] Self-improvement vectors and approaches - Key idea: Multiple paths exist for agents to improve including zero bugs, goal seeking, and design thinking - Evidence: "There's a lot of different ways to think about what self-improvement means in this context" #### ⏱️ [17:00](https://youtu.be/BcB15V9NMuw?t=1020) — [H1] Major unsolved problems in self-improving systems - Key idea: Testing, UI patterns, improvement measurement, and economic disruption remain challenging - Evidence: "There's a lot of big problems to solve and I think this is a really interesting space" #### ⏱️ [19:30](https://youtu.be/BcB15V9NMuw?t=1170) — [H1] Economic disruption of SaaS industry - Key idea: Self-improving programs will enable rapid replication of existing software - Evidence: "I would not want to be running a traditional SaaS company at that point" #### ⏱️ [21:45](https://youtu.be/BcB15V9NMuw?t=1305) — [H2] Ship of Theseus problem for evolving programs - Key idea: Self-modifying programs raise questions about identity and canonical versions - Evidence: "Once a thing has been rebuilt completely at what point is it no longer the same thing" #### ⏱️ [23:00](https://youtu.be/BcB15V9NMuw?t=1380) — [H2] Shipping intent rather than solutions - Key idea: Programs become representations of problem-solving approaches rather than fixed solutions - Evidence: "What you're shipping is intent, it's a representation of a set of solutions or tools for solving a problem" #### ⏱️ [25:30](https://youtu.be/BcB15V9NMuw?t=1530) — [H2] Embracing code disposal in agentic development - Key idea: AI-driven coding makes throwing away and restarting code beneficial rather than costly - Evidence: "I've never regretted throwing out the code I have and starting over" #### ⏱️ [27:00](https://youtu.be/BcB15V9NMuw?t=1620) — [H1] Current state as waypoint to self-improving systems - Key idea: Today's agent-assisted coding is just a stepping stone to truly autonomous improvement - Evidence: "Frame that as a waypoint, right? That's not the destination" ## 📚 References - [Original Video](https://www.youtube.com/watch?v=BcB15V9NMuw) - [Microsoft Security Research](https://www.microsoft.com/en-us/research/group/security-research/) - [Agentic AI Research](https://arxiv.org/search/?query=agentic+ai&searchtype=all) ## 🗣️ Paraphrase - Security researchers need to build self-improving AI agents because threat actors will soon use similar technology to maintain persistent network access, creating a defensive imperative - The field is shifting from traditional programming to "agentic engineering" where AI intelligence becomes both the tool for development and a core component of the systems being built - Successful prompts naturally evolve into reusable workflows and eventually into self-improving program architectures that can autonomously enhance their capabilities ## 📰 Newsletter Ryan Sweet from Microsoft Security Research shared fascinating insights about building AI agents that can improve their own code. This isn't just theoretical - it's driven by the reality that threat actors will soon deploy self-improving attack agents. The key shift is from writing code to asking the right questions and building workflows that agents can execute repeatedly. Think of it like teaching a junior developer not just what to do, but how to think about problems and improve their own processes. What's particularly exciting is how this mirrors WordPress's evolution - from static sites to dynamic content management to block-based editing. We're now looking at systems that can evolve their own functionality based on user needs and performance feedback. For WordPress developers, this suggests a future where themes and plugins could self-optimize for performance, accessibility, and user experience based on real usage data. Resources: [Microsoft AI Research](https://www.microsoft.com/en-us/research/group/security-research/), [Agentic AI Papers](https://arxiv.org/search/?query=agentic+ai), [WordPress AI Experiments](https://make.wordpress.org/core/tag/ai/) ## ✅ Action Items #### DevRel - Developer advocacy: Create demos of self-improving WordPress plugins, present at WordCamps about agentic development - Education: Develop workshops on AI-assisted WordPress development, office hours for plugin authors - Developer experience: Document patterns for AI-generated WordPress code, create SDK for agentic plugin development - Developer marketing: Blog series on "Future of WordPress Development with AI", social campaigns - Community management: Moderate discussions about AI ethics in WordPress development, gather feedback #### OSPO - Policy & Governance: Develop policies for AI-generated code contributions to open source WordPress projects - License Compliance: Review implications of AI-generated code on GPL compatibility and attribution - Security Management: Create guidelines for security review of AI-generated WordPress code - Software Inventory: Track AI tools used in WordPress core and plugin development - Developer Support: Training on responsible AI use in open source development - Community Engagement: Fund research into AI safety for content management systems #### Community - Maintainers: Roadmap for AI integration in WordPress core, governance for AI-generated contributions - Contributors: Onboarding materials for AI-assisted contribution workflows - Sponsors: Budget for AI safety research and responsible development initiatives - Ecosystem: Coordinate AI development standards across WordPress hosting providers #### WordPress - Core: Evaluate AI integration opportunities for automated testing and code quality - Core Performance: Use AI agents for performance profiling and optimization suggestions - Core AI: Establish guardrails for AI features, create evaluation sets for WordPress-specific AI - Playground: Create sandboxes for testing AI-generated WordPress modifications - Community: Develop meetup content about AI in WordPress development - Training: Create learning paths for AI-assisted WordPress development - Documentation: Guidelines for documenting AI-generated code and features ## 🧪 Hypotheses - Self-improving WordPress plugins will emerge within 18 months as developers apply agentic engineering principles - Traditional WordPress hosting companies will be disrupted by providers offering AI-optimized, self-healing WordPress installations - WordPress core development velocity will increase 3-5x through effective human-AI collaboration workflows - Plugin security will improve through AI agents that continuously audit and patch vulnerabilities ### 📊 Methodology - Study design to test hypothesis 1: Track plugin repository for self-modifying or AI-enhanced plugins - Data sources and analysis for hypothesis 2: Monitor hosting market share and feature announcements - Experiments to validate hypothesis 3: Measure core contribution rates and complexity before/after AI tool adoption ### ❓ Questions - How can WordPress ensure AI-generated contributions maintain the platform's commitment to open source values? - What governance structures are needed when WordPress sites can autonomously modify their own code? - How do we preserve human agency and creativity in a world of self-improving content management systems? ### 🔮 Future Research - Impact of self-improving systems on WordPress's democratic publishing mission - Technical frameworks for safe AI integration in content management platforms - Economic effects of AI-powered website creation on web development industry ### 💡 Implications - Practice: WordPress developers need to learn agentic engineering patterns alongside traditional PHP development - Policy: WordPress.org may need new review processes for AI-generated themes and plugins - Security: Self-modifying WordPress installations require new security models and monitoring approaches - Education: WordPress training programs must incorporate AI collaboration skills - Funding: Investment in AI safety research becomes critical for WordPress ecosystem health ### 🧠 Learning Styles Assessment Reading level estimate: 12th grade (technical content with specialized terminology) #### 👀 Visual learners - Present: Speaker slides with clear diagrams of agentic development progression - Improve: Architecture diagrams showing LLM composition patterns, workflow visualizations #### 🔊 Aural learners - Present: Full video presentation with clear narration - Improve: Podcast version with extended discussion, audio summary of key concepts #### 🗣️ Verbal learners - Present: Well-defined terms like "agentic engineering" with analogies to cloud architecture - Improve: Glossary of AI agent terminology, comparison tables between traditional vs agentic development #### 🧩 Kinesthetic learners - Present: Practical examples of prompt evolution to workflow patterns - Improve: Hands-on lab for building simple self-improving agent, step-by-step coding exercises #### 🧠 Logical learners - Present: Clear progression from security motivation through technical implementation to societal impact - Improve: Decision trees for when to apply self-improvement patterns, metrics for measuring agent effectiveness #### 🧑‍🤝‍🧑 Social learners - Present: References to collaborative patterns from complex human projects - Improve: Discussion prompts for team retrospectives on AI tool adoption, peer review guidelines #### 🧘‍♀️ Solitary learners - Present: Individual reflection prompts about personal development approach changes - Improve: Self-assessment worksheet for agentic engineering readiness, journaling questions #### 🌿 Naturalistic learners - Present: Analogies to evolutionary improvement and adaptation - Improve: Connections to natural system self-organization, ecosystem thinking about AI agents ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Well-structured presentation with clear heading hierarchy - Improve: Detailed alt text for any diagrams, ARIA labels for interactive elements #### Keyboard navigation - Present: Standard video controls accessible via keyboard - Improve: Transcript navigation with keyboard shortcuts, skip links for sections #### Contrast and readability - Present: Clear slide design with sufficient contrast - Improve: High contrast mode option, adjustable font sizes for transcript #### Text resizing and reflow - Present: Responsive transcript layout - Improve: Ensure 200% zoom compatibility, prevent horizontal scrolling #### Media and captions - Present: Full video with clear audio - Improve: Professional captions with speaker identification, downloadable transcript #### Forms and interactions - Present: Basic video interaction controls - Improve: Accessible feedback forms for questions, keyboard-navigable timestamps ### 🌍 DEIB Assessment #### Cultural breadth - Present: Focus on universal technical challenges affecting global developer community - Improve: Examples from diverse tech --- # 📄 Full Transcript *Auto-generated from YouTube captions · timestamps roughly every 30 seconds · [watch on YouTube](https://www.youtube.com/watch?v=BcB15V9NMuw)* **[00:00]** So I uh came today to talk about uh building systems that can improve uh their own code and their own functioning. Um I got into thinking about this uh because I work in the security AI research team along with Michael and um you know uh I started thinking about the work that our threat adversaries are doing as they try to build a persistent presence in Microsoft's networks. And uh if they aren't already uh my conclusion is that they very soon will be uh building agents that can improve their own code **[00:30]** in order to maintain that position. So we in Microsoft spend a lot of time thinking about how we can build agents that are more autonomous uh that can uh sort of evolve and adapt to the situations they encounter. Uh and for me that's a super depressing arm race. Uh so I got to thinking is there a better way? Can we do something different? Right? Uh and and for me this idea of programs that can learn and and uh evolve along with you, it goes all the way back like my first computer that my parents ever got me which is like a Radio Shack TR80. I remember my **[01:02]** expectation as a five or six year old was that that thing would would be on a learning journey with me. Um and turns out that that the computer, you know, learning a similar on its own wasn't going to happen in the early 80s. But uh I've kind of pursued that passion ever since. And and I think about it this you know the the possibility with LLMs and agents uh is so tantalizing that you know it's uh sort of we're on the tip of being able to uh treat our uh work **[01:37]** output uh the agents that we build etc. uh as if we're bringing them up, raising them to go out into the world and do work and uh preparing them to be able to do that work well and to continue learning as they go. For me, that's super exciting and that's that's the compiler that I want to build, right? Uh and so this has become called agentic software engineering or agentic engineering. Uh and you know it's uh very similar to how we have designed **[02:07]** software for the last four years. Uh but there's a few things that are changing about the flow and about how we do it. Uh and you know part of that is that uh intelligence is is both uh a tool for building our our agents and our programs and it's also a design element of those agents and programs and and these are two pieces that you know have not been true for most of my career as as a programmer. Um, and so this yields some some interesting changes in in how you **[02:38]** approach what you build, right? Uh, writing the code itself is a lot less interesting than than asking questions. And that leads you on a journey for exploring uh all kinds of different uh paths towards the solution you want to get to, right? And your job as the agentic engineer is to leverage the agents you're building, the LLMs you're working on on that journey of exploration, right? And and to put them together into new patterns, new ways of **[03:08]** doing things. That kind of recursive process of involving the agents that you're using in the design of themselves uh is actually a really important part of how you start to accelerate the work that you do. Um and so at the end of the day like this becomes thinking about thinking like and and so a lot of the science for how we will end up doing this well at scale I think is going to come from uh what we've already learned as a society about how to structure complex projects **[03:41]** right uh whether it's you know uh designing the Apollo program or uh building complex public policy uh negotiating peace around the world etc right? Uh where we get large groups of human experts to think uh and collaborate on complex work, those are the patterns that are going to help us build ever more sophisticated AI systems, right? And so within the scope of the agency you're trying to design, the the the problems the questions that a PM is going to ask in a software **[04:12]** development team, those are really important to everybody on that team now, right? What problems trying to solve? What outcome do I want? How do I get to a quality solution? Right? Uh and as you're doing that, is there any part about this process that you can turn into tools that your agents can use? Right? And we'll talk more about that. Like part of this uh structuring how you're thinking uh as you move through agentic design uh is going to help you figure out how to structure the agent in such a way that it can do that kind of **[04:44]** thinking and ask those kinds of questions as it builds new components for itself. Right? At the end of the day uh this really reminds me of a lot of the challenges that were involved in building cloud in the first place. Right? uh cloudscale global systems are possible because we dove deep into how to build reliable systems out of unreliable components whether it's storage, network, compute uh you know and in in this case it's the unreliable **[05:14]** component is the LLM which is subject to hallucination which uh is trained on a limited data set etc. Right. And how do you compose that LLM as an element uh into a more complex system and and get more reliability out of the resulting system than all the pieces that it's built from. One of the things that I've learned in doing this repeatedly is that, you know, you may start with a prompt and use that prompt a few times, refine the prompt, **[05:46]** change it, uh build it into your program. Uh but eventually over time the prompts that are good become commands uh or workflows that you can use repeatedly right and uh they end up being things that the LLM can call itself to then uh dive off on a particular aspect of the workflow. Um once you start doing that then you'll have patterns for workflows that you use again and again. Here a sample pattern would be the uh workflow **[06:18]** of starting a new uh code file with a specification and then defining the API and then writing the code file and then running your pre-commit hooks on that code file and running all the tests and then going and fixing everything you found and then checking it in then pushing to GitHub right and so that starts to be you know one piece of the development workflow of the whole thing right and that becomes then uh a pattern that can be used again and again by your agents as you're building code if **[06:49]** building code is your objective, right? Uh and so this kind of thing going in a loop, right? Uh yields building more and more tools that are built on one another, right? And so as an example, one of the hardest parts in evaluating uh AI uh written code, right, is doing that end toend test the way that a user would. So what does it mean then to build up an Gentic system that can do that kind of outside in testing? And **[07:19]** that's sort of like the next more complex workflow and the next piece that's further out from kind of that inner loop, right? Um, and where I lead with this is that the the advent of agentic coding is sort of a wayp point. It's a it's a a stepping stone on the way to building self-improving programs, right? Um, we suddenly have all these new capabilities that we didn't even just a year ago uh in terms of of producing vast amounts of relatively **[07:50]** sophisticated high quality code. uh and there's a lot of problems with it. The problems lead us to start to put a bunch of pieces for solutions in place and and in part that's sort of what are all the guardrails we need as we generate code with agents uh in order to try and ensure quality. What are the structured reflection loops where uh the agents can be instructed to look at the work that they just did uh with a different point **[08:20]** of view or multiple different points of view uh and then try to reconcile that with where you want to go and the quality gates that you establish, right? What are the playgrounds where you can run those agentic end to-end uh tests that sort of evaluate the work, give feedback to the agents, let them go and then structure how they approach making a new solution, right? Um by turning to recursion and feeding all of these questions back into the coding systems **[08:51]** that you're building, you will come up with the structures that are on the right hand side of the slide, right? And it will start to help you build uh the pieces and the tools that are necessary to to make a coding uh accelerate itself. Um there's a lot of different ways to think about uh what self-improvement means in this context or what the sources of self-improvement are. It can come from trying to ensure zero bugs, right? Uh it can come from creating goal **[09:22]** seeking agents that you know their job is to achieve a particular objective. Like I built some agents the other day where what I wanted was for them to to get to a particular Azure setup uh with you know uh five or six resources that were related to each other in a certain way and I didn't really care about how they got there. I just gave them a goal and then told them to iterate until they got there. And you know that's one way of kind of improving the system overall. Um I'm particularly interested in design thinking as a vector for **[09:53]** self-improvement and uh bringing you know the kind of design thinking techniques that we use in the classroom or in product design into the agents themselves. Uh and so my colleagues at Microsoft have been doing some interesting experiments around that. Right. Um, the AI world has for a long time experimented with genetic algorithms as a way to mutate a program in order to try and identify improvements. I think that's a lot more fraught uh kind of difficult in this uh LLM world, but it happens from time to **[10:25]** time and it it's one tool in the toolbox you could turn to. Similarly, there's lots of other vectors here, right? whether it's uh user customization or uh you know finding things that were working well in other programs and bringing them into the program that you're working on uh etc. There are are techniques that can be used to pursue uh an objective for self-improvement and and I'm sure this is not a comprehensive list. I think other folks may come up with uh with ones that that are interesting to them, right? Um to get to **[10:56]** a world of self-improving programs, there's a lot of big problems to solve. And so I think this is a really interesting space where all of you that are uh agentic engineers can be working uh innovating playing and we don't have all the answers and I think it's really fun to be in that space. That's why I have done computer science all my career and and why I'm more excited about it now than any day since I opened that first computer I was telling you about you know 40 plus years ago. um you know **[11:27]** uh I think the te the testing and evaluation is one one huge problem we can we can generate so much more code now uh that we have to write agents to sort of pretend to be users to help us test that code right uh I think we still don't know all the UI patterns uh or uh ways of interacting with agents that are necessary to to really help them be able to learn from us or with us uh and so there's some exciting things to answer there right uh how do you know whether something is actually improving green or not is also pretty interesting uh **[11:58]** especially when it's something that's relatively autonomous uh with somewhat nebulous goals and even framing for the agent what the right way to think about that is uh can be really challenging right um and then you know I think there's there's an interesting spot too is uh when the biggest companies in the world achieve uh the ability to build programs that can improve themselves with user feed back with quality at scale. Uh I think it really changes uh **[12:29]** the economics of everything that we're doing, right? Uh and uh I would not want to be running uh a traditional SAS company at the point when that happens because I think any large cloud player at that point is suddenly able to replicate your SAS very very quickly. Uh and uh so that's that's a really interesting kind of inflection point. As soon as the the ability to really do Zelvin brewing programs is achieved uh I think it's going to be very disruptive **[12:59]** for the whole industry. uh you know I think there's a lot of interesting implications for how we design human in the loop here and and where you know if we bring together agents to help facilitate uh workflows where experts collaborate you know uh what does it look like to have humans in the loop to help guide and uh have those agents work on their behalf collaborate with colleagues on their behalf uh and and there's interesting challenges there my my mental model for it is that this ought **[13:30]** to be a spiral where initially uh most of the collaboration it happens between AI agents and as it gets further and further from the initial idea to more detail and more resolution around the problem the human experts with the deep expertise and the tools uh that are outside of the AI agent system start to interact with each other and with the agents more the wider that spiral gets from from the middle uh and the more defined the problem becomes uh I think at the end of the day there's **[14:00]** interesting societal impacts for self-improving programs and uh you know this is uh really exciting for me when I think about you know the potential impacts for product design for education for you know uh building effective government policy all kinds of different things that we're u giving the systems that we build the ability to improve themselves and measure that improvement uh will will unlock the next big stages is in society. Um and then you know so **[14:33]** so as the builders of these systems then how do we work on leveraging that technology and adapting to all this change? Fascinating questions here. Um and there's one more that I want to pose which is if you're a developer u building a program that can go out into the world and uh add to its functionality right uh it brings up you know the the sort of old ship of thesis problem which is you know once a thing has been rebuilt completely at what point is it no longer the same thing **[15:05]** right uh and you know whether whether it's a service that you've deployed into the cloud that's now changing its topology to adapt to attacks, right? Or if it's a a user program that has customized its interface to adapt to the user's preferences so thoroughly, uh that is no longer effectively what what it was that you sent them, right? Uh then the representation of what's you know the canonical thing you're shipping is really something different, right? What you're shipping is intent. uh it's **[15:38]** uh a representation of a set of solutions or tools for solving a problem rather than a specific solution to a given problem. Uh and this is a new way for a lot of us to think about the programs that we're building, right? Uh and so you know I've been pushing the boundaries of what we can do with agentic coding uh sort of continuously for the last three years. Uh and one of the things that I want you to take away from this talk is uh you know as **[16:08]** developers we've been taught to uh that that the code we build is important right uh and and once we've written code it's really difficult to to throw that away uh but I think AIdriven coding especially agentic driven coding changes all of that and across you know all the time I've been working on this um I've never regretted throwing out the code I have and starting over uh which is a newish feeling as a developer right uh and and so you know that this is **[16:40]** something I want you to take away from the session is that when you are building agents uh whose job it is to uh sort of get better and learn with you as you go uh it's important to to be able to throw that code down u and the other the other thing that I threw in here is uh you know at Microsoft especially uh uh there's no shortage of people who will tell you that what you're doing is too ambitious or or that you know uh doesn't fit with where we're trying to get to right now. Um and for me **[17:12]** personally, I I often take that as a sign that I'm I'm on the right track and I'm going the right direction. Uh but as with all things uh in this world of of uh AIdriven progress uh it is important to to exercise our skepticism and to uh hold both ourselves and our work output to account. Uh and and you'll develop over time the more agentic coding that you do a strong intuition for when the model's lying to you or when you need to ask more questions or when you need to **[17:43]** go deeper. Uh and it's just a reminder about that. Um yeah, and so lastly, I think, you know, we're we're at a really uh interesting transitional stage here where uh you know, vibe coding is all the rage and spectrum and development is what everyone talks about and I I encourage you to uh frame that as as a waypoint, right? Uh that's not the destination. I think that moving towards um systems that once we bring them into **[18:16]** being can continue to improve themselves uh is is the right way to think about where all of this is going. Um and so as product builders, as engineers uh sort of then it's our job to start thinking about how to embody the intent uh for the systems that we're building into the program itself. And uh those those requirements become the guard rails, the structure, the frame for what the program is, right? Uh you can **[18:47]** imagine for instance uh you don't want a self-improving program to modify the building system, right? So you need to put some guardrails in place around that so that that part of the code is never touched by the coding agents. Uh except when explicit explicitly instructed to, right? Um and and there's plenty of things like that where uh putting boundaries around these things uh is is now a really important part uh of what you're doing. Um the other thing I I would would love you to think about is **[19:18]** is you know I raised a lot of questions here and uh sort of called out a bunch of um you know benefits and a bunch of problems. My point is I think all the pieces we need in order to build uh really exciting self-improving systems are right here. They're on the table. Uh and it's our job to just start putting them together piece by piece. Uh and I think that's really interesting.