Safe Production Changes With Agents - CourtneyR.dev's 2nd Brain

Safe Production Changes With Agents ![rw-book-cover](https://img.youtube.com/vi/6bHxMYeatHs/maxresdefault.jpg) 🔗 **Source:** [Safe Production Changes With Agents](https://www.youtube.com/watch?v=6bHxMYeatHs) > [!note]+ Document Notes > tags: [ai-agents, production-safety, code-review, agentic-engineering, continuous-delivery, software-engineering] > author: [MISSING] > source: Lead Dev Online Panel > reference: https://www.youtube.com/watch?v=6bHxMYeatHs > date created: {{date:YYYY-MM-DD}} {{time:HH:mm}} > > # 🏷️ Title: Safe production changes with agents >  ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ## 📝 Summary - AI agents amplify existing engineering practices, making good teams faster and exposing weaknesses in struggling teams - Safe production changes with AI require the same foundational practices as before: small steps, quick feedback, easy rollbacks, and good observability - Code review must evolve from line-by-line examination to architectural and solution validation as AI generates exponentially more code - Junior engineers face unprecedented challenges as traditional learning paths through code writing become obsoleted - The fundamental shift is from humans writing code to humans specifying behaviors and verifying outcomes ## 📖 Main Content This Lead Dev panel discussion explores how to safely implement production changes when AI agents are generating code at unprecedented speeds. The core thesis from all panelists is that **AI amplifies everything** - good practices become superpowers while poor practices become disasters. ### Core Safety Principles The panelists identified essential practices that become even more critical with AI agents: - **Working in small steps** with incremental deployments - **Easy rollbacks** and scoped permissions - **Strong observability** and feedback loops - **Specifying what we want and verifying we get it** - **Confidence in deployment processes** Dave Farley emphasized that if your deployment process relied on "push button and hope," AI will accelerate that chaos. Jean Dipli from Kilo (an agentic engineering platform) noted that success with AI agents depends entirely on how safe human processes were beforehand. ### The Code Review Crisis A critical bottleneck emerges: how do you review 12,000 lines of AI-generated code per day? The traditional line-by-line code review model breaks down completely. Jean's team has shifted to: - **PR authors taking ownership** of correctness through local AI code reviews - **Reviewers focusing on higher-level concerns**: Is this the right solution? Does it fit our architecture? - **Enhanced PR templates** requiring problem statements, verification methods, and walkthrough videos ### Junior Engineer Dilemma The panel acknowledged this may be "the hardest time in memory" for junior engineers. Traditional learning through writing code and making mistakes is disappearing. Dave predicts we're at a tipping point where humans simply don't write production code anymore. Recommendations for juniors: - **Learn BDD (Behavior-Driven Development)** as a specification language for AI - **Focus on problem decomposition** and engineering thinking over syntax - **Use AI as a patient teacher** but insist on explanations rather than solutions - **Develop communication skills** for prompting and collaboration ### The Abstraction Revolution Dave Farley framed this as a fundamental shift to "fifth generation programming" - a new level of abstraction where we specify system behaviors rather than implementation details. This requires understanding: - **Business behaviors** (what happens when inventory runs out?) - **Technical behaviors** (what about concurrency and edge cases?) - **Security behaviors** (how do we prevent data leaks?) - **Architectural constraints** (how do we maintain modularity?) ## 🗒️ Notes #### ⏱️ [00:03:45](https://www.youtube.com/watch?v=6bHxMYeatHs&t=225) AI as amplifier concept - DORA report confirms AI amplifies good and bad practices - High-performing teams get speed safely at scale - Others just get more code shoved into existing bottlenecks #### ⏱️ [00:08:12](https://www.youtube.com/watch?v=6bHxMYeatHs&t=492) Core stability principles - Working in small steps - Easy rollbacks - Good observability - Specifying and verifying outcomes - Confidence in deployment processes #### ⏱️ [00:15:30](https://www.youtube.com/watch?v=6bHxMYeatHs&t=930) The 12,000 lines problem - Engineer claims to produce 12,000 lines/day with AI - Nobody can meaningfully review that volume - Lines of code remains a terrible metric #### ⏱️ [00:18:45](https://www.youtube.com/watch?v=6bHxMYeatHs&t=1125) Kilo's 99% AI-generated code - Jean's startup generates 99% of code with AI - Still learning optimal practices - Reality check: humans can't manage 15-20 agents simultaneously #### ⏱️ [00:22:00](https://www.youtube.com/watch?v=6bHxMYeatHs&t=1320) Junior engineer challenges - Hardest time in memory for new engineers - Need patience and knowledge sharing, but business pressure demands speed - Must adapt to new reality where organic knowledge transfer is reduced #### ⏱️ [00:35:15](https://www.youtube.com/watch?v=6bHxMYeatHs&t=2115) Code review evolution - Traditional line-by-line review doesn't scale - Psychological barrier: reviewing 40-minute PR that took 5 minutes to generate - Shift to architectural and solution-level review #### ⏱️ [00:42:30](https://www.youtube.com/watch?v=6bHxMYeatHs&t=2550) Pairing with AI agents - Feels like "throbbling" (three-way pairing) - AI as sophisticated rubber duck - Need for less agreeable, more opinionated AI #### ⏱️ [00:45:20](https://www.youtube.com/watch?v=6bHxMYeatHs&t=2720) Advice for engineering managers - Get hands-on with AI to build empathy - Experience the frustrations firsthand - Hire for communication skills over coding skills ## 📚 References - [DORA State of DevOps Report](https://dora.dev/) - Research on AI amplification effects - [Kilo](https://kilo.dev) - Agentic engineering platform - [Octopus Deploy](https://octopus.com) - Deployment automation platform - [Lead Dev Community](https://theleaddeveloper.com/) - Engineering leadership community ## 🗣️ Paraphrase The panel consensus reveals that AI agents don't eliminate the need for good engineering practices—they make them absolutely essential. When AI can generate thousands of lines of code daily, the traditional safety nets of human review and intervention collapse. Teams must shift from code-centric to behavior-centric development, where engineers become system architects specifying desired outcomes rather than implementation details. The most concerning insight is the junior engineer crisis. As AI handles code generation, traditional learning paths disappear, but the industry still needs people who understand system design, problem decomposition, and architectural thinking. The solution isn't to slow down AI adoption but to fundamentally restructure how we teach and mentor engineers. ## 📰 Newsletter 🤖 **AI Agents in Production: The Good, Bad, and Code-Review-Breaking** The Lead Dev panel this week tackled every engineering manager's burning question: how do you safely deploy when AI is writing most of your code? The short answer? All those best practices you've been meaning to implement? Now's the time. The panelists—including Dave Farley and folks from Octopus Deploy and Kilo—dropped some truth bombs: if you couldn't safely deploy human code, AI code will be a disaster. But if you've got solid CI/CD, observability, and can work in small steps, AI becomes a superpower. **The plot twist**: Traditional code review is dead. Nobody can review 12,000 AI-generated lines per day (yes, that's a real claim from the wild). Instead, reviews are shifting to architectural validation—is this the right solution to the right problem? **For junior devs**: It's rough out there. The advice? Learn BDD, focus on problem decomposition over syntax, and remember that prompt engineering is the new coding. The future belongs to those who can specify what they want, not type it out. Worth watching if you're wrestling with AI tooling strategy or wondering why your deployment process suddenly feels inadequate. Plus, Dave's point about this being "fifth generation programming" is genuinely thought-provoking. ## ✅ Action Items #### DevRel - Create educational content on BDD as specification language for AI agents - Develop workshops on prompt engineering for technical problem decomposition - Build demos showing safe AI-assisted development workflows with proper testing #### Community - Host discussions on evolving code review practices for AI-generated code - Create mentorship programs pairing junior engineers with AI-experienced seniors - Share failure stories and lessons learned from AI agent deployments #### WordPress - **Core Program Team:** Evaluate how AI code generation affects WordPress contribution workflows - **Training:** Develop curriculum for AI-assisted WordPress development with safety guardrails - **Documentation:** Create guidelines for reviewing AI-generated contributions to WordPress core #### The WPCC - **Programs:** Design fellowship tracks teaching behavior-driven development for AI collaboration - **Projects:** Fund experiments in safe AI-assisted open source development practices ## 🧪 Hypotheses 1. **Code review volume hypothesis**: Traditional line-by-line code review becomes impossible when AI generates 10x more code than humans, forcing a shift to architectural and behavioral validation 2. **Junior engineer learning curve hypothesis**: Engineers who start careers post-AI will develop different problem-solving skills than those who learned through manual coding 3. **Safety amplification hypothesis**: Teams with existing deployment confidence will achieve 10x productivity gains, while teams without will experience 10x more incidents 4. **Communication premium hypothesis**: As code generation becomes commoditized, communication and specification skills become the primary differentiator between engineers 5. **Abstraction layer hypothesis**: AI development represents a fundamental shift to "fifth generation programming" where behavior specification replaces implementation focus ## 📊 Methodology **Longitudinal study design**: Track engineering teams over 18 months measuring: - **Safety metrics**: Deployment failure rates, rollback frequency, incident severity before and after AI adoption - **Review effectiveness**: Time spent in code review vs. defects caught, comparing line-by-line vs. architectural review approaches - **Learning outcomes**: Junior engineer progression rates using traditional vs. AI-assisted learning paths - **Communication patterns**: Analysis of PR descriptions, specification quality, and cross-team alignment **Data sources**: GitHub/GitLab analytics, incident management systems, developer surveys, and skill assessment tests ## ❓ Questions - How do we maintain code quality standards when humans can't practically review AI-generated volume? - What specific skills should computer science curricula emphasize in an AI-first development world? - How do we preserve institutional knowledge and architectural consistency across AI-generated codebases? - What new tooling is needed to make behavior-driven development practical at scale? - How do we balance AI productivity gains with the need to develop junior engineers? ## 🔮 Future Research - **AI Code Review Tools**: Automated architectural consistency checking and behavioral verification systems - **Developer Learning Pathways**: Curriculum design for post-AI software engineering education - **Team Dynamics**: How pairing and mobbing practices evolve with AI participants - **Quality Assurance**: New testing strategies for systems with AI-generated components - **Knowledge Management**: Preserving and transferring system understanding in AI-assisted codebases ## 💡 Implications **For Engineering Teams**: Organizations must invest in deployment automation, observability, and behavioral testing infrastructure before scaling AI development. The teams that master this transition will achieve significant competitive advantages. **For Education**: Computer science programs need radical curriculum updates focusing on system design, specification writing, and problem decomposition rather than syntax memorization. The industry faces a potential knowledge transfer crisis. **For Tooling**: Massive opportunities exist for developer tools that facilitate behavior-driven development, architectural consistency checking, and AI-human collaboration workflows. **For Career Development**: Individual contributors should prioritize learning communication, system design, and problem analysis skills over language-specific programming knowledge. ## 🧠 Learning Styles Assessment **Reading level**: College level (Flesch-Kincaid ~12.5). Technical discussion with domain-specific terminology but accessible explanations. #### 👀 Visual learners - **Present**: Structured headings, bullet points, clear topic separation - **Improve**: Add workflow diagrams showing AI-human collaboration, before/after deployment process charts, skill development roadmaps #### 🔊 Aural learners - **Present**: Original video format with multiple speakers and discussion - **Improve**: Chapter markers for key topics, audio summary highlighting main takeaways #### 🗣️ Verbal learners - **Present**: Definitions of key terms (BDD, agentic engineering), analogies to traditional development - **Improve**: Glossary of AI development terminology, story-driven examples of successful implementations #### 🧩 Kinesthetic learners - **Present**: Actionable recommendations, specific practices to implement - **Improve**: Step-by-step guides for implementing new code review processes, hands-on exercises for BDD specification writing #### 🧠 Logical learners - **Present**: Cause-and-effect relationships, structured arguments, hypotheses with testing methods - **Improve**: Decision trees for when to use different AI development approaches, metrics frameworks for measuring success #### 🧑‍🤝‍🧑 Social learners - **Present**: Panel discussion format, multiple perspectives, community recommendations - **Improve**: Discussion prompts for team retrospectives, collaboration frameworks for AI-human teams #### 🧘‍♀️ Solitary learners - **Present**: Individual skill development focus, self-assessment opportunities - **Improve**: Personal learning checklists, reflection questions on current practices ## 📊 Accessibility Assessment #### Screen reader semantics - **Present**: Proper heading hierarchy, descriptive link text for timestamps - **Improve**: Alt text for any referenced diagrams, ARIA landmarks for major sections #### Keyboard navigation - **Present**: Logical tab order through content sections - **Improve**: Skip links for long transcript sections, keyboard shortcuts for timestamp navigation #### Contrast and readability - **Present**: Standard text contrast meets WCAG guidelines - **Improve**: Ensure code examples maintain contrast, consider high-contrast mode compatibility #### Text resizing and reflow - **Present**: Responsive text that scales properly - **Improve**: Ensure timestamp links remain functional at 200% zoom #### Media and captions - **Present**: Full transcript provided for video content - **Improve**: Structured transcript with speaker identification, downloadable format option ## 🌍 DEIB Assessment #### Cultural breadth - **Present**: Global perspective on engineering practices, acknowledgment of English language bias in AI - **Improve**: Include examples from non-English speaking development teams, address AI bias in different cultural contexts #### Inclusive language - **Present**: Gender-neutral terms for engineers, accessible explanations of technical concepts - **Improve**: Define industry jargon more consistently, provide alternative terms for complex concepts #### Neurodiversity support - **Present**: Clear structure with consistent formatting, chunked information - **Improve**: More frequent headings, visual breaks between concepts, consistent terminology usage #### Representation - **Present**: Mix of speakers with different backgrounds and company sizes - **Improve**: Include perspectives from underrepresented groups in tech, address accessibility of AI tools #### Psychological safety - **Present**: Acknowledgment of challenges facing junior developers, realistic assessment of difficulties - **Improve**: Content warnings for career anxiety discussions, resources for those struggling with AI transition ## ✅ Logical Validity #### Strong reasoning - **Causal analysis**: The panel effectively establishes causal relationships between existing practices and AI success (location: throughout discussion). The logic is sound that AI amplifies existing patterns. - **Inductive reasoning**: Multiple panelists independently reach similar conclusions about code review evolution, strengthening the argument through convergent evidence. - **Problem decomposition**: Dave Farley's breakdown of why traditional review fails (volume, speed, human limitations) follows logical steps. #### Fallacies - **Hasty generalization**: Jean's claim of "99% AI-generated code" at Kilo may not represent broader industry reality. The sample size of one early-stage startup limits generalizability. - **Appeal to authority**: Some recommendations rely heavily on panelist credentials without supporting evidence (Dave's "fifth generation programming" concept needs more substantiation). - **False dilemma**: Occasional framing suggests binary choice between traditional methods and AI-first approaches, when hybrid models may be optimal. #### Tone of voice - **Cautiously optimistic**: Despite acknowledging serious challenges, panelists maintain that adaptation is possible with proper practices - **Pragmatically experienced**: Grounded in real-world implementation challenges rather than theoretical ideals - **Collaborative**: Multiple perspectives building on each other rather than competing viewpoints ## 🛡️ DARVO Analysis No DARVO patterns detected. The discussion maintains focus on technical and process challenges without defensive rhetoric, blame-shifting, or attempts to reframe responsibility. Panelists acknowledge difficulties honestly while providing constructive solutions. ## 🎯 Conclusion The transition to AI-assisted development isn't optional—it's happening whether teams are prepared or not. The organizations that will thrive are those that recognize AI as an amplifier requiring stronger, not weaker, engineering fundamentals. The immediate priority is evolving code review practices from line-by-line inspection to architectural validation. But the deeper challenge is reimagining software engineering education and career development for a world where specification and verification skills matter more than syntax knowledge. For engineering leaders, the message is clear: get hands-on with AI tools to understand the challenges your team faces, then invest in the deployment automation and behavioral testing infrastructure that makes AI-speed development safe. The teams that master this transition first will achieve significant competitive advantages in the AI-driven future of software development. ## 🔄 Refactor Appendix — Safe production changes with agents ### 📚 Literature Atoms (H3) #### AI amplifies existing engineering practices rather than replacing them > [!quote] ID:transcript_00_03_45 > DORA report echoed what we kind of all knew but really gave solid feedback and statistics that AI is an amplifier of everything, the good and the bad. If you were already a high performing team, AI enables speed safely at scale. If you were less than that, it's just more code shoved into the pre-existing bottleneck. **Context & use**: - Core thesis established early in discussion - Directly contradicts "AI fixes bad practices" narrative **Tags**: #amplification #engineering-practices **Highlight**: [view highlight](https://www.youtube.com/watch?v=6bHxMYeatHs&t=225)  #### Traditional line-by-line code review becomes impossible with AI-generated code volumes > [!quote] ID:transcript_00_15_30 > I spoke to somebody not very long ago who reckoned that he was producing on his own 12,000 lines of code a day with AI assistance. Who on earth can honestly review 12,000 lines of code a day. I can't not and understand it. **Context & use**: - Identifies critical bottleneck in AI development workflow - Quantifies the scale mismatch between generation and review **Tags**: #code-review #scalability **Highlight**: [view highlight](https://www.youtube.com/watch?v=6bHxMYeatHs&t=930)  #### Programming shifts from implementation to behavior specification with AI > [!quote] ID:transcript_00_28_15 > I think this is a fundamental change. If typing code was your bottleneck in the past, you were doing it wrong anyway because that's not the hard part. The hard part is understanding the problem and verifying that your solution fits the problem. **Context & use**: - Dave Farley's "fifth generation programming" concept - Reframes what engineering work becomes post-AI **Tags**: #abstraction #problem-solving #behavior-specification **Highlight**: [view highlight](https://www.youtube.com/watch?v=6bHxMYeatHs&t=1695)  ### 🧠 Permanent Candidates (H4) #### Code review must evolve from implementation to architectural validation **Claim**: When AI generates code at volumes exceeding human review capacity, effective code review shifts focus from line-by-line correctness to architectural consistency and solution appropriateness. **Evidence pointer**: [[#Traditional line-by-line code review becomes impossible with AI-generated code volumes]] **Scope & conditions**: - Holds when AI code generation exceeds ~1000 lines/day per developer - Requires strong automated testing and specification practices - Breaks down if architectural standards aren't clearly documented **Links**: - [[Continuous Integration]] — because applies-in high-velocity development - [[Technical Debt]] — because extends into AI-generated code management - [[Code Quality]] — because contrasts on review methodology  ### ✏️ Fleeting Queue - How do we measure architectural review effectiveness vs line-by-line review? — https://www.youtube.com/watch?v=6bHxMYeatHs&t=2115 #fleeting #metrics - What does "behavior-driven development as AI specification language" look like in practice? — Need concrete examples #fleeting #bdd ### → Things 3 (max 2 actions) - [ ] Research current BDD frameworks for AI specification use cases — engineering team evaluation #things - [ ] Survey team on current code review pain points with AI-generated PRs — team process improvement #things --- ## Full Transcript [00:12] Hello, hello, hello everyone and a huge [00:15] thank you to joining us today for [00:17] today's panel. This is a lead dev online [00:20] panel discussion created in partnership [00:22] with octopus deploy, one of the cutest [00:25] of the mascot animals. This discussion [00:28] will last about roughly 45 minutes after [00:30] which both myself and the panelists will [00:32] head over to the lead dev Slack to [00:34] answer some of your questions in the [00:35] panel Q&A channel. For those of us [00:38] joining us live, I'm about to drop a [00:39] link in the chat. The Slack isn't just [00:43] for these live events. It's just a [00:44] really great community of engineering [00:46] leaders and staff engineers supporting [00:49] and learning from each other. And at the [00:50] speed of AI, there's more to learn [00:53] about, right? Constantly. And as [00:57] speaking as a journalist, we all need to [00:59] share our lessons learned better. So we [01:02] may have time to answer a couple of [01:03] questions live. So please do submit [01:05] those through the Q&A feature on Zoom [01:08] and I'll sprinkle them throughout if [01:11] they are topical. So thank you so much. [01:13] So I'm not going to lie, part of what [01:16] we're talking about is things you've [01:18] heard before. After all, certainly the [01:21] people on this call have been banging on [01:22] about engineering best practices for [01:24] over a decade now. But now there's just [01:27] very much more code to consider being [01:29] written and released by nonhumans. Is [01:31] that an acceptable term for AI? We'll [01:33] see what they say. So today we are going [01:37] to talk about how to do it safely. Not [01:39] all easy solutions, but if you don't do [01:42] them, your business and your end users [01:44] will surely lose. Plus, now's a good [01:46] time to implement all those continuous [01:48] delivery platform engineering testing, [01:51] uh, behavior-driven design, and user [01:54] centric best practices that you've been [01:55] putting off because you don't have the [01:57] budget for them. But now, if you say it [02:01] enables AI, you may get the budget. So, [02:04] let's just get started with some [02:06] introduction. That's my new theory right [02:07] now. Just say it's for AI and you'll [02:09] probably get the budget, especially [02:10] people's jobs. My name is Jennifer [02:13] Higgins and I'm your favorite freelance [02:15] tech journalist including with lead dev. [02:18] Today I'll be joined by Heidi Jean and [02:20] Dave ready to share their experiences [02:23] and insights. And remember we all have a [02:26] lot of experience on the call but [02:28] everyone's kind of a novice with AI [02:30] agents. We're still newbies or noobs as [02:33] maybe the kids don't say anymore. So, [02:35] can you please each introduce yourself [02:38] and briefly say what do you think the [02:40] title question, what safe production [02:43] changes with AI agents even means? Dave, [02:47] I'll let you go first for this one. [02:49] >> Okay. Hi, my name is Dave Farley. I'm a [02:52] consulting software engineer. Um, I say [02:55] that because it's an internal joke [02:57] because I like Sherlock Holmes and he [02:59] was a consulting detective. Um, I'm like [03:03] everybody else. I'm fascinated and [03:06] interested in what the future of our [03:09] work looks like. I think it's changing [03:11] profoundly. [03:14] And [03:15] maybe because I'm a grumpy old man or [03:17] maybe because I've just seen a bunch of [03:20] things. I think that what we are seeing [03:23] is a deeply profound changing that's [03:25] saying some of the fundamentals are [03:28] still fundamentally important and there [03:30] are important ideas that we've learned [03:33] about how to do a good job of software [03:35] development safely that are even more [03:39] important now than they were before. [03:42] >> Fair enough. Yeah. [03:45] And will they still be ignored? We'll [03:47] see. Jean, I'll let you go next. [03:50] Hi, I'm Jean Dipli. Uh, I'm an [03:53] engineering leader. Um, and I've been in [03:55] industry for over 20 years. Not as much [03:58] as Dave, but uh, I've got a little bit [04:00] of gray developing already. Um, I have [04:03] spent my whole career in startups uh, [04:05] across a variety of industries and uh, [04:08] currently work at Kilo where we're [04:10] building a platform for agentic [04:12] engineering. Um, and specifically over [04:14] the last 18 months, I've been focused on [04:16] on helping engineering teams adapt to [04:18] agentic engineering practices. Um, in [04:21] terms of what safe production changes [04:23] with AI agents uh look like and what I [04:25] think about that, I I think you know [04:27] your chance of success to use AI agents [04:30] to safely take changes in production is [04:33] wholly dependent on how safe it was for [04:34] humans before you introduced agents. Um [04:37] you know I think you know there's this [04:39] common saying of AI amplifies everything [04:42] in your software development life cycle [04:44] and if the cracks were there before uh [04:46] the agents they will continue to be [04:48] there with them. Um so we'll see how how [04:51] this evolves. [04:53] >> Absolutely cracks to chasms right now. [04:57] Heidi please introduce yourself. [04:59] >> Hi I'm Heidi Waterhouse. I work at [05:01] Octopus Deploy and I have been going on [05:04] for years about how deployment and [05:07] release are not the same thing. For the [05:09] love of God, please learn to [05:10] disambiguate those. Uh and I think one [05:14] of the interesting things is that we [05:16] really need to define safe. Does that [05:18] mean there are never any mistakes or [05:21] mistakes are small and uh tightly scoped [05:24] and easy to fix? Because I think aiming [05:28] for perfection while a good goal is um [05:33] just going to make you crazy. Whereas [05:35] aiming for affecting the fewest people [05:38] possible with the smallest possible [05:40] mistakes is is the best failure mode. [05:45] >> Fair enough. And failure modes are [05:46] abound. just nobody's talking about [05:48] everyone's talking about their [05:49] percentage increase [05:52] of productivity that is warranting so [05:55] much change that's not happening for the [05:58] end user. So yeah, last year's DORA [06:01] report echoed what we kind of all knew [06:03] in a way but really gave solid feedback [06:06] and statistics that AI is an amplifier [06:09] of everything, the good and the bad, the [06:10] ugly. If you were already what they've [06:13] what Dora calls a high performing team, [06:15] AI enables speed safely at scale. If you [06:18] were less than that, we'll call it uh [06:22] you're not a failure, you're just [06:23] learning. It's just more code shoved [06:25] into the pre-existing bottleneck. So I [06:28] want to start off with popcorning which [06:31] pop just everybody call out uh [06:35] what the many things that you believe [06:38] create stable systems are and that are [06:41] even more important in the age and speed [06:44] of AI and agentic AI. Just jump right [06:47] in. What do we got? [06:48] >> Working in small steps [06:51] >> scoping. Yeah. Yeah. [06:53] >> Easy roll backs. [06:55] >> Easy roll backs. [06:57] specifying what we want and verifying [07:00] that we get it. [07:03] >> Incremental deployments, [07:07] >> keeping things um Yeah. No, that's the [07:11] same as Dave. Never mind. [07:12] >> Small. Keep it Keep it small. [07:16] >> Yeah. [07:16] >> Good observability. [07:19] >> Yeah. [07:20] It's across the board. All of the we're [07:23] starting to iterate and repeat ourselves [07:25] because across the board what we need to [07:27] do is that we need to gather feedback [07:28] quickly, respond to the feedback and [07:30] adapt. And it's that do that and repeat [07:34] and keep doing that in small steps with [07:36] AI and guide it. Steer it [07:39] >> and don't let it have too many [07:41] permissions. All of the hilarious uh [07:44] fail stories that we've seen about AI [07:46] are about overcoped permissions. [07:50] Mhm. [07:50] >> It's all about having confidence in the [07:53] process of taking code into production. [07:55] >> Yeah. [07:56] >> If your processes are set up to give you [07:58] that confidence, your AI will will [08:01] hopefully be okay as well. But if you [08:04] haven't had that confidence before, if [08:06] you push the button or it runs [08:08] automatically and you hope for the best, [08:10] um you're likely going to endure some [08:12] chaos. [08:14] >> I I think there's an intriguing idea. [08:16] I'm sorry. I think there's an intriguing [08:19] idea in in [08:21] relation to what we've already talked [08:23] about about it employing the good or the [08:25] bad is that what that does is it makes [08:31] your choices even more important because [08:35] you're going to dig a hole and create a [08:37] big mess really fast if your practices [08:40] aren't good. And you're going to be [08:42] flying. you're going to be creating good [08:44] stuff really fast if your practices are [08:46] good and that's increasing the [08:49] differentiation. There was enough slack [08:51] in the system in the past where [08:55] without being too rude about too many [08:57] organizations [09:00] average kind of barely adequate [09:02] performance was survivable. I don't [09:05] think that's going to last very long. I [09:07] don't think that average or you know [09:10] barely adequate performance is is going [09:12] to be a disaster. I think [09:15] >> I think that sorry go ahead Sean. [09:18] >> Okay. No thanks. Um I'll quickly add I [09:20] think also that the fact that uh humans [09:23] used to be a bottleneck and that [09:25] actually allowed us the opportunity to [09:27] catch up with with things that could go [09:29] wrong. You know the process was slow and [09:31] so you know there was an opportunity to [09:33] spot things that went wrong. But as AI [09:36] takes over some of these tasks and [09:37] automates it, you know, the [09:39] acceleration, you know, is there and it [09:41] doesn't give us the opportunity to in [09:43] interrupt it, you know, through human [09:44] observation and so the systems are even [09:47] more important. [09:48] >> Yeah, absolutely. There's [09:51] >> Yeah, I was I was going to say that one [09:53] of the things that I think we've learned [09:55] is that, you know, we used to be able to [09:57] do heroic human intervention. There used [10:00] to be like that one person who could [10:02] save a bad deploy and hit the stop [10:04] button. And we robots are not going to [10:07] pull pull the end on cord. They do not [10:10] have the the capability to say like [10:13] stop. This is not okay. [10:16] >> Okay. [10:18] Is that a problem? Do we need human [10:21] bottlenecks because code is going too [10:24] fast? Now [10:26] >> I I don't think that human bottlenecks [10:29] are fast are fast enough for that to [10:31] work. [10:32] >> So I think we have to automate those [10:34] things. We've got to [10:38] I think this is a fundamental change. To [10:41] be honest, if [10:44] typing code was your bottleneck in the [10:46] past, you were doing it wrong anyway [10:49] because that's not the hard part. The [10:51] hard part is understanding the problem [10:53] and verifying that you that your [10:55] solution fits the problem and all those [10:57] kinds of things is much harder than just [10:59] the typing just the translation into [11:01] code. It means if that was if that was [11:04] if that was the slow bit you weren't [11:06] very good at writing code is in my view. [11:08] So, so, so that so we've optimized to [11:12] speed up the bit that wasn't really the [11:14] problem and then we've left all the bits [11:16] that were really the problem and they're [11:18] still there and they can't keep up with [11:20] the writing of the code now. So, I spoke [11:22] to somebody not very long ago who [11:24] reckoned that he was producing on his [11:26] own 12,000 lines of code a day with aic [11:32] assistance. [11:34] Who on earth can honestly review [11:38] 12,000 lines of code a day. I can't not [11:42] not and understand it. [11:44] >> Did his company We've always said I I [11:47] think lines of code is the only [11:49] interesting metric is to measure [11:51] complexity in your company. We've know [11:54] lines of code is a bad metric. [11:55] >> Yeah. [11:56] >> Do we want individual teammates writing [11:59] 12,000 lines of code or overseeing AI [12:02] overseeing AI doing it? [12:04] >> No. But all of the company, not all of [12:06] the companies, I'm I I apologize because [12:10] Jean is representing one of some of [12:13] these companies that are that are [12:15] talking about agentic programming tools [12:17] are certainly pushing that message that [12:19] that you know we this is valuable. [12:22] Generating code fast, developing [12:24] software fast is what matters. How fast [12:27] it writes the code is secondary it seems [12:30] to me. And so so I I want to be able to [12:32] build software faster. I think that's [12:34] great and I can using aentic assistance [12:38] when I'm doing the right things, but if [12:40] I'm sloppy, it creates crap really fast. [12:45] >> Yeah, I want to talk to our respectfully [12:47] baby startup in the room. Uh, Jean, what [12:51] percentage of your startup's code is [12:55] written by AI? [12:57] >> Would has it that it's [13:00] confidently I would say probably 99%. [13:03] Um and and that's not because we we're [13:07] reckless and stuff, but we have um we're [13:10] very early on, you know, try to adapt to [13:14] what does the new build look like? Um [13:16] we're still figuring it out internally. [13:18] We're we're just as confused as [13:20] everybody else out there. We in our [13:22] engineering meeting the other day, we [13:24] had a conversation. Like we see people [13:26] talking about, you know, running 15, 20 [13:28] agents in parallel and we're like, but I [13:32] can't do that. And then you have [13:33] engineers thinking like they're lesser [13:36] um engineers because they can't run that [13:38] many in parallel. But the reality is you [13:41] as a human cannot um have that much of a [13:45] split focus and deliver quality work. [13:48] And I think what a lot of people, you [13:50] know, talk about is, you know, they [13:52] might have a bunch of things running in [13:54] the background doing small little things [13:56] and so on, but you know, when in my own [13:58] experience, I find that, you know, two [14:02] maximum three topics at a time I can [14:04] focus on. Um, and so I think, you know, [14:08] the there's a lot of falseeness out [14:11] there. people, you know, looking for [14:13] attention, you know, companies wanting [14:15] to, you know, um, kind of like make big [14:18] statements because it's in their [14:19] interest for people to adopt their their [14:21] techniques or products. But I think, you [14:23] know, the reality is we're all still [14:25] learning and we're at the beginning of [14:27] this. U nobody that tells you for [14:30] certain they know the future of what [14:32] engineering looks like really speaks [14:34] from [14:36] experience. But we're in May, so a lot [14:40] of people, probably four or five million [14:43] at least, people graduated with [14:46] engineering degrees. [14:48] What What do they do with that? How do [14:51] they They've always been It's always [14:52] been a flawed system where it's been a [14:54] bit too much green field. But if you [14:57] were advising, as many of our listeners [14:59] are advising hopefully junior engineers, [15:02] because I think we all agree that you [15:04] can't have senior engineers without [15:05] junior engineers. So we need them in the [15:07] industry. It's probably the least [15:09] welcoming time for them. So what should [15:12] they work on right now? What skill sets [15:15] are they not being taught in slow [15:17] curricula? [15:18] >> I think you know it is probably the [15:20] hardest time that I can remember in my [15:23] career for junior engineers because [15:26] there's this expectation of things need [15:28] to move fast and nobody has patience for [15:30] anything anymore. And what you as a [15:33] junior need is people to make themselves [15:36] available to share their knowledge with [15:38] you. Now you can hope for the ideal or [15:42] you can make the most of what the [15:43] reality is. And in these LLMs, you know, [15:46] you can get a good teacher as well. Um, [15:49] you know, you can tweak it a uh its [15:53] system prompt to, you know, make sure [15:55] that it explains things to you and it [15:57] doesn't just do it. you know, ask [15:59] questions and don't let the the LLM [16:01] implement it for you, but do it yourself [16:04] until you understand it. You know, I I [16:06] think the [16:08] the the trap is that as a junior [16:10] engineer, you're going to think that the [16:12] AI is going to do everything for you. [16:14] >> It's not going to [16:16] >> I don't think that's just junior [16:17] engineers. That's not going to teach you [16:19] the judgment and the taste uh kind of [16:21] like skills you need to learn to know [16:24] what is the right thing to do because [16:25] the LM will confidently tell you what it [16:27] thinks is is the right thing to do. But [16:30] do you know when to push back? And that [16:32] unfortunately happens through, you know, [16:34] experience. And that experience, you [16:37] know, I mostly gained through other [16:39] people, you know, whether it was code [16:41] reviews or or kind of like, you know, [16:45] listening to talks or whatever and and [16:46] just team members sharing their [16:48] experience. Um, and so it's a very [16:51] unwelcoming environment at the moment [16:53] where businesses are under pressure to [16:56] to prove that AI can accelerate them. [16:58] they need to justify their AI spending [17:00] and and so it creates a hostile [17:02] environment for juniors and so I do feel [17:05] like there's a there's a a message in [17:07] the industry coming up you know we got [17:09] to slow down as well it's not as loud as [17:11] the voices of those who say you know we [17:13] got to move fast um but yeah it's it's a [17:17] tough time but you cannot be despondent [17:19] about it the reality is yes we need [17:22] junior engineers because they are [17:24] tomorrow senior engineers um but they're [17:27] going to have to adapt adapt to the new [17:28] reality and not just rely on on the [17:31] organic transfer of knowledge that used [17:33] to happen in in companies. Um, you know, [17:36] if you're working remotely as a junior, [17:37] it's even harder because you don't can't [17:39] just tap somebody on the shoulder to ask [17:41] them a question. So, yes, tough [17:43] environment, but it is possible. You [17:45] just have to um, you know, approach it [17:48] differently. [17:49] I think that one of the the things that [17:52] we always should have been teaching and [17:54] this is my bias is we need to teach them [17:56] to write better. I think that prompt [17:59] writing is a skill and that maybe that's [18:04] a thing we should concentrate on. Like [18:06] like Dave says, how do you know you're [18:08] getting what you want? How do you write [18:10] a test for an agentic system? like [18:15] can you get it to prove that it's doing [18:17] the thing that you asked for? Okay. Did [18:19] you ask for it right or did you you know [18:22] sort of handwave and it assumed the [18:25] middle of the bell curve when you wanted [18:27] something out on the edges. So, I think [18:29] that um it's very interesting to see how [18:33] many people lean on LLMs for their [18:35] writing when writing is a very [18:39] sophisticated form of thinking and we [18:42] need that thinking to understand what [18:44] we're trying to make. [18:46] >> I I I I agree with everything everybody [18:50] else has said so far very much, but I [18:53] just want to take a slightly different [18:54] angle on it. Um I um I think this is a [19:00] horrible problem because [19:03] as as Jean Heidi said, you know, [19:08] certainly we and and you know people [19:10] before us and since since us, you know, [19:14] we learned by doing the the the things [19:16] writing code, making like writing lots [19:19] of code, making mistakes, learning from [19:21] the mistakes, and figuring out what [19:22] works better. that that's certainly the [19:24] way that I learned how to do things and [19:28] we are rem I I I would agree with what [19:32] Jean said earlier is that when I'm using [19:35] it's been a while since I wrote any [19:38] code. I still occasionally write a code. [19:41] I do it for examples to show other [19:43] people what code looks like and I do it [19:45] for fun because I like writing code but [19:48] I don't do it for production. If I'm [19:49] doing production the AI writes the code. [19:52] I think that we are, if we're not [19:55] already past it, we are at a tipping [19:57] point where humans don't write code [19:59] anymore. I think that's where we are. I [20:03] And I think that people that are saying [20:06] you've got to review it and understand [20:07] all of the code does and all that sort [20:09] of things. I think they're wrong. I [20:11] don't any more than you know when high [20:14] level languages came in. We used to look [20:17] at the assembler that was generated and [20:19] say do you know is it doing the right [20:20] things? How often do you do that? We [20:23] don't care anymore. That's gone. [20:26] The code is generated. So, so [20:29] programming has taken this step in [20:31] abstraction. So, so if you perceive [20:34] yourself and part of the our problem [20:36] that we're in as an industry is that [20:39] most juniors start off perceiving [20:41] themselves this way. If you perceive [20:43] yourself as your job being to write code [20:46] rather than to solve problems with code [20:49] then that's where you see the difficult [20:51] part is that's where you see your skills [20:53] and I don't think there are so that so I [20:55] think that the best thing that juniors [20:57] can do is look other places for where to [21:01] apply [21:03] their skills their technical thinking [21:05] and I think that there is there's I [21:09] would call it engineering thinking but [21:11] there's a there's Something about the [21:13] way in which an engineer decomposes a [21:16] problem, thinks about how to carve a [21:18] problem up, what these are, thinks about [21:21] the ways in which things can go wrong [21:22] and the corner cases and those kinds of [21:25] things and analyzes [21:27] in a disciplined technical way, all of [21:30] these things. And those are the things [21:32] that we have to prompt the AIS with [21:33] because if you don't, they'll just write [21:35] crap that won't work. So, so you know, [21:38] we've got to specify things, you know, [21:41] like this. You got to think you you need [21:43] to have the kind of mind where you're [21:45] going to think about those. You've got [21:46] to understand the problem that you're [21:48] trying to solve really well. And you've [21:50] got to be able to work through the [21:52] problem and learn bit by bit, step by [21:55] step by step, you know, what the surface [21:57] area of the problem is that you're [21:58] dealing with and communicate that to the [22:00] AI AI that's then going to generate the [22:03] code. And you do this by specifying [22:06] exe creating executable specifications [22:09] tests if you like that specify what what [22:12] it needs to do and then how you will [22:14] evaluate that that happened and those [22:16] kinds of techniques. But those sorts of [22:18] skills I think are going to be in very [22:20] short supply. So if I was advising a [22:23] junior now I would say try and figure [22:25] out how to learn those skills fast. I [22:28] would say you can't go far wrong by [22:30] learning how BDD works and think about [22:32] that as a specification language. Think [22:35] about that as the way of telling what [22:37] your AI is meant to do and I think that [22:39] will give you that will put you on the [22:41] right track I think. [22:43] >> So let's loop back and say BDD is [22:46] behavior de driven development like what [22:49] do you want as an outcome not how it [22:51] gets there. Yeah, [22:52] >> but what do you want to have happen? And [22:55] yeah, that kind of thinking is going to [22:57] be so vital. And it is something that we [23:00] used to reserve for sort of mid-level [23:02] engineers. Like you didn't get to do BDD [23:06] until you had done grunt work on like [23:09] how do I make this work at all. [23:13] >> Okay, we have some questions from the [23:15] audience that are excellent follow-ups. [23:17] I'm gonna take the more recent from [23:18] Greg. Uh, and then we're going to go [23:21] into the reviewing of it all because [23:23] that is the bottleneck with AI and we [23:25] have a bunch of questions with that. But [23:27] Greg says, "Do we need to fundamentally [23:29] change how we teach people software [23:31] development?" We always did. Uh, how [23:34] much effort they should be putting into [23:36] learning to code, programming, language, [23:39] syntax. Uh, I would venture to say [23:42] something we haven't talked about, but [23:44] this is the problem with open source [23:45] always anyway. the domination of the [23:47] English language [23:49] and is that going to bias people that [23:52] aren't native English speakers or as [23:54] compelling writers as those of us in [23:56] this digital room? [23:59] Yes, I I think we all agree we need to [24:01] fundamentally change how we teach [24:03] software development. How [24:08] >> so I sorry go ahead please. [24:11] >> Okay. I think that [24:14] it's very interesting. I was talking to [24:15] a couple juniors last week and they're [24:17] like, "Oh, our our CS professor is so [24:20] old-fashioned. Um, she's teaching us [24:23] like um Java." And I'm like, "Oh, Java [24:26] was what they were teaching when I was [24:28] in college, which was quite a while [24:29] ago." And I'm like, "Actually, it [24:31] doesn't matter. What matters is that you [24:34] learn that you can affect how a computer [24:37] behaves. That you know how to structure [24:42] something so that it communicates with [24:43] the computer and that you understand [24:48] how to make a small change and see the [24:50] result. Like I learned to code HTML with [24:52] a program called Hot Dog Pro which had [24:55] the beautiful advantage of you didn't [24:57] have to compile. You didn't compile HTML [25:00] but whatever. You didn't have to upload [25:02] it. it would show you in real time what [25:04] change you were making so that you could [25:06] see. I think that's what we need is like [25:09] the ability to say like this piece of [25:11] code changing does this thing which is [25:14] BDD but we haven't really been [25:18] explicitly teaching that [25:20] >> like a new way of abstraction but tied [25:23] to business goals to your end users. [25:26] Yeah, I I it's not just business goals, [25:29] I think, but it's what the behaviors are [25:32] that you want of the system, but I I [25:34] think that what we are seeing, what I am [25:36] seeing in in the way that I use AI [25:38] assistance anyway, and I'm doing things [25:40] is genuinely a step to a kind of fifth [25:43] generation language. So, this is kind of [25:46] programming at a different a genuinely [25:48] different level of abstraction. If you [25:50] think about the way in which regular [25:52] programming worked [25:55] kind of weirdly in with the benefit of [25:57] hindsight was that we concentrated less [26:01] on the problem that we're trying to [26:02] solve and more on the solo solution that [26:05] we were trying to solve it with. So the [26:06] the solution was often kind of [26:10] implicit. It it kind of wasn't called [26:12] out anywhere very much. If you were very [26:14] good, you might have some tests that [26:16] said something about the solution. But [26:18] other than that, the sorry about the [26:21] problem, but the solution itself was [26:23] was, you know, was the code and that was [26:25] the deep focus that's gone. So, so now [26:29] what we're talking about is absolutely [26:32] specifying the behaviors that we want of [26:34] this system. I want to make a game. I [26:36] want when this the bullet hits this [26:39] thing, that thing blows up and this [26:42] thing goes off the screen or whatever [26:43] else. I'm going to describe the [26:45] behaviors that I want of the system. But [26:48] in addition to that, there are other [26:49] kinds of behaviors that we as technical [26:52] people [26:54] are probably the only people that we'll [26:56] be thinking about. And you'll say, so [26:57] okay, I'm going to think about it's a [26:59] big deal when two people are doing the [27:00] same thing at the same time. So, if [27:03] there's another player going on at the [27:05] same time, what does that mean in terms [27:07] of the way that things interact? What if [27:09] that person fires a bullet and this [27:11] person fires a bullet at the same time? [27:13] Is that going to break or anything? I [27:15] want to want to think about those sorts [27:16] of things because concurrency is a [27:17] complicated thing. If I'm building some [27:20] Let's move away from games because it's [27:22] I'm I'm starting to struggle with my [27:23] analogies. But but you know if if I'm if [27:27] I'm building a bookshop or something [27:29] like this, you know what happens if the [27:31] book's out out of stock? Those are [27:34] business problems, but they're also [27:35] technical problems. What happens if [27:37] there are too many people trying to work [27:38] with it at the same time? How are we [27:40] going to deal with those sorts of cases? [27:42] We need to figure out all of those sorts [27:44] of behaviors. And that's a mix of [27:47] understanding the behavior of the system [27:49] and what we classically have talked [27:52] about as kind of the architecture and [27:54] the design of the system. So we need to [27:57] worry about that stuff too and we need [27:59] to think about how can I compartmental [28:02] compartmentalize this so that my AI [28:04] assistant doesn't go crazy and do [28:05] something mad. So I still want a modular [28:08] system. I still want to be able to [28:10] change it easily when I learn more and [28:12] get things wrong. I don't want the AI [28:13] coming in and breaking everything. I [28:15] still want to be able to validate that I [28:16] got all of the things that I had before [28:18] when it makes a change because I now now [28:20] I'm not relying on the code anymore [28:21] because it's probably going to change [28:23] more of the code than I expect when it I [28:24] ask it to change things. So all of these [28:26] things those are the kind of engineering [28:28] disciplines that seem at the core of [28:29] what we're doing now. And we didn't talk [28:31] security or anything like that, but does [28:34] that mean that like you the the agent [28:37] has to should by default selfch check [28:40] that it's not leaking private? [28:42] >> No. No. So So I I I think it will help [28:44] if it does a bit, but we've got to think [28:46] about that as well. We've got to think [28:48] about those risks. We we can't rely on [28:50] those things. But those are also [28:51] behaviors of the system. So I want it to [28:53] be secure. I want to make sure that when [28:55] I pay by credit card, my credit card [28:57] isn't isn't leaked onto the dark web. [28:59] and you know [29:02] and killed. [29:05] >> Okay, so we know one of the biggest [29:06] problems is reviewing all this code [29:09] because [29:11] it's exponential. So we have actually [29:13] two questions that kind of touch each [29:14] other on this from fabulous but [29:16] anonymous attendees. So thank you for [29:18] your questions. Someone said they're [29:21] very interested in the Gentic review [29:23] loop assuming we have strong CI/CD [29:25] practices and observability in the rest. [29:27] you make that. [29:28] >> Um, a lot of companies don't still, but [29:31] it seems that the big gap is the review [29:33] loop. What's stopping us from yolo? You [29:36] only live once mode. How do we enable [29:38] humans in the loop reviews? How can we [29:41] keep someone saying, "Yeah, it's good to [29:42] go while opening that bottleneap up a [29:45] bit." And then as the other question is [29:48] if humans can't keep up with reviewing [29:50] all the AI code which it's impossible [29:53] then how can we verify that the right [29:55] decisions are being made for example [29:58] architectural decisions that are [30:00] consistent with the existing like yeah [30:03] so how do we make sure the answers what [30:06] we're releasing is right and good [30:08] especially you're not writing code [30:09] anymore how can you know what good code [30:11] is? Yeah. So, this this was a [30:13] conversation I had with uh another [30:15] engineer earlier this week, so I feel [30:17] like I've got it loaded in my context [30:19] window. Um I I think the role or or the [30:24] the function of code review needs to [30:26] change. Our tools are not geared to to [30:30] facilitate the new reality that we live [30:32] with. And our process and and [30:35] expectations that we've placed on on [30:37] individuals for code review isn't [30:40] scaling. you cannot um keep up with the [30:44] amount of code that an AI can produce. [30:46] And not just that, there's a [30:47] psychological component to it when I [30:50] look at a PR with a lot of that's going [30:52] to take me 30 40 minutes to properly [30:54] review, but I know it took the another [30:57] person 5 minutes to generate. There's an [30:59] inequality in terms of the effort that [31:01] goes into that and that makes me less uh [31:03] desirable to to engage with it in that [31:05] manner. [31:07] Something that we've been experimenting [31:09] with at Kilo is pushing more of the [31:11] responsibility of correctness and [31:14] ownership to the PR author you know and [31:17] that doing basic things like you know [31:19] you're expected to run AI code reviews [31:22] as you develop uh locally even you know [31:25] and yes we have an AI code reviewer that [31:27] runs on the PR and you know it will pick [31:29] up a lot of issues but what we're not [31:32] expecting you know the code reviewer to [31:34] uh to do anymore is you read the code [31:38] line for line, load the whole context up [31:41] because it's too much and it's too too [31:43] too fast to to do all of that. And so [31:47] through things like our PR templates, [31:49] you know, where we're asking the author [31:52] to specify, you know, how did you verify [31:54] this? Uh, you know, add some screenshots [31:56] or a walkthrough video of the feature. [31:59] Um, you know, and obviously what the the [32:02] the role of the reviewer moves to a [32:04] higher level. then it's more like is [32:07] this even the right solution to the [32:09] problem? Um, you know, one of my [32:12] favorite things to add in my PR [32:14] description is what was the problem and [32:16] how does this PR solve it? Because as a [32:19] code reviewer coming in now, I think [32:21] that's the most important thing you can [32:23] look for is is this the right solution [32:25] to the problem? Is the problem [32:28] correctly? [32:29] >> Yeah. Is this even the right problem? Um [32:31] and and then you know through domain [32:34] expertise and understanding of an [32:36] existing system you know look at it is [32:38] this architecturally the right way we [32:40] want to go. Does this match the the [32:43] systems we have uh and processes we have [32:45] on our in our business and so I think [32:47] the the the function of the reviewer is [32:50] changing. The sad thing about it is that [32:53] code reviews was a great way to learn [32:56] about things and the you know now that [32:59] we're seemingly engaging less with the [33:02] the content of it um you know it's a it [33:05] it takes away some of that learning and [33:07] sharing of knowledge uh in in the system [33:10] but I think you know through better [33:13] tooling uh and uh more normalization of [33:17] what code review looks like in the in [33:19] the world of agentic engineering we can [33:22] still get to to being able to understand [33:24] and and learn and share knowledge [33:26] through code reviews even. [33:30] >> Absolutely. We have another question [33:32] that I really love because I've heard I [33:34] I love the idea of what's happening in [33:36] team size. the interesting maybe it's my [33:38] bias is not a codew writer though nobody [33:41] will be now but maybe it's my moment but [33:44] the idea that the most interesting [33:47] changes and I think this is the lead dev [33:49] audience in us all are not technical [33:53] it's the people and processes to respond [33:57] and Paul brings up a great point and I [34:00] asked the room what do you think the [34:02] future of pairing and mobbing practices [34:05] are now that an agent is another and [34:07] probably the only driver in the group. I [34:09] don't think it should be the only, but [34:10] we'll see. As an advocate of pair [34:12] programming, as a review process, a [34:15] sometime harness for testdriven [34:18] development, and crucially a knowledge [34:19] sharer, I'm seeing my team default more [34:21] to working with an agent as a pairing [34:23] partner rather than another human. I [34:27] mean, I I call it throbbling now. I [34:30] think there can be agents involved but [34:31] two humans is definitely how to learn [34:34] especially when we have that gap of [34:36] maybe a junior senior pairing which [34:39] juniors may be more comfortable with the [34:40] AI maybe too comfortable in things but [34:43] how are you seeing these like [34:45] traditional agile processes adapting [34:49] >> so I I I I certainly don't think that [34:52] there's any barrier to the value of [34:55] pairing along with the AI you get get [34:58] the it feels a little bit closer to [35:01] mobbing in some ways as somebody said [35:04] because the you know the the AI is doing [35:07] doing the actual code generation but um [35:11] but I've I've done that a few times [35:12] where I've paired with people as we we [35:14] discussed the problem and and had some [35:16] good results. I think that works well [35:18] and as you said that's a good way of [35:19] learning spreading knowledge amongst the [35:22] team working together. I think that [35:25] works effectively. [35:27] Um, and I think it I I've never I've [35:31] long been [35:33] thinking that pull request models for [35:36] code review were second best anyway. And [35:39] I still think that they still are. So [35:42] they they lag too much that we want the [35:44] we want that conversation at the moment [35:46] when we need that conversation, not a [35:48] few hours later. So So pairing pairing [35:51] is good. It puts us in the right place. [35:54] I recogni when people started saying [35:57] that working with AI assistants probably [36:00] a couple of years ago now started saying [36:02] that working with AI assistants was was [36:04] like pairing I didn't see it that's not [36:07] how it felt to me they felt like smart [36:09] complete to me at that stage but now [36:11] they are now now working working with an [36:14] agent I I use Claude mostly when I'm [36:17] programming working with Claude it feels [36:19] I feel like I'm working with a slightly [36:22] weird person it It feels it feels like [36:25] that. Um and and that's a good that's a [36:28] good thing from my point of view that it [36:31] feels like a collaborative effort on our [36:33] part. He he it does a bunch of things um [36:39] that I don't do and I do a bunch of [36:41] things that it doesn't do and together [36:43] we get a better better result. So So I'm [36:46] okay with that too. But I I think that [36:48] pair pairing is a good use of time and [36:51] may be part of the solution for what we [36:54] do about the junior problem because the [36:56] junior problem's a serious one because [36:58] we're going to run out of seniors really [37:01] fally soon. You know, at the rate that [37:04] things are going that there's not much [37:07] hiring of juniors going on at the moment [37:08] in many parts of the industry. That's [37:10] not a good thing. [37:13] I think one of the one of my big [37:16] hesitations about AI is its [37:17] non-deterministic nature. [37:19] >> Um, humans are also non-deterministic. [37:22] So, it's not like they're perfect at [37:24] reviewing code and they might give you [37:25] different feedback on the same PR on [37:28] different days, but for learning [37:32] purposes, it seems to me that I want [37:35] something that will be predictable and [37:39] honestly for for all purposes like the [37:42] ideation everything that's fine, but I [37:45] find it distressing to think about [37:50] differing things going out um without me [37:54] realizing that they're different. And I [37:56] think that's why I'm AI hesitant is [38:00] because like I don't trust it. And I [38:02] think that one of the things that we can [38:04] do with pairing is say like, "Hey, you [38:07] gave me a different answer last time. [38:09] Why why?" [38:11] >> Um because sometimes it's like, I don't [38:14] know, I pulled a different part of the [38:15] context window. And sometimes it's well, [38:18] it's 50/50. Could go either way. [38:22] In either of those cases, it's useful to [38:24] know like, okay, you know, there's [38:26] there's another route or there's another [38:28] reason I'm doing this. So, I think that [38:33] pairing with an AI does get you like [38:38] like it's a very sophisticated rubber [38:40] duck. It makes you say things out loud. [38:42] It makes you understand your [38:44] assumptions, but also it allows you to [38:49] it's very agreeable. And I think for me, [38:53] AI would be more useful if it were less [38:55] agreeable. If it were a little more [38:59] opinionated, [39:01] >> have you to be antagonistic? [39:03] >> I Well, if it I make it antagonistic, [39:06] then it's just a dick. Um, [39:09] >> it is trained on the internet. It is [39:11] trained on the internet, but I'm like I [39:13] want what what I want from it is like an [39:16] opinionated personality [39:19] that will because that's the thing about [39:21] pairing is like why are we doing it that [39:22] way? What's the point of that? How how [39:25] could we do that differently? [39:28] That that collegial push is a large part [39:32] of how we learn from pairing, I think. [39:34] And AI is it rolls over too fast. It's [39:39] like, "Oh, you're right." [39:42] >> Okay, as we wind down, let's just think [39:46] about, we're mainly speaking to the [39:49] engineering managers and future [39:51] engineering managers on this call. What [39:54] advice do you have for them in this [39:57] time? [40:01] I I give I give them a piece of personal [40:04] advice rather than career advice, which [40:07] is if you're the kind of engineering [40:09] manager that hasn't written any code for [40:10] a little while, go and write some code [40:12] with wi with an AI because it's so much [40:15] fun. It's so easy. [40:20] >> Yeah, I'll I'll echo what Dave says. You [40:22] got to get hands-on with with this code. [40:25] The worst thing you can do is to try and [40:28] lead people but not have empathy for [40:30] what they're going through at the [40:31] moment. Whether it's the doubts, the [40:33] uncertainty, the frustration of swearing [40:36] at an LLM because it confidently does [40:39] something stupid. You have to experience [40:42] that with them um to be able to lead [40:44] from a position of authenticity, but [40:47] also to make good decisions. you're [40:49] going to get pressure from up top [40:51] potentially to, you know, do AI [40:53] strategies and stuff and, you know, [40:55] there's an expectation to 10x your [40:57] productivity because this is what this [40:59] person said or, you know, but you have [41:02] to be in the trenches and experience [41:04] these things so that you can have [41:05] informed conversations and share from [41:08] your own battle scars, you know, what it [41:10] is like to to adopt to agentic [41:14] engineering and and feel where where [41:17] things are moving faster, where are the [41:19] bottlenecks, where do you suffer? Um, [41:21] where where's your process not working [41:23] well? And you have to get hands-on, not [41:26] not to be on the critical path of [41:28] feature de development or anything like [41:30] that. Uh, or not to relive your glory [41:32] days as an IC, but really to to to kind [41:36] of like get this experience. The the [41:37] shift is big enough that you cannot rely [41:40] on your your old kind of like [41:42] experiences as as a programmer. [41:46] hire for communication skills and not [41:49] code skills. Uh the ability to explain [41:52] not only to an AI but to your co-workers [41:55] what you're trying to do is even more [41:58] important than it always was. and study [42:01] it yourself. Take a toastmasters course [42:04] or something. But I blame a lot on [42:07] engineering leadership not understanding [42:10] how they connect to business goals [42:12] because it stops with them and they need [42:14] to advocate for their teams. And you [42:17] probably actually have budget to do all [42:19] the continuous integration and [42:21] deployment. This just say it's part of [42:23] AI. That's the best advice. If you say [42:26] it's part of AI, you may get the budget [42:29] for it. Finally, [42:31] >> it's quite good at doing some of it for [42:33] you if you ask it well. [42:36] >> Just always check and trust your teams. [42:39] Keep that door or Slack huddle open and [42:44] we'll continue the conversation in [42:46] Slack. We had some more questions and [42:49] thank you so much Jean, Heidi, and Dave. [42:52] And thank you listeners. Good luck out [42:53] there. We really seem, I would say, more [42:56] optimistic than I thought we would be. [43:00] >> Thank you all for attending. [43:02] >> Thank you.  📄 **Full text:** ![[Full Document Contents/Articles/Safe Production Changes With Agents.md|Safe Production Changes With Agents by YouTube]] ## Highlights - agents. We're still newbies or noobs as maybe the kids don't say anymore. So, can you please each introduce yourself and briefly say what do you think the title question, what safe production changes with AI agents even means? Dave, I'll let you go first for this one. >> Okay. Hi, my name is Dave Farley. I'm a consulting software engineer. Um, I say that because it's an internal joke because I like Sherlock Holmes and he was a consulting detective. Um, I'm like ([View Highlight](https://read.readwise.io/read/01kv3sxaqr8kcg841kmbw21j7v)) - Tags: 👻 ai highlighted - report echoed what we kind of all knew in a way but really gave solid feedback and statistics that AI is an amplifier of everything, the good and the bad, the ugly. If you were already what they've what Dora calls a high performing team, AI enables speed safely at scale. If you were less than that, we'll call it uh you're not a failure, you're just learning. It's just more code shoved into the pre-existing bottleneck. So I want to start off with popcorning which ([View Highlight](https://read.readwise.io/read/01kv3tx57qmjnwzhj25yrkdbmb)) - Note: DORA finding: AI is an amplifier of everything — good, bad, and ugly. High-performing teams get speed safely at scale; weaker teams just shove more code into the pre-existing bottleneck. - Tags: atom/literature/h1, engineering-practices, amplification - work. >> So I think we have to automate those things. We've got to I think this is a fundamental change. To be honest, if typing code was your bottleneck in the past, you were doing it wrong anyway because that's not the hard part. The hard part is understanding the problem and verifying that you that your solution fits the problem and all those kinds of things is much harder than just the typing just the translation into ([View Highlight](https://read.readwise.io/read/01kv3txsa0r221mmg83h9qb7dj)) - Note: Programming shifts from implementation to understanding + verification: "if typing code was your bottleneck, you were doing it wrong." The hard part is understanding the problem and verifying the solution fits. - Tags: atom/literature/h1, problem-solving, abstraction - code. It means if that was if that was if that was the slow bit you weren't very good at writing code is in my view. So, so, so that so we've optimized to speed up the bit that wasn't really the problem and then we've left all the bits that were really the problem and they're still there and they can't keep up with the writing of the code now. So, I spoke to somebody not very long ago who reckoned that he was producing on his own 12,000 lines of code a day with aic ([View Highlight](https://read.readwise.io/read/01kv3txfk79zxxmmcaq6fj7ez0)) - Note: AI optimized the wrong bottleneck: writing code was never the slow part. Now review/integration can't keep up — someone reported producing 12,000 lines of code a day, making line-by-line review impossible. - Tags: scalability, code-review, atom/literature/h2 - Um and and that's not because we we're reckless and stuff, but we have um we're very early on, you know, try to adapt to what does the new build look like? Um we're still figuring it out internally. We're we're just as confused as everybody else out there. We in our engineering meeting the other day, we had a conversation. Like we see people talking about, you know, running 15, 20 agents in parallel and we're like, but I ([View Highlight](https://read.readwise.io/read/01kv3sxas60j24zfas34wkfr75)) - Tags: 👻 ai highlighted - do you know when to push back? And that unfortunately happens through, you know, experience. And that experience, you know, I mostly gained through other people, you know, whether it was code reviews or or kind of like, you know, listening to talks or whatever and and just team members sharing their experience. Um, and so it's a very unwelcoming environment at the moment where businesses are under pressure to to prove that AI can accelerate them. they need to justify their AI spending ([View Highlight](https://read.readwise.io/read/01kv3sxatx385hgea8mqapwe06)) - Tags: 👻 ai highlighted - these things. And those are the things that we have to prompt the AIS with because if you don't, they'll just write crap that won't work. So, so you know, we've got to specify things, you know, like this. You got to think you you need to have the kind of mind where you're going to think about those. You've got to understand the problem that you're trying to solve really well. And you've got to be able to work through the problem and learn bit by bit, step by step by step, you know, what the surface area of the problem is that you're dealing with and communicate that to the ([View Highlight](https://read.readwise.io/read/01kv3sxaz0m29n3xxsz5z9fsvr)) - Tags: 👻 ai highlighted