Is AI Killing Your Software Quality Standards?  🔗 **Source:** [Is AI Killing Your Software Quality Standards?](https://www.youtube.com/watch?v=yTL4IKTjcuI) > [!note]+ Document Notes > --- > tags: [ai-code-quality, engineering-standards, code-review-automation, software-development-ai, technical-governance, ai-assisted-development] > author: [MISSING] > source: Deep Dev > reference: https://www.youtube.com/watch?v=yTL4IKTjcuI > date created: {{date:YYYY-MM-DD}} {{time:HH:mm}} > --- > # 🏷️ Title: Is AI killing your software quality standards? > ## 🧭 Table of Contents > - [📝 Summary](#-summary) > - [📖 Main Content](#-main-content) > - [🗒️ Notes](#️-notes) > - [📚 References](#-references) > - [🗣️ Paraphrase](#️-paraphrase) > - [📰 Newsletter](#-newsletter) > - [✅ Action Items](#-action-items) > - [🧪 Hypotheses](#-hypotheses) > - [📊 Methodology](#-methodology) > - [❓ Questions](#-questions) > - [🔮 Future Research](#-future-research) > - [💡 Implications](#-implications) > - [🧠 Learning Styles Assessment](#-learning-styles-assessment) > - [📊 Accessibility Assessment](#-accessibility-assessment) > - [🌍 DEIB Assessment](#-deib-assessment) > - [✅ Logical Validity](#-logical-validity) > - [🛡️ DARVO Analysis](#-darvo-analysis) > - [🎯 Conclusion](#-conclusion) > ## 📝 Summary > - **What happened**: A panel discussion explored how AI code generation is transforming software quality standards and code review processes across engineering organizations. > - **Why it matters**: Traditional human-only code review cannot scale to match AI's code generation volume, requiring fundamental shifts in testing, standards enforcement, and engineering culture. > - **Key stakeholders**: Engineering teams, CTOs, product directors, and organizations using AI code generation tools like Copilot, with particular focus on safety-critical systems. > - **Risks and tradeoffs**: Risk of losing code quality control versus gaining development velocity; maintaining human accountability while leveraging AI automation; potential skill atrophy as humans review less code. > - **Bottom line**: Standards still matter but must evolve - focus shifts from documenting "what" to "why," from human-scale review to AI-assisted governance, and from static rules to dynamic, enforceable guardrails. > ## 📖 Main Content > The fundamental challenge facing engineering organizations is that AI can now generate code faster than humans can meaningfully review it. This creates a scalability crisis where traditional quality gates become bottlenecks rather than safeguards. > **Standards Evolution**: Engineering standards must shift from human-centric to AI-partnership models. Clean, readable, maintainable code remains crucial because: > 1. Humans still debug production issues at critical moments > 2. Standards provide the control interface between humans and AI agents > 3. AI agents perform better on well-structured, consistent codebases > **Documentation Philosophy Change**: The focus must shift from documenting "what the code does" (which AI can infer) to documenting "why decisions were made" - business context, architectural tradeoffs, and intent that AI cannot derive independently. > **Governance Through Skills and Rules**: Organizations are implementing "skills" - packaged bundles of rules with context about when and how to apply them, rather than flat rule lists. This enables different standards for different code areas (payment processing vs. internal tooling). > **Risk-Based Review Strategy**: Teams are adopting confidence-based review processes where AI-generated code receives different scrutiny levels based on risk assessment and AI confidence scores. High-confidence changes in low-risk areas may ship with minimal human review. > ## 🗒️ Notes > #### ⏱️ [00:06:42](https://www.youtube.com/watch?v=yTL4IKTjcuI&t=402) Vernon on scaling challenges > - Traditional testing didn't scale even without AI - now "rocket fuel" added to code output > - Identity shift required: testers must become enablers providing tools rather than gatekeepers > - Mixed approach needed: deterministic code uses traditional methods, AI-generated code needs new approaches > #### ⏱️ [00:12:15](https://www.youtube.com/watch?v=yTL4IKTjcuI&t=735) Almog on enduring value of standards > - Standards more important than before because humans still debug critical production issues > - Clean code serves as interface between humans and AI - lowering standards means losing control > - AI agents work better on clean, consistent codebases - "garbage in, garbage out" > #### ⏱️ [00:18:30](https://www.youtube.com/watch?v=yTL4IKTjcuI&t=1110) Dominic on architectural documentation > - Moving toward documenting architectural decisions explicitly for AI context > - Using code review agents that understand company-specific API standards > - Risk-based approach: different review rigor for different code areas > #### ⏱️ [00:22:45](https://www.youtube.com/watch?v=yTL4IKTjcuI&t=1365) AI review vs human review discussion > - AI reviews handle volume and context that humans cannot > - Need different tools for generation vs review to avoid bias > - Human-in-the-loop remains essential for accountability and safety > #### ⏱️ [00:28:20](https://www.youtube.com/watch?v=yTL4IKTjcuI&t=1700) Dominic on governance through AI > - Using AI to enforce communication standards in project updates > - Skills that prompt engineers to think through problems rather than just report status > - Codifying coaching practices that previously required manual intervention > #### ⏱️ [00:34:10](https://www.youtube.com/watch?v=yTL4IKTjcuI&t=2050) Almog on governance implementation patterns > - Converting tribal knowledge into enforceable rules > - Skills as "next generation rules" with context and enforcement methods > - Connecting same standards across generation and review phases > - Analytics on rule performance to create feedback loops > #### ⏱️ [00:45:15](https://www.youtube.com/watch?v=yTL4IKTjcuI&t=2715) Vernon on Bainbridge's paradox > - Automation leaves only difficult edge cases for humans > - Humans get worse at spotting rare problems due to infrequency > - AI's "jagged frontier" - excellent at complex tasks, fails at simple ones > #### ⏱️ [00:48:30](https://www.youtube.com/watch?v=yTL4IKTjcuI&t=2910) Dominic on shifting quality left > - AI automates data gathering for debugging, reducing manual investigation time > - Engineers focus on judgment rather than log collection and correlation > - Confidence scoring allows automated triage and assignment > ## 📚 References > - [Kodo AI Code Review Platform](https://kodo.ai) > - [Inspiring Senior Living Platform](https://inspiring.com) > - [Phrase Localization Platform](https://phrase.com) > - Lisanne Bainbridge's 1983 paper on automation paradox > - Ethan Mollick's "jagged frontier" concept > ## 🗣️ Paraphrase > Engineering organizations face a fundamental scaling challenge as AI generates code faster than humans can review it. While traditional code review processes become bottlenecks, the solution isn't to abandon standards but to evolve them for AI partnership. > Standards remain crucial because humans still handle critical debugging, clean code provides the control interface with AI, and AI performs better on consistent codebases. However, the focus must shift from documenting what code does to why decisions were made, capturing business context and architectural tradeoffs that AI cannot infer. > Organizations are implementing "skills" - contextual rule bundles that specify when and how standards apply to different code areas. Risk-based review processes use AI confidence scores to determine review rigor, with high-confidence changes in low-risk areas potentially shipping with minimal human oversight. > The human role evolves from comprehensive reviewer to governance designer and edge-case handler, though Bainbridge's automation paradox warns that humans may become less skilled at spotting the rare critical issues that remain their responsibility. > ## 📰 Newsletter > 🤖 **Is AI Killing Code Quality? Not If You Do It Right** > The latest Deep Dev panel tackled every engineering leader's burning question: how do we maintain quality when AI generates code faster than we can blink? The answer isn't what you'd expect. > **The Plot Twist**: Standards matter *more* now, not less. But they need to evolve from human-scale to AI-partnership scale. Think "skills" instead of rules - contextual bundles that know when and where to apply standards. > **What's Working**: Teams Hello everyone. So let's let's wait for a few a few seconds like 10 or 20 seconds to wait for people to to log into our webinar today. And yeah, welcome, welcome. And yeah, let's just meet briefly our our panelist today. I'm curious. So, we've got Dominic, Almog, Vernon with us today, and I'm wondering where are you where are you calling us today from? If you if you don't mind starting starting with Almog. >> Hi everyone. Um I'm from Ko uh based in Tel Aiv. Nice to meet you all. >> Thank you. And and Dominic, where are you calling from? >> Hey, I'm based in Victoria, BC, Canada. >> Oh, fantastic. Beautiful. And what about you, Vernon? >> Yeah, my my background is shameless, false advertising because I'm in the UK. There are no rainbows present. There's no sunshine. Um, I'm in a city called Leicester, one of those annoying places that is not pronounced anywhere close to how it's spelled. You arrive, Arnon. So, so I think we've got enough people uh together with us. Uh, so we're gonna we're gonna get started. Hello everyone and a huge thank you for joining us today for a discussion on is AI killing your technical standards. This is a deep dev online panel discussion created in partnership with Kodo. This discussion is going to last roughly 45 minutes after which both myself and the panelists will head over to the latest Slack to answer some of your questions in the panel Q&A channel. We'll drop a link in the chat uh with the name. We may also have time to answer a couple of questions live. So, please do submit those in the Q&A feature on Zoom. and we'll keep an we'll get to them if we can towards the end. I also do check throughout the webinar, so just don't hold back. So, we're going to get started with some more formal introductions. My name is Blanca Garcia Gil, and I'll be joined today by Almog, Dominic, and Vernon to share their experiences and insights. I'm going to ask each of you to introduce yourselves, and we're going to start with Almog. >> Thank you. Um so hi my name is Elmog. I'm a director of product at COTo. Uh before product I was a engineer for 10 years also engineering manager. So I've been both sides. Um COT is an AI code review platform. We plug into your p request workflow and review your code automatically looking for quality security best practices and whether the code follows your team standards. This is what we do and we work with some of the largest engineering organization out there. H and personally for the last six months my main focus was building our rules standard system. Uh so this is why I'm very excited to be on this panel because it's a question I hear from customers all the time. Uh how do you keep your standards alive when agents are writing most of the code today? It's one of the most important questions in engineering right now. So thanks for having me. Thank you, Alog. And next we'll move on to Dominique. >> Hi everyone. My name is Dominique. I'm the CTO at Inspiring. Um we build a full stack connected care platform for senior living. So we help residents that live in senior living communities with hardware, AI models on the edge, software. We have a huge data platform. So very very complex ecosystem with a tiny team of maybe about 40 people. Um but our technology is responsible for preventing falls and keeping residents safe. So our technical standards aren't optional. We have to build a very high quality system because when we have when we make mistakes, people can get hurt. So very top of mind as we've rolled out very significant AI uh usage at our team. >> Oh, thank you. Thank you, Dominic. Uh and uh last but not least, Vernon. >> Hey, greetings everyone. So I'm Vernon. Um I'm a senior engineering manager at a company called Freys and we're a localization platform. So we help teams and organi or organizations take all the text that they're dealing with and localize it appropriately for the markets that they operate in. And then outside of that like every man and their dog. I have a podcast with a friend of mine called Richard. I write a newsletter on Substack. All very boring and uh predictable. Uh and I'm also a family man. So got a wife, three kids. They're all awesome. So yeah, you may hear my daughter in the background at some point ping around the house. So yeah, just warn you about that ahead of time. >> Amazing. Thank you. Thank you all. So today we're going to be looking at the intersection of software quality and AI. The enormous volume of AI generated code is transforming the challenge of pragmatic technical standards as traditional code review is no longer scalable enough to keep up the pace. We will explore how our experts are rethinking culture and processes to shape code reviews and engineering norms ensuring software quality remains high. We're going to be structuring our discussion today into three overarching topics. The first one being what are the implications of AI generated code on your orgs engineering standards. And we're going to be starting with with Vernon. So what specific and measurable changes are necessary for organizations testing standards and tests with integrity when AI agents begin producing code at a massive scale? >> I have no bloody idea. Like this is this is this is why I was super duper excited to be on the panel because we are we are literally trying to figure this out. And so, um, a slightly more serious answer is one important idea that that I have as we figure this out is actually this isn't the answer isn't new because having all of your testing go through the human call, the tester, forget AI, that didn't scale anyway. And now we've added rocket fuel to the process of outputting code. It's like good luck if you don't have any automated tests. Good luck if you don't have any pipelines or you know all that cool stuff that we've been trying to convince organizations in the industry to do for like the past 20 years. That's now table stakes. If you're not doing any of that, good luck. So that's one thing. Um I'd say I'd say another thing is that if like my organization if you have testers in your team and they've been operating in the in the you know the typical way there's quite a there's quite a significant identity shift that's going to have to happen at least with with that group of people but also with the rider engineering or because I think the the folks who are the testing specialists in your organization, I think they're going to have to instead of enabling um instead of instead of delivering information to their teams, they're going to have to in a in a kind of team topologies kind of style, they're going to have to switch into enabling teams and give them tools to help them do the testing. And so that's quite a mindset shift. So instead of us providing information, we're going to be providing tools or assistance or coaching or all of this kind of thing to empower the team because as awesome as I am, if I'm the only tester in your team, I I don't scale. So the only way to do that is to empower everybody else to do the stuff that we do, including the robots, because now we've got robot teammates as well as um organic ones. So, it's a it's a there's a lot of identity shift stuff to do and then there's all the table stakes stuff to do. And the and and the final thing I'll say is um it will probably take a mixed approach as well because the if you look at the stack of the stuff that we're building, it's not 100% AI enabled products and services. So those things that are still deterministic in nature, we can still use the it seems silly to call it old. It only got old like two months ago, but anyway, the old the old way of of checking those things were right. They still apply. So all the pipeline still forward, exploratory testing, all you know, unit test that all works. But then for those part of your stack that is more non-deterministic because it's got, you know, it's gen AI enabled, then we're going to have to use a different approach because that stuff won't work because every time we do something different, we get a different output. We have to reason about it differently. So that was a lot of words. Um, I will stop. >> Oh, that was that was great. And I love how you how you set the tone that we're all just figuring it out. >> We don't know what the hell we're doing. It's it's it's okay. It's okay to start this session just acknowledging that >> we're figuring it out. And I and I hear it's you know you said identity. It's about identity. It's also about providing tools as well as empowerment coaching >> and it's also about understanding what types of different types of testing. Genai power features are tested slightly differently. >> Yep. >> Yeah. Did I did I capture that? >> You did. Perfect. >> Thank you. Thank you. I'd like to you know share from Analog now and please Analog tell us what is what is the enduring value of engineering standards like like clean readable maintainable code when when like Vernon said AI agents are increasingly just authoring code and the consumer of that code is might this be another AI agent? Why do we care? >> Yeah. um qu this question is something that we think about a lot um both in Kodo inside Kod our developers and also I see this conversation with our customers all the time and I think the honest answer has two parts first of all yes standards still matter I mean that's I don't think it goes without saying maybe even more than before but second um and this is where it gets interesting the standards themselves need to change. Sticking to the exact same rule book that we have five years ago, it's just, you know, it's not relevant anymore. Um, so, uh, I'll start by explaining why I think standards still matter. Um, so the most obvious reason is that humans are still in the loop. Um especially in the moments that uh when you something's breaks breaks in production and when there's a security incident when you need a big uh change in architecture or when you're reviewing someone else's code review. Um these are high stakes moments and that's exactly when readability matters. So you don't want an engineer um to debug something at 2 am trying to make sense of uh any messy code that just because an agent wrote it, right? So um so that's that's one thing. Um but the second reason is about control standards is how you keep track of what agents are doing. If you let agents write messy, inconsistent code, you lose the ability to check what they did. Uh clean code is yeah it's clear names, small functions, consistent pattern. It's not just about looking nice. This is the interface between humans and AI. And when you lower those standards, you're giving up control and it's it's very dangerous and even irresponsible when I'm I'm the one that should be owning the ship to ship this product. Um, and third, and this is something that we see very clearly at Kodo, agents actually work better on clean code. A models give better reviews, better suggestions, and make fewer mistakes when when the codebase follow the consistent patterns. And it makes sense, right? Garbage garbage in, garbage out. We all know that. If your codebase is a mess, the agents working on, they produce messier outputs. it makes the codebase even worse and it makes everything messier and that's a really really bad cycle. So standards break that that cycle. Uh so yes standards matter but um some of our standards were built in a world where humans were the main bottleneck and that world is changing now. um core idea uh I uh clarity, maintainability, consistency those they don't change but the specific rules some of them we need to rethink for example developers have always valued short clever code right uh that made sense because one can only only hold so much in our heads at once but agents they don't have these limitations code that's more explicit maybe even what developer would call overdocumented. That might be even better for agents because it gives agents more context and give humans clear picture uh when they step in. And documentation in general it's it's it's another one because we've always documented what the code does. What the function gets this input returns this output. But agents they can figure it out. They they don't need us to document this. what they struggle struggle with is understanding why why was this approach chosen right what what trade-offs were made what's the business context I think we need to shift towards documenting the intent and the context and not just the description of what the code does document the why not the what so to wrap it up the question isn't whether the standards should exist yeah they should exist when AI agents write most of the code standard becomes the main way that we keep quality and control but we also need to be honest about which standards will still still serve us. Yeah. Right. After the whole agents generation uh when the world is not human only world anymore. So yeah the ideas stay but the specific rules need to evolve. >> Thank you. Thank you Am for you know speaking so so clearly and and I think I you know another thing that stood out is that you say well the standards matter but it's this control but it's also a communication interface >> yes >> and it's also enabling agents as well as people and and I I almost yeah I almost laugh when you said oh we have to document the intent I thought weren't we meant to be doing that already you know when you when you write like clean commit log you know that sort thing architect. >> It was very hard to do it. >> Now we have to. Now we have to. There's no other way. >> Yeah. Wow. >> I think you've changed my mind. Like I've been really thinking more about architectural standards and I'm like I don't care what the code looks like as much. But I think you've changed my mind that we need to actually start think documenting more of of the why. What I've been thinking about in terms of standards is more like our architectural standards because for example like we historically maybe have used cues or things that scale for the scale that we have today. But in inherently AI is not going to know when we need to start scaling how many how many different facilities we're going to go install at tomorrow and therefore certain things and patterns that we have in our code today may not work well. This is where like engineers thinking about our scale or providing context in the why I could see how that could work is really really meaningful. So some of what we've done is used and documented well we're moving towards this standard. So now as we keep writing new code documenting in let's say our MD files let's make sure that we're moving to tailwind CSA. So that's not so much a scaling problem but that is a different thing that we're also thinking about. And I've found that to be a really powerful way to drive our architectural standards and actually make sure that we're documenting them versus in the past we wouldn't have even bothered nec or I would have worked with with you know staff engineers said make sure you document this make sure you tell everybody okay go do some code reviews now all of a sudden we've documented that this is what we're going to do and we've even created agents where we have a code review agent that knows what our technical standard is for our APIs that's specific to inspiring so specifically how do we work with our own data and how do we how have we decided how we want our APIs to work internally and then the code review agent will review and then enforce applicable uh making the standard work and so I found that to be like an interesting way to think about how you create scalable standards that matter but from an architectural lens. >> Yeah. >> Yeah. I I love how it's about the things, you know, again the architecture decisions that we were thinking maybe sometimes we don't need to document it. In my experience, they saved me more than once at 2 am in the morning when I was, you know, the fear on call. But now it's like it's important because of, you know, it's almost like we're putting more explicitly what we had in our minds. There was almost like this knowledge being shared above the team's heads and now it has to be explicit so the agents can actually draw on it. Wow. I'm going to pause for a moment because there is one question in the Q&A that I think is rightly aligned with this discussion and I'm just going to bring it in and the question is for anyone who would like to answer from our panelists and they say uh Dan is asking do you see AI reviews as a complete and visible replacement to human reviews or are tools like Codo designed to take the initial heavy lifting out of the equation to reduce the level of scrutiny a human may need to employ? I can answer but maybe someone else wants to answer. >> You want to start Elmo? And maybe you know if Dominic and Barry want to chip in afterwards. >> Okay. Yeah. Because I hear this question all the time. It's um code review becomes the new bottleneck, right? I mean the writing the code was 80% of the time and then 20% of the time we review the code and now it's like the exact opposite. Now we have too many pull requests. So as Vernon said it's not scalable that human will review all the code. Um I think the most important thing is that um to remember that you can't be your own criticizer. So I would not choose the same tool that wrote the code for me to also review the code for me. Right? It's a you need to be uh unbiased in order to review someone else's code. So uh COTo is here and COD is here basically to help with the review process. Uh but still we really believe in human in the loop really believe that eventually human should be the owners of the code. Uh but tools like CO can reduce a lot of uh the the burden of having a poor long pull requests. That's one thing and another thing is because we know AI agents have such a large context today and they can uh roll into other areas uh in in the codebase that one person cannot hold all this information but this is where AI even do a better review than a human being but still it can always make mistakes. So I believe it's it it should be a um joint uh uh experience AI and human but AI takes a lot of the time and the burden the review. >> Thank you. Thank you. It sounds like AI is partnering with the with the human to it. Veron. >> Yeah. So, it's back to what I So, this this question is one of the main reasons why I said we don't know what the bloody hell we're doing because everything Elmog said is absolutely nailed on correct. There's so much volume of code, so much surface area to cover that individual humans probably can't hold all of that in the Well, we can't do it now. So the volume of code that we've got now we definitely can't do it but at the same time in I'm so glad Dominance on is call because it's it's the perfect context. It's not going to be sufficient and you know it sounds funny but this is real serious. It's not going to be acceptable if our AI generated code hurt somebody in the real world. If we go I have no idea. We going to have to go and talk to anthropic and cl about that. It's like that's not acceptable. That's literally not acceptable. You can't make the the the AI agents and all these tools responsible and accountable. So the question becomes what do our governance policies and procedures look like so that in the event of you know those hideous circumstances occurring, how are we keeping ourselves accountable? about what how do we know that the things we're putting into production are are good and particularly in in you know Dominic's situation like >> because it will hurt somebody. >> Yeah. And so we really look at it from a perspective of overall where is the risk where can you go further and push further versus not. So in our main code base which uh can be less automated testing because it's connected to some of the hardware that we have in place etc. That's a place where we expect very clear code reviews every everything gets looked at right now. But what we're starting to experiment with is we have a coding agent that we've coded in such a way that it actually when it puts up a PR automatically with us not getting involved it's starting to put confidence levels on those. So high, medium and low. Today we look at everything. The goal that I'm working towards is that anything that is high in a lowrisk area. So we build internal tooling for our own employees that gets used on a regular basis whenever we go do installations and we're always available to help. We have a process available to that for things like that. The risk is so much more controlled and it's a great place to start to get to a point where we can start to accelerate and make code review not be part of the process. And let's be honest, like that's something that in the past we didn't have great tooling for those teams because we couldn't make the time. But we went from a place where it would have taken three months to build a perfect app for our implementation team to now it took us like eight days to have an app that has 10 different features in it. So all of a sudden we've been able to do that and that's okay that in those places if it's a high confidence change we would ship it automatically. We're not doing it yet, but that's exactly like what I'm designing is if you build my view is you design a system that you can provide governance in. And then as the models get better, as you increase your confidence in the skills, the sub agents, the I love what you said, almag of like you need to have a tool that's different. We actually have two different code review tools that we use. One within the developer workflow and then one uh after we've after we've shipped uh that does it at the PR level. That's what's going to at one point there'll be a moment where we're like, "Yep, this is now just a checkbox most of the time." And so a certain percentage of our PRs in the lowrisk repos are going to be automated and not require that level of code review. Um, thank you. Thank you both Vernon and Dominique for sharing first your questions around around risk. Uh, but then also Dominique, you've already been thinking, you know, how do we make this happen? How do we mitigate this risk? Uh but I think on top of your mention of of governance very I think what I'm hearing is ownership still the teams own the code because otherwise you can't really manage the risk. Who owns the risk? An agent? Not really. Oh thank you. Thank you. I think we're going to move to the to the second topic uh for today uh which ties into how to govern engineering norms across distributed teams using AI. I'm going to go back to you uh Dominic uh because you said you you found possibilities leveraging AI to codify best practices for non-technical aspects. How are you using AI to enforce consistent high quality engineering practices across your teams? You already hinted a bit, but tell us a bit more. Yeah, I get so excited about this because I used to like coach individual staff engineers, managers on best practices and actually I think it ties really well to ownership because I want people to really feel the full ownership of the technical quality but also the product that we're shipping and there's a couple of things that I launched just in the last four weeks that have right away went extremely high in terms of adoption. Um the first is a peer review tool because we were doing our feedback and we've still been doing feedback manually and so helping engineers to actually remember what did I collaborate on with this other person and how can I give high quality feedback based on my interactions versus just the thing that I remember and with skills you can actually get it to not to prompt back and say well you said this do you really mean this and I've used this really well uh for our project updates so engineers own outcomes they don't own a project. They own what they've delivered. And so we expect weekly project updates. And in the past, we had a wiki page and we did a presentation on great communication. Guess what? Engineers didn't communicate very clearly. And we still our projects still slipped. And so now when you post a project update, two things that I used to constantly be checking, especially for our most important critical projects for our success, were are they on track? If they were putting yellow, I was like, but do you have a plan? What's your plan? How can I help you? Now, the skill prompts you to say, well, what is your plan? And they end up having to decide, oh, I'm actually I got help for this one thing that I was stuck on or I ended up doing this. And so, the amount of times that I need to then proactively step in or do coaching is like reduced from, you know, many many times to now it's just like codified as part of the skill. And so I think it's just like such a powerful way to be able to create best practices and coach people in a scalable way which has never been possible before. >> Yeah, I I love that. And it's almost I think what I coach is like prompting people to think through. Don't just stop at at the yellow, you know, status project update yellow then like think why. >> Yeah. Yeah. And there's I I force people to have a TLDDR at the top because often people are like I'm doing this. I'm like, great. Does that mean that we're going to achieve the goal of the project by the time that the project ends? And it even knows when are we scale up when when do um when does this change? Are you actually going to hit like the end of our cycle? And it does the delta to to really push them. And so the end of our cycle is near. I'm really excited to see what happens as people start to put green when there's only four days left. >> Yeah, definitely. And I I love how you said that, you know, sometimes communication is is lacking and how different prompts, different ways you're leveraging for people to communicate. Just just take take the path. Thank you. Uh analog, we're back to you and you talk we talked about standards you and and gut rails. So what what do you what do standards gut rails? Uh you mentioned skills, rules, role files, repos. What are organizations implementing to effectively govern the output and behavior of AI agents? Yeah. So um as I said, we get a really close look at this because this is exactly what our customers are currently doing. They're building their garbage these days. Every day there's a new thing that you need to keep up. And as Dominic said, skills is the new kid on the block. It's not new anymore, but this is what everyone's doing these days. And um well, it's it's it's a mess, but I'll try to organize it a little bit to explain. Um so the first thing that we the first pattern that we see is uh like turning um turning our tribal knowledge into rules basically because rules exist somewhere right in your codebase or in your PR history or in your confluence or you you write rules you don't even know that you have rules sometimes because um for example um my tech lead wrote me a comment two times the same comments. This is probably a rule that we need to just enforce and not let the tech lead remember it every time. So, this is one thing that um that exists already and it's really good to just uh um collect and uh discover all these different rules and into one system that you can uh enforce these rules with. So, this is one thing. Um the second thing is uh I think skills as Dominic says skills is like the next generation of rules. And why is that? Because it's more than just a static list of rules. Um kill is a package that bundles rules together with the context about when you should apply them and also how you should enforce them. Uh think of it this way. You might have a set of rules that only relevant for when you're working on your kernel code or your payment layer, authentication layer. These are different rules. So you want the skills to say uh these are the rules. This is where you need to apply them and here's how to check them. Maybe you can run a script or use a tool or call call an external API. Sometimes it's not always easy to uh enforce and validate if the rule rule is uh violated or not. And this is exactly what skills um give us. This is a huge shift from just having the flat static list of rules. Um it's the ability to say when you're in this part of the codebase here is not just rules but the tools and the processes that you need to follow. Um so this is another thing that uh we see very very often lately. Um another thing is connecting the same rules in both ways in both sides in the generation part and in the review part. We know the generation tools right uh sorry generation rules uh you have cloud MD agents MD copilot instructions all these different tools using different uh um formats of rules and it basically means it tells the agent in this codebase here's how you do things and then you hope that it will follow these rules in so in generation times rules alone there aren't enough in my opinion because agents might follow them or might not and it might read them differently that one what you meant. So you also need to an enforcement layer at the review time that automated checked against the same standards that you have in the uh generation part time. Did the agent follow your pattern or add a dependency that you don't allow it to do it? Uh this is um this is where where it it really matters. The best teams connect both sides. The same rules get enforced during the review and also get fed into uh the coding agent. And lastly, uh and I think this is very important, um measuring how our rules perform. You don't even know, right? You have a list of rules, but you don't even know if the agent follows them or what happened to them. Uh uh so having analytics on how how many rules were violated or passed or merged to production with the violation. This is extremely important and it can give you a lot of data that you can work with. For example, um if I see that the rule is never been violated, maybe the rule is not good enough and we don't need it. Or if we see a rule that always gets violated, maybe we need to understand something and change something in our way of working. So this creates a feedback loop. It's not only about having control of the data. It's also about helping improving our rule system and uh uh have really up-to-date rules all the time and things that we care about uh not just the best practices of the world uh let's call it. Um yeah. Um yeah, I think you know I'm what I'm hearing is almost like you know we're winning we're moving forward because we have more fine grains of applying rules and also different stages. We almost like we have you describing it we have like a central repo and these rules can be enforced in different stages of the software development life cycle which before we weren't really maybe able to in the same way and when you're talking about about uh statistics and data and the feedback I was thinking of of veron what you said earlier about you know testing specialists also you know being providing tools providing coaching you know they can they can do a lot you can see them doing a lot and bringing a lot of you using using a lot of those like creating the rules but also then you know shifting the culture because it's a culture change as well. Okay, I think we have you know I'm going to pull in one more question because it's related what we're just speaking. Uh Andy asked should we should we be using AI tools to analyze code or should we have more deterministic tools and then use the AI to analyze and summarize the outputs? I feel that's a whole take almost would like to take it. >> I don't I think it's the opposite. Like I think really you want AI to proactively do some of the heavy lifting and then uh and then you want to have guardrails in place to make sure that it actually did what you wanted it to do. Like the way that I used to think about this more was I think Alma did such a great job talking about like traditional platform engineering is I used to have like the guardrails at the end and then make sure for example that you were always increasing test coverage like I care deeply about automatic test coverage but now what we've done instead is that we've got for example test driven design sub agent so when you're working on your PR and it's a new feature actually if it's a new bug it will automatically launch create a failing test for that bug and then it'll write the code that fixes it. But that doesn't replace the fact that then at CI time I want to enforce that we've actually written the right amount of test for coverage, which may actually be more than just that one initial test that made sure that the bug works was fixing. So I think about it differently. I'm not sure if I'm really answering the question, but for me it's about how do you do both? >> Interesting. Uh thank you. Thank you, Dominic. Uh, anyone else would like to chip in or if not we'll move on to the next question. Yeah, >> I I I would say if I'm understanding the question, I don't think it would work that way around because I don't I don't think the deterministic tools would find the problems that you would need it to find. Um, and this is on the assumption that we're building that we're building AI. So if we're building AI because of the nature of the technology if the output if the output so in a deterministic system if the question was are you okay and the output was yes we like yes in French um 10 or teapot only the first answer would resolve to yes you've passed but then in the nondeterministic situation the first two would be Yes. And you would want the system to pick that up. So I don't know if I'm understanding the correct the question properly. I don't think it would work that way around. >> Like we just would it wouldn't it we wouldn't be able to do it that way. >> Okay. No. Thank you. Thank you for well Andy. If there's anything else that maybe we haven't touched on, uh feel free to ask in the in the Q&A in Slack. Okay. So we can clarify there uh and iterate the answers. But but thank you. Thank you so much for proposing that question. In the spirit of time, we're going to move on to the to the last section. We have one more question in the Q&A. I don't know if we'll get time. Uh but this uh we'll always take them to Slack to make sure they're answered. So our third third topic that we're going to touch on is how to scale code review and quality to keep pace with AI. I'm gonna go back to you uh Verdon um talking about what is the role what is the new role of the human code reviewer when faced with huge volumes of AI generated code and what responsibilities do you see that a human person might must retain versus what can be delegated to NAI tools. So I think so I I discovered this cool paper by some by a lady called Lisan Banebridge. I'll find a link and I'll share it. And it's the funny thing is it's old like it's it's from 83 and and what she's saying in the in the paper is the more that you automate something that is left is all the difficult stuff and that's the stuff that you need the humans to take care of. But because they see it less often, they get less good at spotting it. It's it's like it's like a a conundrum. It's like you need them to be better at looking for the weird stuff, but humans are not good at just sitting there and looking out for weird stuff and just concentrating for long moments of time. That's not what we're good at. That's that's not our thing. So, that's weird. And then I also I also discovered um this concept called I I suspect everyone else has probably heard about this but just in case you're like me and just found out about this recently. There's this thing by a chap called Ethan Mollik called the jagged frontier and it basically describes this weird feature of AI where it's really amazing at amazing things. So we can give it we can point at our repos and say explain this to me please and it will and it will be accurate and then we can ask it I'm I need to take my car to the car wash that's 10 minutes away should I walk or drive and it will fail and say you should walk you how can you get how can you be good at that and be good at the other thing like what like that doesn't make any sense so we're ask so in this reviewing situation we're asking you know people like us to be vigilant for things that are rare which decreases our ability to see those weird things and also is actually draining for humans to do that kind of work. So yet again I have no answer. I don't even know why I'm on this planet at this point. I've got no answers for anything other problems. So these are the these are these are the challenges that that I am trying to get my head around. Um, I'm not sure exactly what the answers are, but I know that they are the challenges that they're pretty significant. So, I am now going to hand over to the smarter people on the panel and let them answer the question. >> I I think it's a great answer, but it's almost like you're you're lifting us up to a more kind of almost like philosophical base, but also it's it's complex. >> This is not easy stuff. Like, we've known that if we, you know, automate too much, what's left is really hard. We also know that what you don't you don't use you lose. So that's also the other saying with AI if you're not using your skills to like write codes and the things you're going to be losing them. So keep an eye because there's there's a big there's big changes but it's not all overnight. >> Exactly. Yeah. >> Yeah. >> Exactly. >> Thank you. Thank you Veron and we'll we'll we'll move on to Dominic for our last question. Uh and then we'll we'll wrap up. Uh so Dominic if we are focusing on scaling quality you noted that AI agents can also increase reliability. How does using AI to shift quality left by embedding system resilience operating standards up front fundamentally change the scope and burden of the human review human code review process? >> Yeah. Uh Vernon said it so well that we we're terrible at doing boring things. And I used to have to teach engineers when you go deploy, go stare at the dashboard. Go see if an exception has happened or if you're on call, are there bugs that have come in? Are you actually looking at it? And then you spend a bunch of time just doing that first investigation, pulling up the right panel, making sure that you're looking through your tools for the right logs, pulling it all together. But with AI, we we can instead like I'm so excited about automating more workflows such that the engineers in the loop. They have to respond versus they have to proactively initiate, do a bunch of manual work to pull together all of the data and the information they need to debug and therefore like they can actually fix issues much much much more quickly. And so the thing that we're working on right now, we already have a bot that can automatically be assigned to any bug that gets opened or to debugging a Sentry issue. With Sentry, they make it really easy for us to be able to then actually autotriage specific types of bugs to our own agents that we've created that have all the contexts on our code, all of the sub agents that I just talked about to be able to fix them. And it also has a confidence level. So we then we all of a sudden we have our own internal confidence level and then centuries confidence levels. So you can totally instrument these workflows for your engineers where they're in the loop reviewing and therefore much more likely to do it really really really quickly versus in the past you would have maybe spent two three hours or at least spent a significant amount of time just collating the data that you need. >> That's absolutely brilliant Dominic. I I thought at some point you were going to take you know the bane of what my what my existence was with being on call but if I don't have to like search for logs and capture logs and like you know early hours I think that's that's moving forward and AI is great at pulling data and I think that's you know we're reiterating also that AI is you know is a partner you know is helping us do our jobs better focusing on you know what we can do which is judgment and analysis And it's just not about, you know, going into machines, you know, and figuring out what is what is what I need to look for. >> Yeah. Yeah. I get really excited about the power that it has to pull all that information together. If we make it so much faster such that when the engineer is participating in the process, they're doing it for the parts that matter the most. >> Absolutely. I think I think, you know, I honestly think that also alleviates a lot of the stress of being on call. >> Yeah. Veron, go ahead. One thing, one one thing I'm wondering about is you know that phenomenon in PRs where you just see looks good to me like look right like so I I wonder what is going to be the AI cuz what what Dominic describes is dream world but also I feel like as awesome as it could be there's a risk that again because we're humans and we have to review things all the time like yeah looks fine looks fine looks fine until it's not fine >> and now you're you know back in the the bad old days of now something is for real on fire and because you didn't pay enough attention it's going to be to see how that goes >> over the over the weeks and months and years ahead I think >> this is why the best engineering teams are already having like if you already had invested in closing the loop in the systems that keep you up that make it easy to just revert if you push something or not is like I think this is where the best engineering teams are shining using AI because they've already got the resiliency they need to be able to leverage and manage those issues >> 100%. Yeah. >> Yeah. >> Well, I I wanted to thank you know to all our panelists for sharing your insights today and being just completely honest. And now it's time to to wrap up. So I'd like to just go around the room and please share, you know, what is your top tip for keeping technical standards alive in this AI area. And we're going to go in reverse from where we started and we're going to start with you Vernon. I think it is to try and be intentional about what good looks like before you start. I'm not saying overthink and spend a million years figuring it out. Just have some kind of um objective hypothesis about what what good looks like before you start using this stuff and then you'll be able to iterate more easily. >> Absolutely. Thank you. Uh and next uh Dominique >> I mean be much more practical right away. Go pick one thing that you care deeply about that's that's challenging for your standards and either go implement a skill a sub agent that's responsible for it. So one thing that we've done for example is security. We have a sub agent for security that verifies every single PR to ensure that it's high quality. It wasn't that hard to implement and the more it gets used, the more it gets updated by the engineers who are also using it so that it catches more security issues up front before they ever ship. >> Fantastic. Thank you. And uh wrapping up will be along >> I say um don't just write your standards down. You need to make them enforceable. It's really really important. uh if it existed only in a document or in someone else's head they will never be a reality only a wish. So make them enforceable. >> Brilliant. Thank you. Thank you so much uh everyone for all you know all your experience and your insights during the panel. We're are going to be heading quickly to the lead dev slack in the panel Q&A channel and we'll copy there was one question left. We'll copy it there. uh and we'll keep the the discussion there. So, thank you for joining and we hope to see you next time. Bye., Is AI Killing Your Software Quality Standards?, YouTube, 01kv3smg42rsn58pxkyz0hg8dd.