Allow anyone to code with AI 🔗 **Source:** [Allow anyone to code with AI](https://www.youtube.com/watch?v=ZZJVRg15yHo) > [!note]+ Document Notes > tags: [ai-coding, developer-productivity, software-development, code-ownership, production-safety] > author: LeadDev > source: YouTube > reference: https://www.youtube.com/watch?v=ZZJVRg15yHo > date created: 2024-12-19 12:00 > > # 🏷️ Title: Allow anyone to code with AI > <!-- deep-read:sections start --> ## 🧰 Action Inbox - [H1] Implement collective accountability in code review processes — AI shifts individual to team responsibility #devops - [H2] Add security consideration prompts to AI coding workflows — dramatically improves vulnerability prevention #security - [H3] Design governance frameworks before accelerating AI adoption — prevents system overload #governance ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [LeadDev Platform](https://leaddev.com/) - [Harness DevOps Platform](https://harness.io/) ## 📝 Summary - **What happened**: LeadDev panel discussion on creating safe AI coding practices in production environments - **Why it matters**: AI coding adoption is accelerating but creating new bottlenecks and safety concerns in software delivery systems - **Key stakeholders**: Engineering leaders, developers, DevOps teams, and business stakeholders implementing AI tools - **Risks and tradeoffs**: Speed gains versus system stability, individual versus collective code ownership, cognitive safety versus productivity - **Bottom line**: AI coding requires systemic thinking and human oversight rather than simple tool replacement ## 📖 Main Content The panel explores the fundamental shift from commanding machines to collaborating with AI in software development. This represents a psychological shift from control-based to collaboration-based coding, requiring new governance frameworks and accountability models. **Key challenges identified:** - **Bottleneck shifting**: AI accelerates code generation but pushes pressure downstream to testing, review, and deployment - **Cognitive safety**: Developers struggle with understanding AI-generated code that works but lacks transparency - **Accountability gaps**: Traditional individual code ownership evolving toward collective team responsibility - **System overload**: Infrastructure and processes unprepared for increased code volume **Technical debt considerations**: Panel suggests AI might actually reduce technical debt by enabling faster iteration and testing, allowing developers to run lightweight experiments without accumulating long-term obligations. **Security vulnerabilities**: Research shows AI-generated code often ignores security considerations unless explicitly prompted, but simple prompt modifications dramatically improve security awareness. ## 🗒️ Notes #### ⏱️ [00:00](https://www.youtube.com/watch?v=ZZJVRg15yHo?t=0) — [H1] Introduction and Panel Setup - Key idea: Creating safe spaces for AI-assisted coding in production environments - Evidence: "45-minute discussion on how to create a safe space to code with AI" #### ⏱️ [02:30](https://www.youtube.com/watch?v=ZZJVRg15yHo?t=150) — [H2] Psychological Shift in Coding - Key idea: AI coding represents fundamental change from commanding to collaborating with machines - Evidence: "We have moved from commanding logic to co-creating intent with these machines" #### ⏱️ [05:45](https://www.youtube.com/watch?v=ZZJVRg15yHo?t=345) — [H2] Cognitive Safety Concerns - Key idea: Developers worry about understanding AI-generated code that works but lacks transparency - Evidence: "People still want to understand what's happening inside of the system" #### ⏱️ [08:20](https://www.youtube.com/watch?v=ZZJVRg15yHo?t=500) — [H1] Governance Framework Necessity - Key idea: AI acceleration requires robust governance to prevent system brittleness - Evidence: "If your product organization is not fit for purpose... then things start to fall down" #### ⏱️ [12:15](https://www.youtube.com/watch?v=ZZJVRg15yHo?t=735) — [H2] Systems Thinking Approach - Key idea: Software development is a system where AI shifts bottlenecks downstream - Evidence: "We are moving certain things faster, but now we're introducing more pressure into other parts" #### ⏱️ [15:40](https://www.youtube.com/watch?v=ZZJVRg15yHo?t=940) — [H3] 90% AI Pilot Failure Rate - Key idea: Most AI adoption pilots fail due to lack of systems thinking - Evidence: "90% of AI adoption pilots fail... because we're not understanding that this is a system" #### ⏱️ [18:30](https://www.youtube.com/watch?v=ZZJVRg15yHo?t=1110) — [H2] User Readiness for Delivery Speed - Key idea: Users want faster predictability, not just speed - Evidence: "People say they want speed but I think they don't. They want faster predictability" #### ⏱️ [22:10](https://www.youtube.com/watch?v=ZZJVRg15yHo?t=1330) — [H3] Feature Impact Statistics - Key idea: Only 20% of features drive positive business impact - Evidence: "Only around 20% of features that are deployed actually drive some sort of positive customer impact" #### ⏱️ [25:50](https://www.youtube.com/watch?v=ZZJVRg15yHo?t=1550) — [H2] Measuring Outcomes vs Effort - Key idea: Success requires measuring both numerator (outcomes) and denominator (effort) - Evidence: "Productivity... it's outcomes over effort" #### ⏱️ [28:40](https://www.youtube.com/watch?v=ZZJVRg15yHo?t=1720) — [H3] DORA Metrics Limitations - Key idea: Traditional DORA metrics inadequate for AI-assisted development assessment - Evidence: "Those metrics are fantastic... but they actually don't help you answer the question of is my team achieving their outcomes" #### ⏱️ [31:20](https://www.youtube.com/watch?v=ZZJVRg15yHo?t=1880) — [H2] Humans as Quality Control - Key idea: Reframe humans not as bottlenecks but as quality control and system conscience - Evidence: "Don't make them faster, make them smarter... human as quality control" #### ⏱️ [35:15](https://www.youtube.com/watch?v=ZZJVRg15yHo?t=2115) — [H1] Code Ownership Evolution - Key idea: AI disrupts traditional individual accountability toward collective team responsibility - Evidence: "AI is actually disrupting accountability in the code review process" #### ⏱️ [38:45](https://www.youtube.com/watch?v=ZZJVRg15yHo?t=2325) — [H2] Emotional vs Legal Ownership - Key idea: Code ownership follows emotional accountability before legal frameworks - Evidence: "Ownership is emotional first before it's legal" #### ⏱️ [42:10](https://www.youtube.com/watch?v=ZZJVRg15yHo?t=2530) — [H3] Security Prompt Enhancement - Key idea: Simple security prompts dramatically improve AI code security awareness - Evidence: "Ask it to consider security vulnerabilities and it's shocking how much it actually changes the output" ## 📚 References - [Quotient Developer Productivity Platform](https://quotient.com) - [Facets AI Cloud Orchestration](https://facets.cloud) - [Harness DevOps Platform](https://harness.io) - [GitHub Copilot Error Study by UpLevel](https://uplevel.work) - [Microsoft AI Development Research](https://microsoft.com/research) ## 🗣️ Paraphrase - AI coding shifts development from commanding machines to collaborating with them, creating psychological and technical challenges that require new governance approaches - Traditional individual code ownership evolves toward collective team accountability as AI-generated code introduces shared responsibility models - Success with AI coding requires systems thinking that addresses bottlenecks across the entire software delivery lifecycle, not just code generation speed ## 📰 Newsletter **AI Coding Safety: From Speed to Systems Thinking** The latest LeadDev panel reveals a critical insight for WordPress teams adopting AI coding tools: speed isn't everything. While AI can accelerate code generation, it often shifts bottlenecks downstream to testing, review, and deployment processes. **Key takeaway**: Only 20% of features drive positive business impact, making measurement and governance more important than pure velocity. For WordPress developers, this means focusing on systematic improvements across your entire development workflow. **Practical tip**: Add security consideration prompts to your AI coding workflows—research shows this simple change dramatically improves vulnerability prevention. **Community connection**: Consider implementing collective code review practices where AI-assisted code becomes a team responsibility rather than individual ownership. Resources: [Learn WordPress](https://learn.wordpress.org/), [Developer Docs](https://developer.wordpress.org/), [WP-CLI](https://wp-cli.org/) ## ✅ Action Items #### DevRel - Developer advocacy: Create AI coding safety workshops demonstrating governance frameworks - Education: Develop curricula on systems thinking for AI-assisted development - Developer experience: Document collective code review processes for AI-generated code - Developer marketing: Campaign messaging around "safe AI coding" rather than just speed - Community management: Moderate discussions on AI coding accountability and ownership #### WordPress - Core: Establish AI assistance guidelines for core contribution workflows - Core Performance: Profile AI-generated code impact on WordPress performance budgets - Core Program Team: Coordinate AI coding policies across working groups - Community: Create meetup content on responsible AI coding practices - Training: Develop learning paths for AI-assisted WordPress development - Documentation: Update contribution guidelines for AI-assisted code submissions - Security: Review AI-generated code scanning and vulnerability detection processes - CLI: Evaluate AI assistance integration for WP-CLI command development ## 🧪 Hypotheses - Teams using collective accountability models for AI-generated code will have higher code quality than individual ownership models - Organizations implementing governance frameworks before AI coding adoption will achieve better long-term outcomes than those prioritizing speed - AI coding productivity gains will plateau without systematic improvements to downstream processes like testing and deployment - Developer satisfaction will correlate more strongly with cognitive safety (understanding AI decisions) than with raw coding speed improvements ### 📊 Methodology - Study design: Compare code quality metrics between individual vs collective accountability teams over 6-month periods - Data sources: Pull request review times, bug rates, developer satisfaction surveys, system reliability metrics - Experiments: A/B test security prompt inclusion in AI coding workflows, measure vulnerability detection rates ### ❓ Questions - How do we measure the cognitive load of AI-assisted development on developer wellbeing and long-term productivity? - What governance frameworks scale effectively across different team sizes and technical contexts? - How should code review processes evolve to accommodate AI-human collaboration while maintaining quality standards? ### 🔮 Future Research - Long-term studies on developer skill evolution in AI-assisted environments - Cross-industry analysis of successful AI coding governance implementations - Investigation of AI coding impact on technical debt accumulation and resolution patterns ### 💡 Implications - **Practice**: Development teams need new accountability models and review processes for AI-generated code - **Policy**: Organizations require governance frameworks addressing AI coding before accelerating adoption - **Security**: Current security scanning may be inadequate for AI-generated code patterns and vulnerabilities - **Education**: Developer training programs must include systems thinking and AI collaboration skills - **Funding**: Budget allocation should prioritize governance and infrastructure alongside AI tooling investments ### 🧠 Learning Styles Assessment Reading level estimate: Flesch-Kincaid Grade Level 12 #### 👀 Visual learners - Present: Panel format with multiple perspectives, clear speaker identification - Improve: System diagrams showing bottleneck shifts, AI coding workflow visualizations #### 🔊 Aural learners - Present: Full 45-minute audio discussion available - Improve: Chapter markers for key topics, isolated audio clips for specific insights #### 🗣️ Verbal learners - Present: Rich analogies (IKEA effect, fabric manufacturing), defined technical terms - Improve: Glossary of AI coding terminology, more concrete examples #### 🧩 Kinesthetic learners - Present: Discussion of hands-on coding experiences and practical challenges - Improve: Step-by-step governance implementation guides, interactive checklists #### 🧠 Logical learners - Present: Statistical evidence (20% feature impact, 90% pilot failure rates) - Improve: Decision trees for AI adoption, causal diagrams of system interactions #### 🧑‍🤝‍🧑 Social learners - Present: Panel discussion format with multiple expert perspectives - Improve: Follow-up community discussion prompts, peer collaboration frameworks #### 🧘‍♀️ Solitary learners - Present: Comprehensive transcript for individual study - Improve: Self-assessment questions for AI readiness, reflection prompts #### 🌿 Naturalistic learners - Present: Systems thinking approach connecting AI coding to broader ecosystems - Improve: Sustainability considerations of AI coding practices, environmental impact discussions ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Transcript available with clear speaker identification - Improve: Structured headings for topic sections, descriptive labels for speaker transitions #### Keyboard navigation - Present: Video player supports keyboard controls - Improve: Timestamp links for direct navigation, skip-to-content options #### Contrast and readability - Present: High contrast text in transcript - Improve: Adjustable text size options, enhanced visual distinction between speakers #### Text resizing and reflow - Present: Transcript responsive to text scaling - Improve: Ensure 200% zoom maintains readability without horizontal scrolling #### Media and captions - Present: Full transcript provided - Improve: Synchronized captions for video player, downloadable transcript formats #### Forms and interactions - Present: Basic video controls accessible - Improve: Enhanced keyboard shortcuts documentation, accessible feedback mechanisms ### 🌍 DEIB Assessment #### Cultural breadth - Present: Diverse panel with different company backgrounds and perspectives - Improve: Global case studies of AI coding adoption, non-Western development practices #### Inclusive language - Present: Professional terminology with explanations - Improve: Plain language alternatives for technical jargon, avoid assumptions about team structures #### Neurodiversity support - Present: Structured discussion format with clear topic transitions - Improve: Content warnings for complex technical discussions, multiple information processing options #### Representation - Present: Mixed gender panel with varied professional backgrounds - Improve: Include perspectives from different career stages, company sizes, geographic regions #### Psychological safety - Present: Respectful discussion of concerns and challenges - Improve: Acknowledgment of job displacement fears, supportive framing of skill evolution ### ✅ Logical Validity #### Strong reasoning - Deductive strength: Strong logical flow from AI adoption challenges to systematic solutions - Inductive strength: Good use of statistics and research citations to support general claims - Causal reasoning: Clear cause-and-effect relationships between AI acceleration and downstream bottlenecks - Analogical strength: Effective analogies (IKEA effect, fabric manufacturing) support complex concepts #### Fallacies - Appeal to authority: Over-reliance on panelist expertise without sufficient data validation - Hasty generalization: 90% pilot failure statistic needs broader context and sample size clarification #### Tone of voice - Overall tone: Professional and collaborative, balancing optimism about AI potential with realistic concern about implementation challenges ### 🛡️ DARVO Analysis - Deny: No denial patterns observed - Attack: No attacks on critics or concerns raised - Reverse victim/offender: No victim/offender reversal detected - Emotional cues and rhetoric: Balanced discussion acknowledging legitimate concerns while promoting responsible adoption ## 🎯 Conclusion AI-assisted coding represents a fundamental shift from commanding machines to collaborating with them, requiring new governance frameworks and accountability models. Success depends on systems thinking that addresses <!-- deep-read:sections end --> ## 📺 Video  --- # 📄 Full Transcript *Auto-generated from YouTube captions · timestamps roughly every 30 seconds · [watch on YouTube](https://www.youtube.com/watch?v=ZZJVRg15yHo)* **[00:06]** Thank you for joining us today for a discussion on how to create a safe space to code with AI. This is a lead dev panel created in partnership with harness. The panel will last roughly 45 minutes after which both myself and the panelists will head over to the lead dev Slack channel to answer some of your questions in the panel Q&A channel. We may also have time to answer a couple of questions live. So, please do submit **[00:36]** those through the Q&A feature on Zoom and we'll get to them when we can. So, let's get started with some introductions. My name is Heidi Waterhouse and today I'll be joined by Alex from Harness Himmanshu and Lizzie. They're going to share their experiences and insights. I'll ask each of you to introduce themselves. So, let's start with Lizzie. **[01:09]** >> Sure. Um, thank you so much for having me. Um, so my name is Lizzie. I'm co-founder and CEO of Quotient. Uh Quotient is a developer productivity tool that identifies friction, recommends action, and measures impact as engineering processes evolve with AI. >> Excellent. Himanchu. >> Hey. Hi Heidi. Hi everyone. Thanks for having me. Uh currently I lead growth for facets. It's a AI cloud orchestration platform. Prior to this **[01:39]** I've worked with I worked extensively uh in dev tool space and also as of now I'm working on this I'm working on a book called diary of a CTO where in the next two years I'm talking to 200 plus CTOs across the world and publishing and documenting it h and yeah that's what I have on my plate and really happy to be here. >> What an ambitious project and Alex. >> So yeah my name is Alexander Bach. I work for harness which is a DevOps **[02:09]** platform that tries to do end-to-end delivery right from continuous delivery and integration to all the way to feature flagging experimentation and for harness I I work as a subject matter expert for feature flagging experimentation I've been a practitioner for the last 10 years building a lots of like large scale enterprise um experimentation programs at big banks and media companies and now advise our clients and prospects of like how to measure uh what they're actually building and how to measure the success also on top of that. **[02:41]** >> Excellent. Well, thank you all so much for making the time to be here today and thank you to our audience. We're looking forward to getting some interesting questions and discussions. So, let's start with framing. What if the vibes of vibe coding are bad? Coding has always seemed like the intersection of art and science, but now we've added AI assisted coding to the mix. And it seems like vibe coding is replacing both craft and precision with **[03:13]** intuition and handwaving. Guesswork and approximation are no way to run a business. So, how can business leaders embrace a new way of doing things, but in a way that's safe, sensible, and compliant? The idea of this webinar is to give people a place to talk about how to make AI coding safe and something that you would trust to put into production and how to make it feel safe because there's making something safe and then there's making **[03:44]** it feel safe and those aren't the same thing. And sometimes when we get them misaligned, we have more problems than not even adding the tool at all. So I have some questions and I think let's start with what do you worry about with AI coding and do you think it's the same things your team worries about? Uh let's go Hamachu. So there are two things I think **[04:17]** overall like largely if you look at what has happened is uh especially around AI and engineering I don't see it as a tooling revolution I see a lot of I see it as a perception shift I mean the way human the way we relate to creation and coding like for decades writing code meant writing code was an act of control you told machines what to do but now with AI for the first time we are entering a world of collaboration you know we have now the **[04:48]** machines can talk back you know and that's a deep psychological shift we have moved from commanding logic to co-create to co-creating intent with with these machines and this is raising some and I think there's a lot of confusion around to what extent should we integrate it to what extent would it be of help how do I use it But one of the things that people worry about is they're still not able to make sense of it. Like they don't know where **[05:19]** does it really fit in. Uh there there are multiple aspects to it. One of the things which I feel is that while we know on an anecdotal level we know where AI can come in and help but on an architectural level it's it's still it's it's still an exploration and all the playbooks around engineering that we have had they were written in the past. So there are no there are no books there are there's there's no literature which can prepare you for what's happening right now. So it's a lot of experimentation and yes people are definitely **[05:51]** people are worried about their safety. Uh two things there. One is earlier when pe when developers or when engineers used the word safe, it meant that your code is production safe. You know, it's not going to uh it's not going to leak data that and that was just that's the technical safety layer. But now there's a deeper kind which is another definition which is deeper and that's cognit cognitive safety. And people people still want to understand what's **[06:22]** happening inside of the system. So not knowing is what is worrying people. So you're pushing code at times which is generated by AI but you don't know where it came from what it is exactly saying but it works. So that's what that's one thing that you know I think worries people. >> Yeah. How about you Alex? I mean touched a lot of like really interesting points there like for me personally like AI and coding in itself is is it's a pro it's a product problem **[06:53]** like that starts with the developers and ends with a final product somewhere right and obviously there's multiple avenues of like how AI can like accelerate development make your life easier but also make your life potentially miserable right so I think in itself I think Like AI is a great tool like to accelerate production and productivity, but if the right if the governance framework and the controls are missing, then it does the potential **[07:23]** the opposite, right? Um, yes, you might be able to quick faster code, you might be able to like churn out lots of new features and everything, but if your product organization is not fit for purpose, if your infrastructure is not for purpose to actually deal with this huge amount of a of generated code that now you can buy code in in just a day, then things start to fall down and you maybe end up creating like very brittle system that just will explode at one point. So I think it becomes more and **[07:55]** more important that even smaller companies start to think about how can I derisk that? Yes, I can move faster but how do I make sure that I can move faster in a controlled and reliable way right and that starts with how do I make my my to Hamash's point make sure that my security scanners are working correctly and I'm actually scanning all the stuff that I'm generating really nicely in a in a controlled way. How do I make sure that I can deploy things really fast and reliably without falling **[08:26]** down all the time? And all the way to the end was like, how do I make sure that what I actually code actually solves a customer problem at the very end, right? Because just because you can move faster, build things faster, doesn't necessarily mean it's the right thing for the customer or to your point Heidi earlier, for the business itself, right? Mhm. >> So it's it it touches so many like different aspects and to M's point like we haven't figured everything out yet but I think there's a good trajectory of where the business is going realizing **[08:57]** that by coding is not everything. Let's just take a step back and figure out where does the human actually needs to be in the loop to really solve the problem >> and like what technologies do we need to almost combat AI with AI. >> [laughter] >> I mean, the security vulnerabilities introduced by AI agents are amazing. Uh, Lizzie, what do you think? What do you worry about and what do you think your team worries about? >> Yeah, I think we're in a unique position because probably what my team worries **[09:27]** about is like what's happening in the industry just based on the type of work that we do. Um, I think what's really interesting is like software development is not a single uh process. It's actually a system. everything from the way you develop software to the way you test software to the way you deploy software to the way you review software. And so it's been very interesting to see as you know we're in the first inning of AI uh usage I think in the software development life cycle. Um and we've seen places where it naturally fits in **[09:57]** really well. Things like writing and generating code. That's great. But what that actually does in a system is that it just shifts the bottleneck further downstream in the system. And I think that is why you're seeing so much data around how uh just like 90% of AI adoption pilots uh fail. And that's really because we're not understanding that this is a system. So we are moving certain things faster, but now we're introducing more pressure into other parts. And so I think what we think about and worry about quite a bit is how do we actually relieve pressure on the **[10:29]** whole system so that engineers can actually benefit from you know the increased speed gains in the ultimate outcomes of delivering value for their customers. I think like you know oftentimes the word productivity gets thrown around so much but if you go back to a first principles approach it's outcomes over effort. So can you get the outcomes that you want for your customers and are you doing it at a level of effort that's commensurate or sustainable for your engineering team. I think AI has the opportunity to basically accelerate on both sides, the numerator and the denominator. But I **[11:01]** think what I worry about is that if you don't take a systems approach to looking at it and you don't have a good sense of like what your system looks like right now, then what you're really doing is you're just shifting bottlenecks into a place that might have instability like um poor testing to Alex's point or um maybe like the review process is very difficult um and then you're not actually seeing those gains and you're maybe um placing the blame in the wrong areas as opposed to again evaluating the system more broadly. Alex, it looked like you had something you wanted to add in the middle there. **[11:32]** No, >> no, I just fully agreed agree with you just said is like if if you just try to throw more things towards a system that somewhere has a bottleneck somewhere, if it's deployments or testing, it's going to fail and everything needs to be scaling accordingly to support this amount of code that comes through it in a in a good way, right? >> Fully agree just there. >> Thank you. Uh so I just launched a book called progressive delivery and one of our key **[12:03]** points is that users are not ready to accept change as fast as we can deliver it to them and I think that's an interesting problem as we talk about speeding these things up like Lizzie was talking about. I was wondering if any of you had a comment on whether you think users are ready for a faster tempo of delivery. Uh, Alex, >> um, I mean there are changes happening left and right everywhere in any sort of **[12:35]** digital product, right? Some are minor, some are big. I think what most users tend to notice first is like front end changes, right? Um, specifically as an NV perspective, if you if you I don't know, you not log into your Netflix and you see just slightly different how they organize the panels, how you you can't find your show any anymore, that's a a change that users tend to notice instantly. But with like say a chatbot or a like a recommendation engine or **[13:05]** anything, those kind of like backend changes or things that not necessarily immediate viable or visible, they require a lot of like more data gathering to actually figuring out does it actually affect user behavior in the long run, right? Because with a front end change, you can pretty much test really quickly for a proper experiment. Has my change done anything else? So there is there is this problem in in general when you run big experimentation programs at scale with lots of people in production is is that you have this concept of a novelty effect right it's **[13:37]** when something is new and shiny people click on it >> h because I don't know it's a different there's a button at a new different point it's like what is what is this button I click on it but that doesn't necessarily mean that it's so like instantly derives or gives value to a customer right so there's always an an element of time needed for customers to learn how this new ch front end change or even like backend change actually changes the behavior of how a customer interacts with a product. Right? So just **[14:08]** because you're able to ship more features instantly doesn't automatically mean that all these features then drive the output that you want to have as a as or like drive from a from a from a c from a company point of view. So there is always this element that you need to give customers this element of of time to learn it, measure the outcomes and then ensure that it solve the business outcome, right? >> Which is another bottleneck in this entire like flow of like if you just you're able to ship things faster, are **[14:38]** you actually building the right things and are you actually then measuring the impact that it does for those things? Mhm. >> Hamanchu. >> Uh like firstly I don't think anyone is anyone is really ready for how fast things are moving these days. Like you think about it, you blink and the world updates. You download an app today and tomorrow it's completely different. People are pivoting left, right, and center. People are trying to figure things out. And with AI you actually have a possibility and many people are doing it. They're shipping multiple **[15:09]** features. I mean in a 24-hour cycle you would have like three to four sheep features shipped in a day and that's what's happening everywhere. Uh so we have built systems that can evolve faster than our brains can adapt. And the thing is people say they want speed but I think they don't. They want faster predictability. >> They want to wake up and feel like that they still understand. >> Yeah. Like they want to feel that they understand. So when personal tra personal take when people **[15:41]** or when companies brag that they're shipping a 100 times a day I get it that's cool but people on the other side if they can't absorb that pace then it's chaos with better marketing. [laughter] Uh I think the real question is not can we move faster but it is can we move fast without losing trust because once people stop trusting the pace once they feel the product is moving way faster than their ability to learn and adapt they'll **[16:11]** check out and no and and sorry no AI is going to fix that and that's just a human thing like we need the buffer between progress and panic like that. >> Yeah. [laughter] A buffer between pro. How about you, Lizzie? >> Yeah, to to offer maybe a different lens on this, I think what I've been thinking a lot about with increasing the pace of delivery is how does that change how we iterate um and specifically I think about technical debt. So it's very **[16:42]** common when you're developing new features um to accept a certain quantity of technical debt just so that you can start testing with your customers to make sure that this does you know meet the value bar and it does actually achieve the goals that your customers are looking for. And so I think one of the things I'm curious about with this increased speed is does it actually allow developers to run tests in a way that's um a little bit more sustainable uh and maybe reduces the amount of debt that they're picking up on the other side. Um so I I'm not sure but I think **[17:14]** that's something that I'm really maybe excited about with this increased pace. And that maybe goes back to you know Alex's point that like front end changes we feel very quickly but there's a lot more that happens behind the scene. And so I think that for um engineers, this could end up being a really great thing to help them run those experiments in a lightweight way, but in one that doesn't maybe sign them up for some technical debt that they have to pay off on the other side. >> No, 100%. And technical debt is always this massive thing, right? We always build things with the hope that it bu **[17:44]** does something for the business, right? And no one in their right mind would ever say or wake up in the morning, oh, I'm going to build something because it's going to turn into tech technical debt, right? There's always someone had an idea. It's like this feature is going to drive revenue. It's going to make my user super happy. It's going to solve some sort of problem, >> right? But the reality is most features don't do much at scale, right? they maybe do solve some problems for some customers, but it's the holistic product that's that solves a customer problem. **[18:17]** There's there's always this like um statistic that I throw around um which is based on like lots of like couple of academic papers and like just my own experience of running a lot of experimentation programs for the last 10 years is that only around 20% of features that are deployed actually drive some sort of like positive customer impact or like a positive business impact. 60% of these kind of like features actually do either very little. Maybe they change a user behavior, but then they're not necessarily like driving revenue or like **[18:48]** customer customer acquisition so on. And the the really scary thing is like 20% of these these features maybe drive negative outcomes, you know? So you're actively building a feature that actually could harm your customer experience in the long run, right? So if we don't actually measure what we are building and we're just building more and more, we're just accelerating that problem because at one point our our our treasure trove of good ideas is probably just going to get smaller and smaller over time. So that positive number probably just goes down because at one point we exhausted the ideas of like how **[19:21]** to run a business and how we can build things, right? So I think it becomes more and more important that everything gets measured in the long run. >> Yeah. >> The right way. Lizzie, you work on that. How do you how do you measure actual outcomes? >> Yeah. Well, outcomes is really an interesting one. I think like again, you want to look at both the numerator and the denominator. Um there's a few like I mean obviously the direct measurement is like how much does your revenue change in the sort of 3 to six months after you **[19:52]** release release a product and also you can look at um retention as another really important feature, right? So how do you um increase or maintain usage amongst your customers in a way that you believe the product is meant to be used right? So that's definitely like two really key things that you can look at for outcomes. Another way you can look at it from an internal perspective is looking at engineers and making sure that um across the teams you have that sort of high level sense of impact and autonomy in order to like achieve those goals. when you look on like the **[20:24]** denominator side, so effort wise, there's a lot of really interesting dimensions you can look at and a lot of those are changing with AI. So, um we look a lot at things like um what how is the overall experience across these various touch points where you can have bottlenecks, right? So, um the code development process or code writing process like tends to actually usually be not even in the top five biggest pain points for engineering teams when they're delivering. um the review process, the testing process, the deployment process, the release process, **[20:55]** these are all areas where we actually tend to see a bunch more friction. And so you actually need to be able to look across numerous data points across numerous parts of those system in order to figure out where are those bottlenecks that a team might want to be focusing on. Um so it's never a single number but I would say like maybe to offer a hot take we've spent um 10ish years really focusing on um the you know evangelizing the Dora core four or actually now five metrics. Those metrics are fantastic for looking at the overall **[21:25]** health of systems from the high level but they actually don't help you answer the question of is my team achieving their outcomes and is my team doing it at a level of effort that they can sustain and that will allow them to continue to deliver. So I think if you're looking at like AI adoption, how development pace is changing and how to look at success in the long run, those metrics are actually likely not going to be the best way to get to that answer. >> That's really interesting. Like they are a great set of metrics, but as soon as you measure something, we start gaming the system. So it's always an **[21:57]** interesting problem. That sort of leads into another question I had which is how do you help people keep up with the culture of continuous learning when they are the human in the loop bottleneck like all of a sudden the slowest part of the system is the human. Uh do you have any ideas on that Himonu? >> If you're enjoying this then lead dev's got loads more to offer and there's tons to explore. [music] >> So the reason I love lead dev so much is they've got such a range of content on so many different topics in the world of engineering leadership and management. **[22:28]** So lead dev helped a lot on building my own career, right? And it's [music] great to also see how it's helping the new generation as well. Make sure to sign up and join us at leadv.com. I mean do we really want to face um praise it as human in the like human in the loop as I mean where they are the bottleneck **[22:59]** >> uh I think you don't like you need to flip the idea we need to look at human as quality control uh and that's not a bottleneck that's the conscience of the system give them better context text, better dashboards, shorter review loops. Uh, don't make them faster, make them smarter. But as I'm saying this, I know that this is too idealistic. Like don't make it faster. But the goal right now, everyone wants to be as fast as they could be. Uh, but what's really happening is with AI, **[23:32]** what has happened is everything is amplified. Like if you were writing good code earlier with giving it proper context with you know uh using the proper naming conventions everything you if if you're doing everything right AI amplified it but if you were leaving loose ends now AI has come and you know it has amplified bottlenecks and I guess it was I'm not sure was it Alex or Lizzy I guess Lizzy was you you were saying that now uh you know uh now we have shifted the pressure left like it's you **[24:04]** some a developer is writing the code but it's the quality guys uh the QA is you know it's it's on their head to really understand what has been what has been written here and with with this happening uh we we really need to first of all we need to so uh okay some context here I graduated in psychology so I try to look at everything from that lens uh and one of the important things to to do there is we we we **[24:35]** change how we look at things. We need to change our perception where we really need to think of humans as the last layer of quality control. uh and the goal again is not to be faster but to be more better informed and once people understand why the AI did something they stop treating it like a black box and start treating it like a junior engineer and that's when you know collaboration starts working not because people go faster but because they finally have the clarity to make smarter **[25:06]** calls uh so how can people keep up I think it goes goes back to our first question like what does safety look like in this whole in this whole space? uh can a developer can an engineer can we create an environment where somebody can come and say that okay this worked I pushed this I pushed this you know these lines of code it worked back then but I really do not know what it's doing but it did solve the problem back then and can **[25:38]** I know I mean I don't know to what extent is that possible but if we if we are able to do that I think that's when we'll start answering uh deeper questions with AI that's when we'll start looking at where AI is actually helping. Yeah, >> that's a good point, Lizzy. [clears throat] >> Yeah, I think um I guess I just want to make sure that I'm answering the question correctly. Do you mind if I restate it? I want to pull up one of Honu's points, but I want to make sure I'm in the right direction. >> Yeah. Yeah. >> So, what I asked was, how do you help **[26:09]** people keep up with a culture of continuous learning when they have become the human in the loop bottleneck? >> Yeah. Yeah. Um I think that uh having humans I guess there's kind of this like philosophical question that is like do you think software engine software engineering is going to be augmented or replaced um by AI right and I think like when AI as like a promise first came out maybe in like 2022 and we started to see co-pilot I think that was like a huge question that we at least internally **[26:39]** heard so much from customers and stakeholders and others and I think I really believe that this is a new layer of abstraction in how software is develop software engineers kind of develop software. Um but I actually think it's important that humans are still you know the bottleneck if you will or um kind of guiding and driving that process because I think to what Hamu's point like we kind like that sort of oversight and that sort of mental um capacity and understanding of what's **[27:09]** happening across um the development life cycle and ultimately the purpose that it's serving is actually really critical. I think that we're seeing this even today. context is probably one of the biggest um shortcomings with AI right now and so much of that context still lives within humans and actually that ability to sort of adapt to that context is something that is very human. Now again we can play forward in 5 to 10 years and say maybe these agents got so good at context that we actually can remove humans in the loop. But I think **[27:40]** what's really important today is about who are we actually serving when we're developing software. It's about solving problems for end users, right? helping improve their day-to-day experiences, delivering value, helping their companies do better work. And so, I think it's actually really important that the humans are still a part of that experience. And in fact, like now instead of focusing so much on the day-to-day sort of um semantics, they can focus much more strategically on are we building in the right direction? And I think that's something very new that um software engineers in the past maybe **[28:11]** didn't have as much creative autonomy to do that. But it's why you see like productled engineering teams tend to do really well at supporting their customers because their engineers can be much more closer. They can be more proactive in the way that they develop software. And now actually you'll be able to see more and more of that as engineers focus less on you know the nuances of did I write this method correctly and more on the higher level. Is this feature gonna actually solve this customer's painoint? >> Alex, **[28:41]** all excellent points, busy there. Um, what I think I want to add to this is to the point I kind of raised earlier is like if you wanna have a an environment in you can continuous learn everything you need to have the ability to actually have time to digest what happened you know and understand what is going on and and for that I think if if the problem is that a lot of new stuff is coming out like new code is coming and a lot huge **[29:11]** amount of code is coming in like some function within the organization ization is overload over overworked and over uh overloaded with this huge amount of code that comes through they can't learn as humans to try things properly right so ideally you come up with a with a technical system maybe driven by AI right to actually catch those cases triage them kill them and then flag them in in in such a way that the human can to Himantra's point become the quality **[29:43]** gate Right? Because there's a lot of cases and a lot of like things that things can be catched by an eye quite easily because it's like syntax small syntax errors, formatting errors, I don't know, some some some API unit testing fails, API connections being exposed in the wrong way. So there's a lot of things that AI can like take the work off certain things but then these edge cases where I most likely will fail because AI is still a statistical model that is based on sort of sort of normal **[30:14]** distribution and what it's basically trained of like what I have seen most likely in the past right but the things where things will break is at this very long tail of the distribution of things where it hasn't seen before right so there still needs to be a human to look at these cases right um So it all boils down to like how do you create a system where you have this augmented loop to Lizz's point ideally that it enables you to do the right things and then upskill the developers and in the long run to **[30:44]** like focus on these like highly niche edge cases where their skill set is most valued, right? Because what is a developer ideally is someone who who takes a problem, a complex problem and finds a very sleek solution in the long run for it. And that's why they're getting paid a lot of money. is right right and that's where their skill comes come from like a developer is not pred button color or I don't know change like a div from one area to another or just I don't know like some very low level stuff like a good developer focus on the big business problems to create **[31:15]** something a very unique system for a complex problem to make it very sleek for the customers to and to use at the end >> okay so that leads me to think about who owns code that has been co-written by AI. Like is it all the human is responsible for what the machine does or is it like there's AI specialists and code specialist? Um Lizzy, do you have an **[31:46]** opinion on that? >> Yeah, this is a really interesting topic. So for a bit of context like our team looks at um the changing research that is happening in the world of software development and particularly how it gets um augmented with AI um and we publish all this research. I can um share a link or rather a summarization of this research so you don't have to read the 50 pages that you know Microsoft and Google and universities are um writing out. One of the more recent studies that we saw that was actually really interesting was that AI **[32:17]** is actually disrupting accountability in the code review process that is kind of starting to make people think a bit differently about that question of code ownership. So um if you take the traditional code review process, uh there's an author and that author is sort of um assumed to have the most context of what's happening in that feature, right? So when you review that software, you might be asking questions of that author and they're the ones kind of representing that information, making those changes. Um and then ultimately, you know, once they get the approval from their team, they'll sort of ship that um or sort of merge it into a main **[32:49]** feature branch. But what's happening now is that there's sort of becoming this um movement from what we call individual accountability towards collective accountability [clears throat] in the code review process. So now when people are committing code and then putting it into a code review process, there's sort of this tacit understanding at least by the author which is quite interesting that uh that code is sort of shared by the whole team and that there's an understanding that AI was a part of the process. Therefore they might have they should still have the most context but they won't have all of the context and **[33:21]** it's really on the team to help make sure that this you know passes the various safety checks. So when you hear that you think like you know there's a philosophical question of what is a code review and what does it actually serve. Um but I think it's actually like the the transition in thinking is one that really should again make us think more about like what is the system how are we supporting the system and where are those bottlenecks changing. So for example if we are assuming that the software development life cycle requires us to have a you know highly accountable author that assumption might be changing **[33:53]** now. And so how does that change the overall system and what could be the potential outcomes of it? So I think we're starting to see that change is is happening right now and I think it's going to shift bottlenecks, make people think differently about safety checks um and ultimately again level up the software development life cycle with these new tools in play. Himanchu >> uh see uh on paper it's the company once said [laughter] but in practice I feel **[34:24]** ownership sits with someone who's willing to say uh I'll stand by this when it falls >> okay so like account >> if nobody feels that yeah and if nobody feels that quality disappears we've already seen this debate uh in the creative world like I guess it was in 2020 23 or four where UI US copyright office it clarified that AI generated art won't be copyrightable and uh unless a human contributes meaningful input into it. Now we don't **[34:56]** know what is meaningful or how much is meaningful but there have been debates around it. So authorship or ownership follows accountability not automation. uh uh I think it was a study by up level that I think they did a report in 23 or 24 uh that people who are using GitHub copilot introduced 40 40 to 50% more code errors when they accepted AI suggestions without any without sufficient reviews. So the takeaway is **[35:28]** very clear. Oversight and ownership they still matter. And the way I look at it is ownership is emotional first before it's legal. And keeping that uh emotional connection that this is mine and I'll stand by it is what is needed. But again I think it again ties back to our first question. Are we making people safe feel safe about using AI? If we do it then yes they will come up and they'll say that okay I own this part I own this section **[36:00]** but until that happens I think everything else of course we can keep debating about it we can start we can keep thinking about who owns it who's responsible for it it's going to be an endless debate but can we make people so like if you go 10 or 15 years back when people coded something or when they wrote something they felt that okay this this was this is something that I wrote and people would come and share about their monoliths. Uh, apparently that's not happening anymore. Uh, so do I have a **[36:31]** direct answer to who owns who owns it? I think legally I would say company owns everything. [laughter] But yeah, that's all I can say. I don't think I have uh a deeper insight on this two. >> Yeah, I I think that's an interesting ambiguity. Do you have anything to add, Alex? to I don't know >> who owns code that has been co-written with AI. >> Yeah. I I I Yeah. So in terms of ownership, I think the person who writes the code or commits the code should be the owner, right? Um **[37:01]** >> okay. But the AI wrote the code. >> Well, if you committed to it or like for sort of code review, right, then you should be responsible for it. >> But I think to to the point both and Dizzy makes, if if you if you don't have a clear owner, like then quality drops at one point, right? If you don't feel responsible for building it or you build something, you lose something around this. And I don't know if you heard if you guys heard about the guys this effect called the IKEA effect, which is a psychology of human bias where like **[37:32]** when you build something yourself or you even just partially build something, you feel more attached for it and you have ownership for it and you attach more value to it, right? So if you completely remove even the coding but altogether and you just hit the button, I think that is lost. And then if that is lost, if something goes wrong or something doesn't look nice or I don't know there's an outage, who takes responsibility of that one? Right. Right. If I create, if I write something and it causes like I don't know an S1 outage somewhere, I will be feel highly **[38:05]** um accountable for that because I wrote that code. But if AI co wrote it somewhere and caused the issue and no one in the team takes ownership of the whole thing, everyone is just raising their hand. Was not my fault. Was not my fault. You should have checked it. you should have checked it. It just creates this environment where people then starting blaming each other and then that causes more issues. And if you put this in like I don't know like environments where it actually can cause serious harm like I don't know medical medical devices, medical software or big **[38:35]** financial institutions that break down for some reason because someone pushed something production and no one can pay their bills anymore. That will have detrimental impacts of life. So I think ownership >> we just lost your audio, Alex. >> Oh, can you still hear me? >> All right, >> I can still hear him. I can hear him. >> Let's see if it's easy to come back. >> Okay, can you hear me? Okay, you said you should could set them here. Okay, cool. Long story short, accountability I think should be the number one thing and nothing should ever be fully without **[39:07]** even if in if if even if like city say 60% of the code business is AI generated there should be one person who says I am responsible for that code in my opinion >> I think that's right and I think um just to add one point I think a common analogy that people give is like you know if uh software developer it's almost like um fabric [clears throat] manufacturing or clothing manufacturing where like in the past like an individual was doing the stitches themselves and now we have like automated looms and so you're standing behind the loom but at the end of the **[39:37]** day like you the team is still responsible for the quality of the product that's ending up with the customers and so it's just we are taking a step back now and trying to figure out what does that accountability look like but it has to be there. >> Yeah that's really excellent points. Um, I think that one of the things that is interesting in all of these answers is like it's very much it comes back down to the human. There's uh that that famous bit from IBM where it's like a a computer can never experience **[40:08]** consequences. So, it can never be allowed to make decisions. Like, yeah, that's where we're at and where we've always needed to be. So, we're low on time. If any of the audience wants to pop a question in, we'll take it. Well, we might take it. Uh, but let me ask, do you think the challenge of AI code in production is errors or some other problem like scaling or security or speed? Uh, Lizzy, **[40:40]** >> I think it's all of the above. I mean, of course, there's like the immediate impact of um you put something into production that maybe didn't have the right safety checks or you didn't [clears throat] have some unexplained behavior that obviously causes like a P1 or a P 0 and um you end up needing to roll that back. I think that's very important. But I also think both scaling and security are really critical things to consider. I think again this kind of goes back to that like you know like I I I sometimes feel like I'm beating the dead horse saying this but like this is **[41:11]** why software development is systems thinking. So you know the fact that we're able to generate code faster is a great first step but that's like the first step on this ultimate journey of building more sustainable um software development life cycle that actually supports this you know mound of new code. I think what I've been you know maybe evolved in my thinking on over the last um few years is for example I think as a you know practitioner and a software engineer over many years like I see like clean you know easy to read **[41:41]** code as like kind of the hallmark of great software development but I'm ask I've been asking myself for the last few years like does it actually matter as long as you have a system that can interpret that like you know maybe I the human amble to interpret it quickly but if I can trust another you know agent or tool to interpret that context maybe clean clean short you know simple is not uh as important as it was anymore and does that change how scaling looks and um obviously from a security perspective though like that opens the door to **[42:13]** potential vulnerabilities if we're not thinking about that um I will end my thought by just saying that there was a recent study that came out that said that um AI generated code very often does not consider security when it um uh like generates uh the you know sort of output for a team to use. And so a very again and I can I can link this study but a very simple revision that engineers can take to actually like you know really solve that problem is in your prompts to engineers or when you're setting context for your agents ask it to consider **[42:44]** security vulnerabilities and it's shocking how much it actually changes the output to be more mindful of that consideration. I think we shouldn't have to like tell all engineers to do that. that should come out of the box, but is something that is a small modification that you can actually use to um help solve some of those issues. >> Yeah, Alex, >> to Lizz's point, right, software delivery is a is a system problem, a platform problem, right? So, the more the more systems you have, the more tooling you have, the more most likely **[43:16]** your brittle the more brittle your systems might become and the most likely thing is going to fail, right? And that's kind of like the reason why like the the the methodology of harness as a whole is just you know one platform to do to everything across the plat SDLC right from security scanners CDC CI experimentation and honor thing so there's only one system to evaluate the entire thing right because the more systems you have the more levers you do you do more things the more likely something gets missed somewhere and that's where bad things could happen. **[43:48]** Yeah, Himmanu. uh something that we didn't touch but has been around AI is you know uh like if you slip away all the noise around and the compliance talk that AI with AI I'm losing my job which is you know what everyone is talking about but you know when we look at error security or speed I don't think these are the real challenges these are the in the way I look at it these are the symptoms I mean AI doesn't create more bugs like when we **[44:20]** talk about errors it just creates different ones. And the [clears throat] danger is uh it's not that the code is crashing, it's working perfectly fine for all the wrong reasons. So your systems stay up. Your dashboards are there. They're telling you everything is fine only for you to find out later that the intent was off. So the real errors are are not syntax issue. I guess they are more alignment gaps and that's where tools like you know SEIPs or visibility platforms **[44:52]** uh or or any any other observability tool on top of your entire SDLC cycle becomes important. uh security has evolved to you know it's not just about protecting data but more about I guess you know protecting the decision integrity making sure that the reasoning behind automation stays aligned with our intent um and I think with speed now you know of course you can make things 10 times faster but the goal is also for us to become 10 times clearer with it so **[45:22]** yes uh the nature of risk is uh like AI changes the nature of risk But it also elevates the nature of leadership because now leaders have to design for understanding not just execution. You know context has to come in. It has to be at the center of whatever they do. So I think the teams uh or or the engineering teams and the products that will define the future are the ones who are going to deepen the judgment and not just replace the whole AI uh replace the whole coding and you know the input part **[45:53]** of it. >> All right. Well, this has been a super interesting discussion. I've really appreciated having time to talk with all of you about this. I think that what we see today is that we really need to remember that AI is joining us in a complicated system and we can't think of it as a replacement for anything. It's an addition to the system and it changes the system by the fact that it's there. And I think [music] that's really key to **[46:25]** our understanding of how to make it work for all of us. If you are in the audience, we would be happy to answer any questions that you have on the lead dev Slack in the Q&A channel. Um, and we really appreciate you sharing our time with us today. And thank you to all of my panelists who did a great job. Thank you.