Make AI Productivity Gains Stick  🔗 **Source:** [Make AI Productivity Gains Stick](https://www.youtube.com/watch?v=-6XbGcLEm7U) > [!note]+ Document Notes > --- > tags: [ai-productivity, developer-workflows, change-management] > author: [MISSING] > source: Lead Dev > reference: https://www.youtube.com/watch?v=-6XbGcLEm7U > date created: {{date:YYYY-MM-DD}} {{time:HH:mm}} > --- > # 🏷️ Title: Make AI productivity gains stick > ## 🧭 Table of Contents > - [📝 Summary](#-summary) > - [📖 Main Content](#-main-content) > - [🗒️ Notes](#️-notes) > - [📚 References](#-references) > - [🗣️ Paraphrase](#️-paraphrase) > - [📰 Newsletter](#-newsletter) > - [✅ Action Items](#-action-items) > - [🧪 Hypotheses](#-hypotheses) > - [📊 Methodology](#-methodology) > - [❓ Questions](#-questions) > - [🔮 Future Research](#-future-research) > - [💡 Implications](#-implications) > - [🧠 Learning Styles Assessment](#-learning-styles-assessment) > - [📊 Accessibility Assessment](#-accessibility-assessment) > - [🌍 DEIB Assessment](#-deib-assessment) > - [✅ Logical Validity](#-logical-validity) > - [🛡️ DARVO Analysis](#-darvo-analysis) > - [🎯 Conclusion](#-conclusion) > ## 📝 Summary > - **What happened**: Lead Dev panel featuring AB (CTO at Metro Fox), Anastasia (MVP Product Engineering at GitKraken), and Maxim Najim (Distinguished Engineer at Target) discussing how to sustain AI productivity gains in development teams. > - **Why it matters**: AI tools provide initial productivity boosts, but making those gains permanent requires deliberate change management and systematic measurement approaches. > - **Key stakeholders**: Engineering teams, CTOs, dev leaders, individual developers at all levels. > - **Risks and tradeoffs**: Over-reliance on AI can cause skill atrophy; quality may suffer if speed is emphasized without proper verification; costs will increase as AI subsidies end. > - **Bottom line**: Sustainable AI gains require treating adoption as major organizational change management, not just tool rollout. > ## 📖 Main Content > The panel explored three key types of AI gains: behavioral change (adoption patterns), system improvements (DORA metrics), and customer impact (business outcomes). Success requires moving from "vibe coding" to "agentic engineering" with disciplined workflows. > **Core gains identified:** > - **Test coverage improvements**: AI translates specs into comprehensive test cases > - **Delivery speed**: Features that took 3-4 days now complete in under one day > - **Context switching efficiency**: Staff+ engineers can manage 6-7 parallel agent sessions > - **Ramp-up acceleration**: AI agents help explore unfamiliar systems and domains > **Distribution patterns**: Gains are strongest among senior managers and junior developers. Mid-career engineers often resist AI, feeling it threatens their coding-based identity. Staff+ engineers benefit most because they already know how to delegate and set clear goals. > **Measurement framework**: Three-pillar approach covering adoption metrics, system changes (velocity, quality), and business impact. Teams must follow work end-to-end rather than measuring stages in isolation to avoid shifting bottlenecks elsewhere. > ### 🗒️ Notes > #### ⏱️ [00:03:12](https://www.youtube.com/watch?v=-6XbGcLEm7U&t=192) AB on productivity gains > - Increased test coverage: developers creating more comprehensive test cases with AI assistance > - Higher delivery speed: architectural design phase remains human-driven, implementation accelerated by AI > - One developer can work on three parallel workstreams simultaneously > #### ⏱️ [00:06:45](https://www.youtube.com/watch?v=-6XbGcLEm7U&t=405) Anastasia's three-pillar measurement approach > - Behavioral change metrics: adoption, engagement, frequency of use, autonomous agent operation > - System change metrics: DORA and SPACE frameworks, developer satisfaction tracking > - Customer impact metrics: experiment velocity, shipping frequency, customer satisfaction > #### ⏱️ [00:09:30](https://www.youtube.com/watch?v=-6XbGcLEm7U&t=570) Maxim on staff+ engineer role transformation > - AI raises ceiling on engineering capabilities but fundamentals become more important > - Staff+ role shifts toward shaping intent, designing verification loops, creating alignment > - Six to seven coding agent sessions running simultaneously for different projects > - Context preservation enables involvement in broader project surface area > #### ⏱️ [00:15:20](https://www.youtube.com/watch?v=-6XbGcLEm7U&t=920) Uneven distribution of gains > - Biggest gains: senior managers and junior developers > - Mid-career engineers resist due to identity threat around coding skills > - Solution: train mid-career engineers in delegation and management skills > - Hiring focus shifting to "product engineers" who understand business context > #### ⏱️ [00:18:45](https://www.youtube.com/watch?v=-6XbGcLEm7U&t=1125) Measurement challenges and solutions > - Follow work end-to-end, not just stages > - Track incident rates alongside delivery speed > - Use cohort analysis to compare AI-adopters vs. non-adopters > - Layered measurement prevents optimization of wrong metrics > #### ⏱️ [00:22:10](https://www.youtube.com/watch?v=-6XbGcLEm7U&t=1330) Pitfalls and fragility points > - Skill atrophy: developers stop exercising engineering judgment > - Over-reliance: copying stack traces to AI without basic debugging > - Context limitations: AI struggles with large codebases requiring domain knowledge > - Cost sustainability: current token subsidies will end, requiring efficiency optimization > #### ⏱️ [00:28:15](https://www.youtube.com/watch?v=-6XbGcLEm7U&t=1695) Making gains stick - systematic approach > - Treat as change management, not tool adoption > - Standardize agent instructions across teams through shared markdown files > - Create learning feedback loops from mistakes back to agents and teams > - Monitor engineering organization daily/weekly for quality degradation signals > #### ⏱️ [00:32:40](https://www.youtube.com/watch?v=-6XbGcLEm7U&t=1960) Skills for AI-augmented engineering > - Understanding over outsourced thinking > - First principles reasoning and architectural thinking > - AI literacy and supervision skills > - Domain knowledge and system thinking remain critical > #### ⏱️ [00:38:25](https://www.youtube.com/watch?v=-6XbGcLEm7U&t=2305) Preventing skill atrophy > - Form opinions before consulting AI > - Practice deliberate comparison between human and AI solutions > - Focus on problem-solving rather than code generation > - Maintain curiosity and continuous learning mindset > ## 📚 References > - [Lead Dev Slack Community](https://theleaddeveloper.slack.com) > - GitKraken Developer Tools > - DORA Metrics Framework > - SPACE Metrics Framework > ## 🗣️ Paraphrase > Three engineering leaders shared their experiences making AI productivity gains sustainable in development organizations. The key insight is that initial AI adoption often focuses on speed improvements, but lasting benefits require systematic change management treating this as cultural transformation rather than tool rollout. > The panelists identified uneven distribution of AI benefits, with senior engineers and juniors gaining most while mid-career developers often resist due to identity concerns around their coding expertise. The solution involves retraining these engineers in delegation and product thinking skills. > Measurement requires a three-pillar approach covering behavioral adoption, system improvements, and customer impact. Teams must track work end-to-end rather than optimizing individual stages to avoid simply shifting bottlenecks elsewhere. > ## 📰 Newsletter > **AI Productivity Gains: Making Them Stick for Your WordPress Team** > Great discussion from Lead Dev on sustaining AI productivity improvements! Key takeaways for WordPress developers: > 🎯 **The Reality Check**: Initial AI speed boosts often fade without systematic change management. Treat AI adoption as cultural transformation, not just new tooling. > ⚖️ **Uneven Benefits**: Senior devs and juniors see biggest gains. Mid-career developers often struggle with identity concerns as coding becomes commoditized. Solution: upskill them in product thinking and delegation. > 📊 **Measurement Matters**: Track adoption → system improvements → customer impact. Don't just count commits—follow features end-to-end to avoid shifting bottlenecks to code review or debugging. > 🛠️ **WordPress-Specific Applications**: > - AI-assisted test case genera Hello everybody. We're just going to get started very very shortly. But thank you to those people who I know are already here waiting for us to to get on. Uh really appreciate you joining us today. Okay, I think we're going to get cracking then. Um, so yes, thank you so much for joining us for what promises to be a fantastic discussion on making AI productivity gains stick. And this is a lead dev online panel discussion created in partnership with Git Kraken. The discussion today is going to last roughly 45 minutes after which both myself and the panelists are going to head over to the lead dev Slack to answer some of your questions in the panel Q&A channel. And to get there, uh, if you're not familiar already, you can go to theledeveloper.slack.com and either sign in or join up. You don't have to hold on to your questions till afterwards though. Please feel free to submit any that might come up during the course of the conversation through the Q&A feature on Zoom. If I can bring them in, hopefully I will try to do my best to do so. Otherwise, we'll uh hopefully get to a couple at the end if we've got time. And if not, we'll port everything over to Slack for you. But before we get started, I will do a very quick introduction. My name is Sally Lates. I will be your host today. My pronouns are she and her. I work as a fractional technology leader and adviser and I'm based in the east of England in the UK. But as I've said, the discussion today is all about making AI productivity gains stick. So I imagine everybody on the panel or everybody watching here today has probably seen some different benefits in the course of their work and their life. But making sure that these gains really hold up as AI becomes more of a permanent part of how teams work can be another matter. So what we're going to be talking about today is discussing the different kind of gains that we can get that our panelists have experienced in the course of their work. How we can measure these, how we can um most importantly get them to stick as part of our everyday work. So here to share their experiences, their stories. I'm joined today by um some incredible panelists and I'm going to let them all introduce themselves very shortly in alphabetical order because I love a bit of structure. So AB will be starting with you. Um could you tell us a bit about you please, the context of your work with AI and why you're interested in this topic? Hi all, my name is AB. I'm currently the CTO at Metro Fox and as a small team, we function and ship like a much bigger one now. And that's because AI has become an integral part of our workflow. And so in so many ways, I get to see the gains as well as the pains. and I'm excited to speak about them all today. Thank you. >> Very excited to have you here. Thank you. And Anastasia, over to you, please. >> Thanks, S. Uh, great here to be with everyone. Um, Anastasia, MVP product engineering, the Git Kraken. Um, many of you may know us, um, uh, through tools such as Git Kraken Desktop, uh, or GitLab, a plug-in in VS Code. But over the last 18 months, we've been deeply investigating and trying to understand how AI is impacting developer workflows and how we can help developers and dev leaders through this. So into this discussion um I'm bringing my personal experience as I'm running engineering team and we are going through the same transformation as well as um inputs from lots of um dev leaders that we were talking to um that again like going through the same and wrestling through the same pain points as you are happy to be here. >> Thank you. I'm really excited to hear some of those those data points some of those anecdotes that I'm sure you can bring and Maxim over to you as well please. Yeah, it's great to be here. Uh, thanks Ali. Um, I'm Maxim Najim. I'm a distinguished engineer at Target and I've spent about two decades working in the tech industry across large tech organizations. Like many of you, I've been using AI in my day-to-day engineering work. Um, in my recent article I wrote for lead dev, I wrote about how AI is changing my work as a staff plus engineer, especially how it can reduce the time it takes to understand unfamiliar systems. So to me, it's clear that AI can help us move faster. But I think that the deeper question is, can AI help organizations make better, safer changes faster? And what I found is that answer depends on a few things which I'm excited to uh to to discuss today. But what I will say now is like AI raises the ceiling on what engineers are able to do. But for those gains to last, we have to kind of raise the importance of the fundamentals even more. So like good architecture, good guard rails, keeping the craft of software development at the center while uh using these tools what they you know to to do what they truly help us with. Well, you're all dropping some really nice sort of clickbaity little tidbits here already. So, that's uh very exciting. And thank you all so much for giving your time today um to to chat to me and to chat to um everybody else out there as well watching. Um but we're going to start by taking this idea of gains. So, again, I'd love to kind of go around the virtual room and get everybody's perspective to get us started because I know that everybody's kind of coming at this from a slightly different angle. Um, so AB, if you could kick us off again, what kind of gains have you personally experienced AI bringing to your work or or to the environment that you've been in? >> Yeah, that's a great question. I can speak to two gains in particular and the first one being increased test coverage. Testing used to be this thing that although very important required a lot of time and investment and did feel like manual labor. So you would find that developers in most cases did the barest minimum. However, now with AI being able to translate specs into test cases, we're seeing higher test coverage. We're seeing PRs with more test cases, more scenarios being tested and that's giving us more stability and more confidence when we ship because we know that if something were to go wrong, we would know ahead of time. The second one is increased delivery speed. Right now we are seeing that what we as engineers need to lock in most times is the architectural design and those decisions. And once we have that down AI can then go and do the code implementation. Before now a feature could take about 3 to four days after you have locked in the um the architectural design. But now once you do that AI can get that done in less than a day and that same developer can then work on three other work streams at the same time thereby having more gains in terms of speed. And is that kind of echoing what you've been seeing as well, Anastasia, or do you have any other kind of like different viewpoints to add into the mix? >> Absolutely. Absolutely. Similar kind of insights. Um I'm looks I'm looking um also at uh the gains from maybe this three pillars. The first one is around adoption. Um and are we doing the cultural change and initially it it could be like very very like AI assisted like uh very minor adoption and then go into AI first uh kind of flow. Then if you have this leading indicator of um people adopting AI, do we see that our system is changing such as we have better test coverage such as if you have better velocity um if you release more frequently. So this is second pillar but what we are all marching towards is what's the impact to the business? Are we able to deliver value to the customers uh more? So I see the the gains in all of those areas and they're also super interdependent in in in our case. Uh if you talk about impact um then when we really leaned into EI first uh type of model we see um just in 3 months um 2.3 times perform uh improvements in in velocity and um around 98% of of our code changes are AI assisted but it wasn't a a journey that um just we did overnight there was a a journey with initially developers resisting then going through experimentation adoption and then scaling and optimizing and of course something that helped is changing the cultures uh um releases in the models um with OPUS like I think the whole world for everyone has changed uh with GPD 5.1 um so we really saw um ma maj major major changes but yeah um so see similar kind of observations as Abby And Maxim, I know that you really care about the cultural side that Anastasia mentioned. You really care about the adoption side. Um, but thinking about the gains specifically, how uh how has that kind of affected your work, what what sort of things have you been seeing? >> Yeah, I have about like three of them that I think are major gains that I've gotten. So, one is kind of just the ramp up time. For example, if I use AI agents to kind of explore a domain or system I don't know well. So I have an AI agent read the documentation, compare that to the code and kind of help me understand how a system is structured. Uh that kind of like onboarding work usually takes a lot of time and so AI could help me kind of compress that time and kind of reach a first useful understanding. From there, you know, I take that initial understanding and validate it with engineers who actually own those systems. And I'm I'm a staff plus engineer. So I have to create a lot of documentation, a lot of lead a lot of conversations. So I've also been seen a real gain in the I call the blank page part of my work. So things like drafting a design doc, turning messy notes into like clear arguments, summarizing trade-offs, um adapting communication for different audiences. So for me, AI kind of lowers that activation energy. So instead of spending an hour just getting to a workable starting point, I can get there much faster. Then I can spend more time kind of applying my judgment, refining the argument, um getting alignment. And another kind of big shift has been around just context switching. So on my desktop right now, I have six or seven coding agents sessions running at a you know, they're open and running. And so each each um tie to kind of a different project I'm involved in. And they help me kind of research a topics, synthesize notes, review the ideas, uh create artifacts for collaboration. And so over the last four or five months, that has changed the shape of my work. I've been able to kind of stay involved in more projects than I normally could because AI helps me preserve the context, kind of re-enter projects faster, and then kind of turn my scattered thinking into something I can share with others. So for me, it has expanded the surface area of the projects I'm involved in where I can provide more useful input, feedback, and direction without relying kind of on my own memory or finding long stretches of uninterrupted focus time. >> I've now got this picture of you with your five or six AI agents kind of like doing all the work in the background whilst you're here chatting to us. So, thank you. And I hope they're not too distracting with everything that's going on. Um but just to kind of like pull on that thread a little bit more um your role I think we've all got some quite different roles here but in particular what's your experience of um how those gains are perhaps distributed around the people that you've been working with or how have you seen them affecting different roles? Are they pretty evenly distributed? Are they concentrated at different levels of seniority or or types of uh work? What's what's your experience Maxine there? Yes, I I think the AI gains are strongest when people kind of bring the right context, the right judgment and verification to these AI tools. So for for SAP plus engineers, that's that changes kind of the center of gravity of the role. If if code becomes cheaper to produce with AI, then the scarce thing becomes knowing what code should exist. You know, what constraints matter and how does it fit into the wider ecosystem. And so that means AI is is most valuable in the kind of work where staff plus engineers are already spent spend a lot of their time. And so I think AI makes the staff plus role more important and shifts those that role towards kind of shaping intent um designing verification loops creating alignment across teams. So those are already like staff plus responsibilities but AI makes that role even more important. Um and so I I also think step plus role is making sure the AI adoption does not stop at like people are just using the tools. It has to show up in the team's operating model how people plan, shift, verify and learn and uh that is when you see kind of the true gains of AI and that's where that productivity gains can and can can start to stick. I was actually reading a lot on on this and um I would like to compare with experience that that we were having in in our teams. Um and maybe we're having a little bit different experience than what is the industry term definitely what I see that that it's not evenly distributed at all. Um and the biggest gains that we are getting um is on seniors managers and followed by tadam by juniors. Uh so and this is both true in in in develop in across devs and and QA midcareer engineers that worked so hard to become those people who gained like to that that experience that they're writing great code um they are a little bit stepping back they feel that AI is threatening to their identity um and um I really love growing people I I see that this is a mission for for uh for dev leaders and uh for me this is how can I help them to uh to get this new identity where they'll be not feeling threatened. Um so in this case how we can train them to become managers because basically like why it's so easier for staff plus for for Maxim um to to orchestrate six like 10 agents in parallel because they know how to delegate. They know how to set clear goals. they know how to review the code not micromanage uh how to do the retrospectives and provide the feedback back to to this. So these are the skills that that we are teaching for the mid-career engineers for them to get the new the new identity. Um and definitely when we are going more towards um hiring um then in this in this case uh we are looking for true product engineers um coding is not enough um they need to be orchestrators uh and also understand the business and be able to solve the problems not just uh writing writing the code. I've definitely been hearing that a lot. I think the the balance of the the product thinking is going to be incredibly important um in terms of people's development and and things like that. But we're going to come on a little bit more to some of the human stuff. Um I'm really glad you called that out. I think it's very very important. But one of the things that I know can really help people is to understand um the measurement the the sort of the structure around these kind of topics to make them a little bit less ambiguous. So when you think about measuring the gains um and measuring the impact on different ways of working and and different facets of the business, how do you think about that Anastasia? >> Um something that that um I'd like to start with is um now measurement are becoming even more important than at any other other time. Um we are going through the major change management. So, so that's why be able to measure understand what to measure is is super critical and talking to the dev leaders who had that experience before or who are obtaining that that experience urgently now. I think these are the teams that are showing the biggest improvements. Uh for me what to measure um I introduced this uh this this concept with three pillars. Uh one is being this behavioral change. um than system change and customer impact and all of them are having certain measurements that uh that um um you you would you you would be adopting depending on the phase of your AI adoption for example for behavioral change it could be initially is there engagement that is happening is there adoption is there frequency of the use but when uh you are in more um maxims type type of experience um uh are your agents able to run autonomously for a long time without much of the of the supervision without making too much of mistakes. So there will be different type of measures uh on the behavioral change for the system. Um I would I would rely on on the DORA type of metrics uh on on uh space type of metrics. Um here super super critical with I in addition to the adoption we need to as dev leaders track developer satisfaction because in this world when you're running tens of agents it's so easy to get to burnout and like super easy not because people are are really like trying to do as much as possible just because that enderfin that you're getting from this short from from the quick wins It's just like keep going, keep going and and now it's like oh it's already like 18 hours without non-stop like easily. And yeah, the last is is customer impact. Um are we running more experiments? Are we shipping faster? All of those things. Do we see customer satisfaction improvements? These are the the metrics that I would be looking at and they're all important. We can't ignore um just adoption. uh we uh we we this is leading indicator that your your system is changing and the end goal is is customer of course and just as you started speaking we had a question come in um on the zoom Q&A um which again just a reminder to everybody you're very very welcome to submit things there as well um I think you've perhaps answered quite a bit of it so the question was um could you please name some specific metrics and value that are specific for AI and overall improvement and efficiency with using AI. But then there's a follow-up of how are you able to report this up to the board and executive leadership to show how AI is able to improve products collaboration and ultimately success. So I wonder if there's anything else you just wanted to kind of build on with your with your previous answer or any other points that could be useful to this uh this person. >> Yeah, on the on the metrics um again it depends on where you are in adoption. Um so so that's why initially it could be maybe line like lines of code or acceptance rate that kind of things on on the far spectrum it it could be are your agents working efficiently um and taking into account the cost that will be coming up very very soon as as a problem your uh your investment having the the right level of ROI um in Dora it's it's all clear space also like just just take those frameworks and and apply um for customer impact what I'll suggest is um especially talking to uh to the board and and uh exposing the the business metrics look for the signals what are important for your company. Sometimes it could be um uh is your go to market faster then are you shipping faster? Do you have more experiments? Sometimes it could be um is your is your organization cost efficient? So get get those questions that are important for the board that are important for exact level um uh and and try to translate the metrics the system metrics that you have towards that. Um for us uh we we were sharing the productivity gains with with with a board and uh something that I shared just January versus April it's uh 2.3 across all the team. Of course it's not equally distributed but it's across the teams. So you can use this this comparison before certain milestone and after certain milestone. In this case there could be also some other things that are changing in parallel your process changes um or your uh your tools introduction. So there could be a little bit of variance. Um other way how you could measure is building on cohorts and comparing. um you will have people who are um in learning AI adoption and you will have um really those people who are going going aggressive they're going through the same or organizational changes. So um there if there's significant um uh number of people in in your team and you can you can derive from that then potentially you can do the cohort uh analysis and compare uh one cohort versus another and represent this as um as the data for your board. >> Thank you. I think you've touched on some really interesting nuances there. One of the things that I'd just love to to stick on for for a minute longer is just how do you separate those real genuine productivity gains um from gains that maybe just shift the work elsewhere? So into review or debugging or or rework. How how do you kind of actually get down to the nitty-gritty of measuring the the real gains? >> Um I can I can take this question as well. I'm very passionate about it. about about about the the data. Um, an analogy that I like to to uh to use is now we are getting all these muscle cars that are 10 times faster, 100 times faster, are we getting home sooner? And I think in most of question in most of areas it's like no. And then the question raised why why is why is not it's not that. And here um a little bit of bummer. There is no one sizefits-all. There is no one recommendation. Oh, it's because of this and this. But my recommendation is is um is use this layered measurement and this is why why it measures without this uh without the endto-end visibility. Um you can say that my coding has improved to that my code my my test coverage have improved by that. But you'll be not tying this. Are you seeing the business outcomes? So having the the layered measurement is um is super important. Um for for us some of the bottlenecks that we we gain after this uh this transformation is is like we have uh PR review bottleneck, we have requirement bottleneck and this gives opportunity as as for for organization to tune for the next things. So bottlenecks are not are not bad things like if you know about them because this gives you an opportunity to really tune and get even more from uh from from um what what you have and yeah any anyone else would like to add >> I want to add something to that I feel that it's highly important to follow the work through as opposed to following stages. So some people say oh we had two times PRs and then PRs have have increased by four times or five times. However incident rate has also increased. So we need to be able to follow the work through from inception to delivery as well as what happens after. So if we're able to say we're shipping 10 times faster, however the incident rate has stayed the same or even reduced now we're sure that we haven't moved that work elsewhere. So if we follow the work through completely, we're able to tell if we're really having AI gains or the work just moved elsewhere. >> Well, let's stay on that slight downer then. Uh Amy, I'd love you to take us through um maybe some of the pitfalls that you've seen um or maybe some of the gains that look great but then ended up being quite fragile um and you know disappeared or something like that. Could you tell us a few stories about uh some things in the in the less good realms then? >> Yeah. Okay. So I can speak to this in two contexts. In the context of the user and in the context of the usage. In the first, I'm seeing more and more developers become overly reliant on AI such that they begin or they stop working their engineering muscles. For example, there's an incident and the first thing the dev does is to take a copy of the stack trace, place it on um on cloud code and then return to say plot said the issue could be a b c d and you're wondering with the amount of context you have if you just took some more time and looked more closely you would see that the issue couldn't be any of these things and it's totally something else. So with more people outsourcing their thinking to AI, we're seeing skill atrophy and that isn't good for any team at all. And that's why I agree with something Stasia said around mid devs feeling like their job is gone because most of what they knew was just to code. And now that AI is doing more of that, we're seeing that it does seem that they don't have much to actually bring to the table anymore. Now, in terms of usage, I would say that I've seen AI gains look fragile when we apply them to um large code bases, especially where you need a lot of context to be able to generate good working code. And most of the time the context lives either in an RFC somewhere or in someone's brain somewhere else. And it's difficult and very challenging to pull all of that together and give that to AI. So in those situations you will find that in isolation the code looks good but then as part of a whole it doesn't help anyone at all and there's actually higher churn rate. So in areas where you need a lot of context, I'm seeing that we can't really find stable AI gains there unless we do the work of investing in providing that context. >> Yeah, thank you. And um I've had a question come in that Oh, sorry, Anastasia. Did you want to chip in? >> Yes. Yes. uh if you if if you talk about the context something that that is is important for us to keep gaining this uh this this values is how we can learn from the past mistakes. Um so agents will be not repeating this over over and over again. Um this this is something that will be making us less fragile in in in the future. Um and another pitfall that that I saw is um people were getting to AI adoption uh and then they stopped experimenting that we already um have AI adopted uh we we're good let's focus on our road map or whatever thing is that AI is changing so fast and this is not just step function that you conquered and that's it. So what we need to do is keep having this build measure um learn loop otherwise we'll be losing our our gains. And something that I personally um is is is is afraid of of um is that lots of tokens today they're subsidized like we are not paying the full price once the real prices will come and um I'm looking right now at at yesterday's announcements from cloud code that starting from June 15 paid cloud plans include monthly cloud agent SDK credits. So this is beginning towards that. So that means that if we go in in that route and it we will be going there 100%. um all the gains that we have with AI with uh be able to work with parallel 10 of agents in at the same time we'll be just losing those uh and for now for for us I think what we need urgently do while we still on that subsidize we we need to um uh to see how we can use those uh tokens efficiently but you can do this only after you scaled so first we need to scale AI adoption But after that optimize and we have an amazing phase right now to to get the biggest ROI from each token that we're getting. >> Super important point I think on a number of different fronts. But as you say the the sustainability angle is going to be I think a real discussion point uh once this initial kind of revolution has happened. There's going to be I think another wave of uh evolution after that. Thank you. Um I wanted to go to a question from um Vishal, thank you for the the question which was what have you done for your code bases to generate reliable code which follows the standards and has context of other code bases and it feels like this is coming from a place of Michelle's possibly had a bit of a painoint. So Maxim, I'd love to throw that one over to you if possible. Um I wondered if you had any thoughts on this one. Yeah, I think I think you have to have in your repo the right instructions and markdown files about the what your repo is about. There's also um you know agents MD and skills MD things that you can markdown files that can help uh define how to interact with the codebase. And you definitely want to expose your architectural intent more explicitly like your ADRs, um, reference implementations, things um, and give those as MCP tools to the agent so that it's able to discover those as it needs needs to do that. Um, and so I think that's really the context engineering aspects of of doing that correctly. Um so you need you need to have those standards those examples easy to retrieve for for the agent. Uh and then you also need better verification right so whatever the agent produces you also need to uh have the right uh quality checks and verification in place so that it's not you know whatever it produces you're blindly accepting. >> I'd like to add something to that. I've encountered situations where we have different developers, they have their own agents and all the agents have their own MD file and they're doing things their own way and it's like what's the standard here exactly? So I think it might also help as a team to come together and say what are our standards put that in an MD file what are our architectural decisions if you have a file like that where you add the links based on the domains or the models and this should be git like you should be able to actually commit this to git so that as it changes everybody gets those changes and then all the agents are working with one instruction as it regards the standards and where they should pull docs from. I think that has actually helped my team because we were at a point where everybody's agent is just doing what they want and then you have to constantly be reviewing the code to say oh no this is not the standard but once we were able to standardize that in one file that everybody then shares things became much easier. And talking about measurements, sorry that this is that this is that person uh something that will be very good um across all your repositories have that kind of assessment as a score um like um uh how we are trending. Are we are we improving on on our um uh on our instructions? Are they standardized? Um and this can be building a road map towards what do you need to invest next. So just having that visibility. >> Well, we could probably stay here all day talking about challenges, difficulties, hearing stories from you folks, hearing stories from the the folks watching. Um, but I did want to return us to the topic uh or the title that we were going with for this session, which is how do we get the gains to stick? So, we've heard about a few ways um that we can actually improve adoption. we can improve some of our practices, but what are the really deliberate things that we can do to mitigate those potential issues um or to make sure that we are actually actively working towards getting gains to to stick. So, Anastasia, I know you really care about this. Is there anything else that you'd like to add into the mix here? >> Um yeah, yes, absolutely. um in order for games to stick um we we need to treat this not as as a tool adoption um or a tool roll out. This is our like biggest change management that we went through for a long time. Uh this is our workflow and c cultural change. So that's why setting the goals what um setting the standards are super important. Adoption is just like minor minor step here. Um and as a part of this transformation, growing people, growing developers, um uh QA folks to become the the the the product engineers. Then um uh I think I mentioned this in one of one of the one of the questions is how we can create that learning feedback loop um that um if something is not working or we are got getting not the efficient results from uh from from the agentic workflows how we can bring this back to you as a developer like have that then how we can bring this to the team through for example this g versioning inside repository that Abby was was was talking about and then actually to the agents themselves. So they will be reusing that that information uh uh and that learnings together. Um and yeah for for for me again measurement person here uh love the data so much. Um we we need to monitor our engineering organization at daily weekly basis and see what are the what are the trends uh because something maybe was working great but then some small portion have changed and you have this butterfly effect where the quality have degraded um and you think that everything is great everything is is good my velocity is is is amazing and you are having this side effect so measuring velocity speed quality and and please please measure developer experience. Burnout is is real thing that we need to take care of. Um and when you're looking through this through this data then identify what are the new raising questions new raising issues and this is what will be helping this the the changes the the improvements that you have already today stick and you will multiply you will improve and improve and improve. Let me add something to that. This might sound really funny, but there's something I do on my team which is surfacing AI errors to the whole team. We kind of joke about it, but it helps to reinforce that that learning that AI is not the driver. You are the driver and AI is the tool and you shouldn't let AI be the driver because AI can actually make mistakes. So at the times where we see this, we just kind of surface it to the team and be like AI just did this and this wrong. That just lets everyone know that yes, even though AI is giving us these gains, we should actually still have that human in the loop such that we are overseeing the work and we don't get to the point where it's like everything hits the roof. Yeah. And I like the distinction between vibe coding and kind of agentic engineering because I think it gets to something important. So, you know, vibe coding could be useful. It helps people explore, prototype, get something out quickly. But if we want AI productivity gains to last, teams have to be move towards more like a disciplined agentic engineering. To me, that means the agent is like operating within a real engineering workflow. There's clear intent. There's curated context. There's explicit constraints. There's tests, boundaries, and reviews. And that requires more more than just better prompts. It means teams need to to have better specs, uh, you know, better instructions, better internal documentation, a lot a lot of stuff we've been talking about. Uh, we Yeah. And we also need to make the systems, you know, more legible for agents. So they might you means better API documentation, MCP servers uh you know for the agents to have to inspect our environment better and yeah context engineering you know matters matters a lot here um but has to be maintained like any other uh parts of our engineering system engineering systems and the failure mode for me is not always that you know AI generated code that fails immediately or is bad code often the code works today but quietly it kind of raises the cost of every future change. So the team that makes that the gain stick will not necessarily be the team with the most, you know, tokens or the newest models. It's those that uh they will be the teams that have the strongest feedback loops and the most disciplined engineering practices around these AI assisted work. >> Well, let's stick a little bit there on the on the more human side of things then. What kind of skills, Maxine, would you say are the ones that are going to become most valuable as AI improves, helping us to really kind of maximize those gains? >> Yeah, I think as as AI improves, I think the one of the most valuable human skills is is understanding. So, there's a there's a line I I like a lot, which is you can outsource some thinking to AI, but you can't outsource your understanding. And so, AI can help draft the spec. It can explore the code base. It can generate options. It can write the test. It can propose a solution. But as the engineer, I'm still I still need to understand the problem deeply enough to know whether we are solving the right thing. And I need to understand the solution deeply enough to kind of explain why it might work, where it might fail, uh what the trade-offs, you know, might be and whether we can maintain it, right? And that that becomes even more important as AI output gets more polished. And you see kind of the the weaker AI answers are easy to reject, you know, but a plausible polished answer is more dangerous because it can look right while hiding kind of a bad assumption or a security issue or design choice that does not fit the system. So skills like judgment, taste, domain understanding become more important and all those skill depend on on us understanding you know and what what AI is proposing. And if um and if I if I don't understand the problem, I can't give the agent good intent. If I don't understand a system, I can't provide good context. And if I don't understand the solution, I can't, you know, reasonably approve it. So I would summarize it like AI can generate options, but engineers still own the understanding. And the more capable AI becomes, the more important it is that humans kind of stay accountable for what the system is actually doing and and why. Ay, anything to add on that front? >> Yeah, I'd say two things are very important now and that's understanding things from first principle and architectural thinking because if you don't have the deep understanding, you can't actually critique AI's work or push back where necessary. So right now it has become extremely important for developers, engineers everywhere to begin to understand things from first principle so that you can actually push back and critique work of AI. >> I agree with uh with all points from Maxim and Abby. Um maybe one more thing that that is um that that I'll add here is being uh like understanding AI and AI literacy and um know how to supervise AI efficiently. Um this is the skill that we're obtaining at very rapid pace right now. Um that this will be this will be super useful combined with your human judgment with ethics with with deep do domain knowledge with um uh with with this um system thinking. Um yeah, we've actually had we've had a ton of questions. So thank you so much um for everybody who's really been engaging with this. We're not going to get to them all. Um but I did just want to pull out one last one which I think is related to this of what ways have you found to avoid engineering skills atrifizing or atrophing? Um and I think my interpretation of that is maybe um some of the more um dare I say air quotes traditional engineering skills. We've talked about the over reliance on coding agents and things like that. But I'm guessing it also ties into um getting left behind, not being up to date. So, do any of you have any thoughts on on how to keep skills relevant, how to keep them up to date, and how to do some deliberate practice around some of the things that we might lose along the way? Um for me maybe it will be um keeping curiosity for for for um deaf leaders for as developers as as as SQA and keep learning keep learning. This is this is one second um is uh how you we can understand better customers and um their problems what we are trying to solve what's the best way to solve this. So again code writing code is is cheap now solving problems is still very relevant. Um this will be number two and number three um we still need to understand how system is working. We still need to understand u what their architectural principles. So this these are the things that um for me will be top three. There'll be definitely more more other things toward that. >> I can add something to that which is the fact that you should try to form an opinion before you ask AI anything. So before you say how can we architect a system that does A B and C try to architect it yourself first and then you can compare find gaps and that way you are actively growing as opposed to just asking AI to do everything for you and then you just review. So this way you still be flexing the muscles and growing them. >> I feel I was going to say you're absolutely right and then ran the risk of sounding exactly like AI. So, thank you for that. Um, so I'm just going to kind of bring us to time in a sec. Uh, I think this time has absolutely flown by. So, thank you for sharing everything that you have. But what I'd like to do is just go around the room really quickly, give you a chance to just um offer any closing advice that you'd like anybody listening today to to take away from this conversation. So, uh, Maxine, let's start with you. >> I would echo what Anastasia was saying. I think you know you have to measure quality not just velocity and it's easy to just count the commits and PRs and feel good about you know AI but the harder and more kind of important question is will the code base be healthier in 6 months from now and so building a visibility into the quality signals looking at code duplication look at code health look at incidents maintainability uh look at you know whether teams are actually able to change the system safely over time. I think the team that wins with AI in the long run will be the teams that make the quality signals just as visible as the speed signals. >> Fantastic. And Amy, how about you? Any closing wisdom? >> I would say as a developer, do everything that you can to make sure that your skills do not atrophy. >> Lovely. Short and sweet. And Anastasia, >> this is this is super wise. Abby um superwise um for me um we we're going through the major transformation that we we never went through previously. Um and as as this is major change management be able to build system tune this measure and learn and keep iterating and having this loop is super important. It was never that important as today. So that that will be my my recommendations um to have really gains that will stick that will multiply that will grow and not have the major side effects on your company um and and your customers. >> Well, thank you all so much for sharing. It's been great to hear some different perspectives. It's been great to hear um how you've practically been uh attacking some of these topics and and making them work for you. And thank you as always for sharing all of your all of your thoughts, all of your wisdom with the audience today. Um what we're going to be doing is heading over now to the lead dev slack and we're going to be in the panel hyphen Q-N a channel. Um so for all the folks um there's a few outstanding questions, some really really good ones, some meaty ones. I'd love to have had time to discuss the mid-level devs one a bit more. I think that's a really fantastic conversation. Um, if you want to continue the conversation, if you want to get the answers to your questions, please head over to Slack with us now to continue the discussion. And just a reminder, if you're not in there already, you can get to it through the lead developer.slack.com. But thank you all so much for joining panelists, everybody watching, and we hope to see you next time. Thank you., Make AI Productivity Gains Stick, YouTube, 01kv3t2esb3g9ncyvbt9bvhn7z.