The High Cost of AI Immaturity  🔗 **Source:** [The High Cost of AI Immaturity](https://www.youtube.com/watch?v=aLH3Pa_yzWs) > [!note]+ Document Notes > --- > tags: [ai-development, developer-productivity, engineering-leadership, code-review, developer-experience, ai-governance] > author: [MISSING] > source: Lead Dev > reference: https://www.youtube.com/watch?v=aLH3Pa_yzWs > date created: {{date:YYYY-MM-DD}} {{time:HH:mm}} > --- > # 🏷️ Title: The high cost of AI immaturity > ## 🧭 Table of Contents > - [📝 Summary](#-summary) > - [📖 Main Content](#-main-content) > - [🗒️ Notes](#️-notes) > - [📚 References](#-references) > - [🗣️ Paraphrase](#️-paraphrase) > - [📰 Newsletter](#-newsletter) > - [✅ Action Items](#-action-items) > - [🧪 Hypotheses](#-hypotheses) > - [📊 Methodology](#-methodology) > - [❓ Questions](#-questions) > - [🔮 Future Research](#-future-research) > - [💡 Implications](#-implications) > - [🧠 Learning Styles Assessment](#-learning-styles-assessment) > - [📊 Accessibility Assessment](#-accessibility-assessment) > - [🌍 DEIB Assessment](#-deib-assessment) > - [✅ Logical Validity](#-logical-validity) > - [🛡️ DARVO Analysis](#-darvo-analysis) > - [🎯 Conclusion](#-conclusion) > ## 📝 Summary > - Engineering leaders discuss the gap between AI adoption rates (98% usage) and actual productivity impact > - Code reviews have become a new bottleneck as AI generates larger, more frequent PRs but humans still need time to build mental models > - Teams need stronger engineering discipline, not weaker practices, to succeed with AI > - Platform infrastructure for agentic workflows requires centralized development environments, identity management, and observability > - Token costs are becoming prohibitive, with some teams maxing out budgets by mid-week > - Successful AI transformation requires psychological safety and gradual change rather than top-down mandates > ## 📖 Main Content > The panel explores how high AI adoption rates don't necessarily translate to productivity gains, creating what they term "AI immaturity." While teams report 98% adoption across AI tools, measuring usage alone provides little insight into actual value creation or return on investment. > Code review emerges as a critical friction point. AI can generate code in seconds, but humans still require the same cognitive time to understand, verify, and provide meaningful feedback. This creates a bottleneck where productivity gains in code generation get lost in the review process. > The solution isn't weaker practices but stronger engineering discipline. Teams need two review cycles: first between developer and AI agent (with automated checks, static analysis, and tests), then human peer review focused on intent, architecture, and domain knowledge rather than low-level inspection. > For autonomous agents, organizations underestimate platform requirements: reproducible ephemeral development environments, scoped short-lived credentials, full audit trails, and LLM gateways for governance. Local machines can't scale to hundreds of background agents needing enterprise data access. > Cost management becomes critical as teams hit token budget limits early in work weeks. The "dark factory model" targets $1,000 per developer daily, but most organizations can't sustain that spend. Solutions include using expensive models (like Claude) to supervise lower-cost implementations. > Change management proves essential. Individual developers face identity threats and skill anxiety. Successful adoption happens when AI solves team problems nobody was addressing, creating social proof. Organizations need psychological safety and gradual evolution rather than ground-up transformation. > ## 🗒️ Notes > #### ⏱️ [00:03:12](https://www.youtube.com/watch?v=aLH3Pa_yzWs&t=192) Introduction and context setting > - Most teams have high AI adoption (98%) but unclear productivity impact > - Gap between adoption metrics and actual value creation > - Need to bridge adoption vs. impact divide > #### ⏱️ [00:08:30](https://www.youtube.com/watch?v=aLH3Pa_yzWs&t=510) Michaela on measuring adoption > - Adoption metrics show usage and experimentation levels > - Don't indicate productivity gains or ROI > - Can reveal friction, policy problems, or trust issues > - Useful for baseline comparison but not impact measurement > #### ⏱️ [00:12:45](https://www.youtube.com/watch?v=aLH3Pa_yzWs&t=765) Kate on productivity metrics beyond adoption > - Teams exist to deliver value to business > - Need to measure value delivery vs. reliability costs > - Revenue per developer should show step change > - Engineering needs to connect to business financial statements > #### ⏱️ [00:18:20](https://www.youtube.com/watch?v=aLH3Pa_yzWs&t=1100) Eric on latent risks > - Value may be in developer skill improvement, not code artifacts > - Higher skilled developers better at reviewing AI-generated code > - Problematic if engineers can't explain code shipped last week > - AI amplifies existing system quality (good or bad) > #### ⏱️ [00:23:15](https://www.youtube.com/watch?v=aLH3Pa_yzWs&t=1395) Michaela on code review transformation > - AI creates larger, more frequent PRs but humans need same mental model time > - Two review cycles: developer-agent collaboration, then peer review > - First cycle uses automated checks, static analysis, tests > - Second cycle focuses on intent, architecture, domain knowledge > - Avoid large PR trap - maintain small, focused changes > #### ⏱️ [00:30:40](https://www.youtube.com/watch?v=aLH3Pa_yzWs&t=1840) Eric on agentic workflow infrastructure > - Need reproducible, ephemeral, policy-controlled development environments > - Remote agents require centralized infrastructure, not local machines > - Identity management with scoped, short-lived credentials > - Full observability and audit trails through LLM gateways > - Environment reproducibility essential for debugging failures > #### ⏱️ [00:35:55](https://www.youtube.com/watch?v=aLH3Pa_yzWs&t=2155) Token cost challenges > - Teams maxing out budgets by mid-week > - Agentic workflows are "token eating machines" > - Dark factory model targets $1,000 per developer daily > - Need cost-effective strategies using cheaper models supervised by expensive ones > #### ⏱️ [00:40:30](https://www.youtube.com/watch?v=aLH3Pa_yzWs&t=2430) Change management and adoption > - Social proof powerful - team workshops create routine usage > - AI adoption for team problems nobody was solving > - Individual developers face identity and skill threats > - Need psychological safety and gradual evolution > - Organizational values around human contribution essential > ## 📚 References > - [Lead Dev Community Slack](URL not provided) > - [Coder AI Development Infrastructure](URL not provided) > - Dan Shapiro blog post on tricycle method (URL not provided) > - DevX developer experience platform (URL not provided) > ## 🗣️ Paraphrase > Engineering teams are caught in an "AI maturity gap" where high adoption rates (98%) don't translate to meaningful productivity gains. While developers use AI tools extensively, organizations struggle to measure actual business value beyond usage statistics. > Code review has become the new bottleneck. AI can generate code instantly, but humans still need significant time to understand, verify, and provide feedback on the output. Instead of weakening review practices, teams need stronger engineering discipline with two-tiered review processes: automated developer-agent collaboration followed by human peer review focused on higher-level concerns. > Organizations planning autonomous AI agents underestimate infrastructure requirements. Success demands centralized development environments, proper identity management, comprehensive observability, and governance frameworks - not just better local tooling. > Cost management is becoming critical as token expenses soar. Some teams exhaust monthly budgets by mid-week. The sustainable path involves using expensive models to supervise cheaper implementations rather than unlimited spending on premium services. > Human factors matter most for successful transformation. Developers face identity threats and skill anxiety. Organizations succeeding with AI adoption focus on psychological safety, gradual change, and solving team problems rather than mandating individual behavior changes. > ## 📰 Newsletter > **AI Development Reality Check: Beyond the Adoption Hype** > Engineering leaders are discovering that 98% AI tool adoption doesn't guarantee productivi Hello everyone and huge thank you for joining us today for discussion on high cost of AI maturity. This is a lead dev online panel discussion created in partial with coder. Um the discussion will last roughly 45 minutes. Um and then after that myself and the panelists will head over to the lead dev slack to answer some of your question in the panel Q&A channel. Uh for those who are joining us live uh we will be dropping like a link in the chat. It's really a great community of engineer leader supporting and learning from each other. So please join. We may also have time to answer a couple of question live. So please do submit those through Q&A feature on Zoom and we'll get them if we can toward the end. Um let's start with some introduction. If not, you know, you're going to get bored. So my name is Simony. Uh and I'm head of engineering at Crack Technology here in UK. And today I'll be joined by Mika, uh Kate, and Eric to share their experience and insight. And I'll I'll ask you each one of you to introduce. Let me start with Mela. You go first if you don't mind. >> Yeah. Thank you so much for inviting me. I'm super excited. Um I love the topic AI and how this shapes reshapes our our work environment. Um a little bit about myself. So I have been focusing on developer experience, developer productivity, code reviews for a long time already and um well this is I think the most exciting time in my career now with AI changing and rattling through our engineering practices. Um I'm a researcher by heart and a developer by heart. Um, I really value empirical findings, datadriven decisions, but also very much human- centric approaches. Right. >> Thank you, Kate. What can you go next? Do you mind? >> Um, I'm Kate. I'm the author of the engineering leader, which is published by O'Reilly. I'm a fractional CTO at twill and I also am building the platform DOI your career with my friend Jean Sue. >> Awesome. Eric, please go next. >> Yeah. Yeah. Good to meet you all. I'm the field CTO for the Emia region at Coder uh based in London uh and the author of the AI maturity model of which we're uh sort of speaking to today. So uh representing Coder where we're building the AI development infrastructure which includes uh cloud-based development environments uh and the platform layer as well as AI governance for those agentic workflows. Um I started out about six years ago as the founding sales engineer uh and I'm quite hands-on with customers to this day uh serving mostly regulated industry. So that's really the lens of which I will speak through. Uh and what's really sort of exciting uh to me about this conversation and sort of the market today that we're in uh is the chat is now about how do we move from just AI code pilots to running these agents safely. So keen to discuss that here today. And this is really like a perfect segue for me. So just for for the audience just to contextualize a bit what we're going to talk about. So uh my take is that most of us have gone through the first hard. So engineer teams are using it today probably um code completion or local agents. Um the honest question is where that is actually changing what we ship. For example, in my organization weekly after it hit is probably like 98% across all AI tools last month. But is it a useful signal? Um adoption number don't tell us anything about whether we're getting better at our work. And that gap between adoption and impact is what I think we mean for I maturity or immaturity if you like. Um we don't want to use AI just for the sake of it. Um so what do we need to do to bridge that gap? How much does the environment make the difference? Is it the tooling? Is it about the culture? Is the way of working? Is the code um or at least which part of that make the biggest difference? So I want really today I really want to dig into this how self asset where your team actually is what it takes to move you to the next step and yeah and see how it goes. Um I think I want to start with a a bit of a you know scene setting question. Uh and Michael I think you would be perfect for this one. So um is it valuable to measure adoption? Um what are the challenges and is it higher or is better? Um what what do you think? >> Yeah, that's a good question. I I like it and it's Yeah. Um let me start like this, right? If your goal is to know how widely how how strongly I is used in your ORC, I think adoption metrics are a clear signal on that. Right? So it tells you whether or not um this is used depending a little bit like it's al not always seamless and not always straightforward right but let's say if we have some good metrics around that it could tell you that um but they don't really show you anything about productivity right neither about return of investment um I think there's a lot of FOMO going on in AI right so everybody wants to jump on it and that's because especially the large organizations that sell us AI make these strong claims about AI adoption and productivity gains for example right so if this is if this is what you're interested in then well adoption is not the right metric to look right but it's it's a good metric also if you want to know the level of experimentation or learning that's happening I think right so um if devs are using it um using the technology they will get familiar with it in one way or another right like um it also could show you for example if there's friction or challenges that they are experiencing right so if you see that your adoption uh numbers are really low. You can also compare that with baselines, you know, in other organizations, the data that are out there. Then it could actually show you that maybe there is some, you know, some challenges that you have to dig deeper. Could be also like trust or policy problems. Um but the metrics itself itself doesn't tell you anything about, you know, what's actually going on, right? So yeah, it's it's it's one metric, right? It's easy to get to. Yeah, I think that's why a lot of people are pulling those numbers. Um, but yeah, if you want to understand if AI is the right fit or the problems that you're experiencing or you know, productivity gains, then obviously adoption metric doesn't really tell you anything about that. >> Okay. And the other thing that I think is interesting to talk about is that uh for me um adoption is a developer experience metric not necessarily like a you know a a target or it shouldn't necessarily be a target. So that's something we want to make sure to add. Um but once you know just to follow up and maybe Katie you can help me with this one once weekly active user sheet saturation in a team. So we got like I don't know we got like above 90% now. So what I feel like this is stopping be a useful signal of progress. So what the next line number you should be looking at instead. What kind of things we should see like going up or down? >> Yeah. Uh I mean I think to answer this question you have to come back to like what is the purpose of a team which I don't think like I feel like in the Zer era it's like there was this vanity metric of headcount and then the you know a manager existed to be an abstraction over that headcount and so those vanity metrics are are gone but we haven't actually replaced them with something useful and so I think like a team exists for a purpose and the clearest articulation of that purpose is they're supposed to deliver some value to the business Right. And I don't think we knew how much teams were delivering value before often, but but now like I think that metric becomes critical, right? And so the whole premise of AI and like why companies are willing to to do things that seem like objectively unhinged in pursuit of it is because AI adoption should result in a step change in productivity outputs, right? And and what is the productivity? That's the delivery of the value to the business. But then on the other side of that, and this is what developers talk about a lot, is they're like, "Okay, you might get that step change upwards, but what about the possible step change downwards in reliability, right?" And so those are the things that that need to be managed, right? But you have to know what they are. If you didn't know what your reliability metrics were before, now you're in a having a worse time. If you didn't know what the goal of your team was before, now you're having a worse time. And so I think you have to come back to these like very foundational definitions of the value of a team and say are we delivering more value net right which is the value to the business minus the cost to the business which is the reliability issues. >> Can I can I jump in here? Um so well metrics I've been leading um the research behind defax the developer experience platform that measures a lot of things right. So um and I I totally um see what Katie is saying, right? If you don't have a baseline, how are you going to compare? How you you know, are you getting worse? Are you getting better? But maybe um measuring return of investment, right? Measuring the impact, right? That that has on your business is really really tricky. But what we can do quite nicely in the engineering space is measuring our engineering systems, right? the the health of our code base um the you know developer experience be it how fast our CI/CD pipeline is right a friction in those processes so even though I think like this would be the ultimate goal there's a lot of research and a lot of attempts to really measuring the the real impact that you have you know with changes in technology maybe a first step um and I think a lot of companies are not there yet is measuring your engineering system right your processes your your teams um the deliverers that you have around that the experience the developer experience that you yeah that you that you also ship with your systems and the the processes that you build. I agree with that up to a point, but like like you know it's it's not true that we can't measure the value of the the developer team to the business like this is literally what financial statements do and the expectation of financial statements has shifted right and so revenue per developer for example like there's supposed to be a step change in revenue per developer in part because it pays for this tooling and so yes we can measure the developer experience but I think like you know it's time for engineering to be a little less insulated and be like our value is, you know, whatever system we're building and instead we need to be real like our value is whatever the value is to the business and that should show on the financial statements. >> Yeah. And just to speak uh from the market perspective, Michaela, you mentioned it's difficult to measure ROI. That's what I've seen speaking to large enterprises where they have a hard time understanding, you know, what that developer productivity metric looks like and what is the actual bottom line number to the business. I ideally yes we want to create a a value per developer head and and what that means to the business but uh I think enterprises are still struggling with determining what that metric looks like and it's really a shifting goalpost now because sure you can you know previously measure engineering processes like the amount of pull requests created uh lines accepted and ultimately time to market from a product perspective but uh you know are you measuring the right things and is that actually tracking with the business it's that's the difficult question to answer. >> I think it's also a little bit of a mindset thing. I really have a hard time and and I think we are we are pushed into that direction that engineering is more and more seen as a cost center instead of a value center, right? >> And I think that's a very dangerous thing to do for a lot of other things. um you know um even making this transition happening um as as as soon as you look with this perspective with this lens on your employees right like cost center I I think it's very it's very different uh and very difficult and we have to be very careful about that right often is if when you're measuring something there you know what you're measuring changes but not because you know the underlying work really changes but because of the measurement that you're doing right and there is a lot of literature on that a lot of science on that once I'm starting to probing exactly what's going on here and there right behavior changes and very often not in the in the right direction right so I'm careful about that you know obviously you can you know measure everything if it's purposeful if it really makes sense if the pro proxies really tell you something is you know it's a different story I think a very tricky story um but what is quite real is as long or as soon As you start measuring behavior changes, right? But not for the sake of you know improving really the processes but more to satisfy whatever metrics or you know goal you are actually putting towards that right >> yeah I feel like we can do like a full session on this I'll be >> done I have of anything to to add but I'll reframe myself let me uh thank you all um let me uh Eric let me follow up on something that um so we talk about like um standard productivity metric lyrics and so on. But what do you do you have any experience what's the most damaging form of AI immaturity if you like you have seen that doesn't show up in startup productivity metrics meaning you know like something is going right but it doesn't show in your metrics >> yeah that's I guess something I would like to call latent risk right it's sort of okay at the top line we're seeing some form of productivity but is is it the right metric that we're measuring against and with AI now coming into the picture Right? You could almost determine that maybe the value is not necessarily in the artifact such as you know pull requests or lines of code changed or time to merge but it's actually in the skill set of the developer. And what I mean by that is for example what would be problematic is if we're measuring by a total number of uh lines of code generated. Of course that's going to go up into the right with AI. AI spits out a lot of code. It's well it's well known to do that. But that's also increasing the cognitive load on the human who is tasked with reviewing the AI code, right? And so the higher skilled the developer technically the greater value that they have because they can course correct and review that code uh with better with better results. Right? So ultimately if your engineers can't explain the code that was shipped last week, then you know the productivity number that you have potentially is incorrect. Yeah, I have this spicy take on AI slop, which is it's what people call something when they know that they could have got the exact same output out of cloud and and it's not valuable to them, but if you can kind of train it to produce like like the people who I think are having tremendous value with AI have tuned it to um be an amplifier on their judgment, you know to Eric's point about skill and then then I think the other side of it is the people who have not tuned it which I think is understandable. We haven't been doing a great job of helping people really adapt and learn these skill these these new tools. It's just been this like bud and panic going on right like they're being forced to use them. they don't add any value on the output. But the output tends to be really long and that just wastes a lot of people's time and then it also impedes adoption, right? Because it makes both the person using the tool and the tool itself look a bit stupid and so then people who do have the judgment to say not ship something like that end up kind of increasing their resistance to it. Certainly and as well AI as you mentioned right can output pretty much anything but it doesn't necessarily answer the question are you building the right thing and are you building it well right and that's >> determining what is correct to build is very much a uh market and product design and orientation conversation by humans rather than than AI >> that's true >> yeah actually coming back to like a a good thing to measure like I one of the things I personally look at is how often And does our claw MD change? Because that is a measure of of like how much how many improvements are we making to the system we're using to build versus just what we're shipping on the system itself. >> Thank you. This is quite insightful. Let me let me move one step ahead then. Let let me So we talk a bit about like you thinking about evaluating where we are as an organization. That again is be a problem that you need to sit down and think hard but what are the things that we can do to progress it. So let's assume that you got this you now you have the right things set up to measure if you go in the right direction not what do we do and so code of view for me come to my mind and Eric as well mentioned it. Cor is one of the practices that most exposed by AI in a way as almost like have been described as the new bottleneck as if what does a health team practice look like now? Um and how is that different from years ago Mikuela? Um so is that um you know what do you see? >> Yeah, >> I'm super happy to take that question. um code review has been an interesting practice right I've been giving workshops on code reviews for many years now but now with AI somehow it's completely you know like it's completely the work the new workways that people are setting up are completely falling apart once they are hitting code reviews right um and I think one of the reasons is that we are confronted with larger PRs with more PRs right we can generate code in you know in seconds but we still have or need the same amount of time to build a mental model about that, right? To to verify it. Um and so at one point or another within this, you know, life cycle development life cycle, we have to or we are confronted with the realities of human capacity and human capabilities, right? So what can we actually um meaningfully review, understand, you know, give input in, judge, right? And um and I think this is where everything rattles a little bit and some people are very you know very afraid because their productivity gains that they you know have by by speeding up the code generation right like this task I think I don't know 14% of a developer's workday right now only takes like a fraction of that so you now you want to you know weave that through the development cycle and it you know the code review somehow makes a stop there and say no you know this takes as long as it took before maybe let's say Right. Um I don't think that it's necessarily a bad thing, right? The question is more what can we do with this time that we are saving for example on the code automation part to produce even better output. Right? Can we maybe we are still taking like this 14 you know 14% of your workday on the code generation part but we are we are reusing that so that the output that hits for example code reviews is already much better and I think I see already patterns emerge into that right so I see something when I look at code reviews we before talked about code reviews mainly when it's a peer-to-peer review right but I would say there's a first review cycle right that happens already between the deaf and the agent not a peer-to-peer not a human to human, right? So the developer that works with an agent collaborates with the agent. Maybe some people call it pair pairing, but I don't see it as a pairing because you're not there when it writes the code, right? It's not this pairing experience. It's more like well maybe you give it a plan or you know a prompt and it makes a plan and then it generates the code. This is an instance, right? Sort of. So you have the code there. It's actually a review, right? So you look through the code and then you know this is I would say that's the first review cycle and this should become a very strong cycle. And here we can use all the AI all the machine um power that we have that's not only geni that could be also static analysis right tests. So you have to and you have to really you know invest heavily in this verification um part right so to make sure that you can actually take the or make the most out of of AI right so this means that really upping your game in automated checks they should be in this first review cycle it's not you know you don't need any other humans for that it's the agent and it's uh the developer and they go through it ideally the agent itself is wired towards those automated IC checks maybe you know MCP server or other technology that you want to use for that right so the agent actually can trigger that has access sees what's happening can adjust on that right so you're already increasing the quality output and then the second layer is peer review but here I think the focus should change right so um away from this low-level inspection maybe not every generated code you know has to be looked at but a different a different focus right there are things that machines are weak at intent validity of approach architecture maintainability maybe something that's you know specific to your organization domain knowledge right could be also accident um prevention especially in high-risk areas right so I've been teaching risk based reviews for years now right where we really analyze the risk that's behind a code review and then dedicate more or less of our time on that and I think this becomes even more crucial right some organizations do that already automatic meter for example has some um risk core for example that you apply to output that's generated and so that drives you and I think maybe if I I know I'm talking a little bit long um I will try to keep it short at the other parts but um I think what's what's also interesting is that I see that the artifacts are maybe different right so I don't think it's only the code that we should look at maybe also the artifacts that the agent produces right so maybe artifacts that were produced in this deaf agent cycle right that could be again the prompt or the plan you know exposing to others there's the the the verification loop that you already did. What what did I already check here? Right? Um and then lastly, I think um one big trap that I see people fall into is this large PR trap, right? They think that because it's so easy to generate now, large PRs are acceptable, but you have to actually fight this, right? So large PRs means that even if we are completely agentics, let's say in the inner loop, right? You have to produce that input and that has to be high value, right? the specification and you have to verify the output, right? And so the larger that whole thing gets, the more it falls apart from a quality perspective, I think, right? So all in all all I would say that the healthy teams really have more engineering discipline, right? They need really more engineering discipline, not weaker one. Um, and this will amplify and this will also help with with code reviews. Yeah, >> thank you. >> Yeah. And that being said, right, I like to say that uh AI already amplifies the systems and structure of your software development life cycle that you already have in place, right? So AI will make sure that you are well aware of the quality of your CI/C pipeline, right? Are you testing the right things? Are you do you have the right checks in place to ensure that code quality downstream? Uh and I've even seen AI being used to help reviewers on that cognitive load, right? maybe sub agents that are spawned to look for very specific parts of the changes that were made whether it is uh you know testability architecture uh any duplicate code that was generated right and so that at least points the human reviewer in a specific direction of the code changes to make their review more efficient so I guess you could almost say that AI could be used to help with the uh expanded amounts of cognitive load that AI already generates. Thank you, Eric. While I have you here, you know, I have a question for you. Um, we talked about remote agents and autonomous workflow think out of reach. This was the premises of something that we said before. So, what investment do teams underestimate before they're ready for that leap? So you know we know that is something that every company probably should think about understand but is that what what what is in the trying to achieve that people miss >> yeah it's it's it's the platform itself and what I mean by that specifically the development environment in which the agent is going to run right that is that is the baseline that is the first step of the interloop process before any code is generated and before it makes it into git and ultimately CI/CD downstream into into production. Uh I I think over the last year, it's been nice to run agents locally. You just have one cloud code instance on your machine. But in an enterprise context, you're going to need to run tens or hundreds of agents in the background, and they're going to need access to that enterprise data and context that might not necessarily be available on that local machine. as well. That local machine is uh static from a compute perspective and it might require reconfiguration of packages and dependencies for that agent or a set of agents to be productive over the course of several different types of projects. So what that informs is these de these remote agents need a reproducible and ephemeral and policy controlled development environment. So what that looks like in practice is moving the environment off of that local machine and into some form of centralized development environment infrastructure. Ultimately, if there's a backlog of trying to get an environment stood up from a human perspective, you can only imagine what that backlog is going to look like with an agent trying to get access to an environment to do the work and complete the task. Additionally, think about the identity problem, right? agents need scoped and shortlived credentials and not borrowed human tokens that might live on the local machine and so you can deliver those types of agent first tokens in more of that centralized infrastructure. Additionally, when we talk about governance, these agents also need observability. So you need full audit trails of not only the changes that were made but all of the tool invocations and prompts that were sent to the agent. So, think of an LLM proxy or AI gateway that we're starting to see come to market, right? Light LLM has their own, Versell has their own coder as well as going to market with one. Ultimately, regardless of which vendor is chosen, you still will need some form of that gateway to implement that level of observability to meet governance standards for agentic coding. And then as well, you need these environments to be reproducible, right? So, agents are going to fail in a variety of different ways that are frankly non-deterministic. And if you're unable to reproduce that environment quickly, uh you won't be able to reproduce the failure. So what we see are customers now coming to us and other players in the space and saying we need to standardize our development environment and move it into uh the cloud where our source code and and data and other pipelines live. >> That's that's quite it's quite insightful. Can I can I add something completely different like a complete different perspective? I agree 100% of what Eric said, right? But um a very different very practical problem that I see coming up more and more and I maybe it will get bigger and we don't have a crystal ball right to see exactly what the market will do but is that agent that work like really autonomous right and we have all these checks that I also described right and so they're they're hooking and asking and questioning and you know redoing prompt after prompt. I mean this is generating so many tokens. what I see like I have teams already that are maxed out at the beginning of the week, right? Like they are like two day day two of their work week and they are like done with their budget, right? So I think I mean it's a very different take on on that question, right? But it's definitely something I think teams should um think about, right? Like maybe also local machines or local AI. It's not that powerful yet, right? It's a very different way of working. It's like back two years when everything was very painful. Hopefully this will also change right but I mean this is also a reality that we have to deal with right uh that tokens get more expensive you know there are additional planned increases you know in what actually I costs right and the things that we are talking about the things that I see um are needed that agent and agentic works really work smoothly this is like a to token eating machine right like >> this consumes a lot of tokens um and and it's Not it's not in reach for every company, right? Or for every developer. >> Yeah, that's true. Like the Oh, sorry. The the target >> the dark factory model, the target spend is $1,000 per developer a day. And this all comes from Silicon Valley, right? Where dev salaries are higher and like they're talking about the 996. And I think for most like normal companies, that's completely out of band. Um but also coming back to the other things that we've been talking about of like the guard rails and the training you know like there is there are lower cost models and I do think it is possible if you have built your guard rails and you have um you can move to a a way where for example your most expensive plan like like Claude is supervising one of these lower cost ones like Dan Shapiro wrote this blog post about how he built um some kind of video thing for his camera, but he used a lowerc cost model. He just used a tricycle, right? Which is something that does an intense like plan, build, test, validate. Um and so even though he used this lower cost model, he was able to use some he was able to build something very usable. And I'm very inspired by this. like I use the tricycle for for harder problems myself, but I'm also, you know, existing in one of those places where I don't think we're going to want to just kind of go up into the right on claw tokens forever. And so I'm like, all right, I'm going to view this, you know, whatever it is $100 plan as the limit. And when we start really hitting that consistently, I'm going to say how did this plan is going to become the supervisor and these lowerc cost things are going to become the implementers. >> Yeah. And we're seeing that in the market at large. I think Uber just publicly stated yesterday or the day before that they're not able to draw a straight line from token spend to actual business value. And I actually was with a a large oil and gas uh customer yesterday uh in their London offices and they said to me the era of cheap AI ended in Q1 2026, right? And that is informing their move to consolidate a variety of different AI tools. In the last year, I was speaking with a financial services customer. They were bringing in cloud code, codecs, and every coding agent on the market to really see what would work. And that's when the tokens themselves were subsidized by the model providers. And now we're starting to see that uh that flip of a switch and and the consolidation occur. >> Yeah. And I agree what Kate said and also Eric, but this takes time, right? like it's not something that comes out of the box similar to the AI harness that has to you know you have to take into consideration those considerations around cost don't come out of the box people have to you know like the the discussion definitely started already but I think there is more to do >> I think it also takes a level of expertise right like you know if you wanted if you like I'm going to get started with AI today but I don't want to pay subscription I'm going to use this like other thing well the UX is going to be worse you're not going to learned some things, you're probably going to have a really bad time. But I and and so I think you know this this panel called the high cost of AI immaturity like there is this portion of AI immaturity that is just expensive. And if you pick the wrong metrics and you don't build your expertise mindfully, then it's going to stay really expensive. And I'm kind of seeing this period as much more of like I need to learn as much as I can so that I can figure out how to do it in a cost-effective way soon. Um, I spent months on the $20 claw plan, you know, and my partner was trolling me, my collaborator was trolling me, and I was like, "No, I need to understand what everything costs." And eventually, I lost patience. I up my plan. I went on an absolute tear. But I really value that time because it really taught me like, "What do I need to do what?" Like, if I have to actually budget with this token spend, like what's really important, what's not, what's too expensive to be worth it. And I think this metric of like just like token spend as a vanity metric I think is stupid one because it doesn't correlate to output. But I think it's kind of um second order stupid because it doesn't actually teach us how to be effective with this thing that is ultimately going to be more expensive than it is today. >> Yeah. >> Maybe I have to add something. Sorry. No, Simon, go ahead. No, it's mostly like I would really love before we finish to talk a bit more about way of working and you know the other aspect that we haven't touched on. So if it's okay Mela is okay if I if I move. Yes. >> Yeah. >> So I think what we haven't touched a lot on is what this transformation look like from a human perspective and quite a few of you think hard about this problem. So I want to make sure that he has enough space. So I'll give you an example for me. For example, social proof is very powerful. I had a team that kind of like um you know it went from um you know kind of like sparotic to routine usage of AI um and it did it because it did like a sort of like a workshop together and then they told every other teams that they started using in a different way. So for me social process is very important. So I'm wondering from you what is the lightest touching version you ever seen unlock the most behavioral change? So because again this is behavior. So Kate looking at you for this one to start with. >> Um first on a team level I think anything that gets implemented for the good of the team and in a way where it runs automatically. So at my last job I had my teams running um AI code review from like the start of 2025 and it was like we tried it and it actually found something on a god level developer and the team was like okay if they have value to this guy it has value to the rest of us. And so we adopted that early. Um and it and it was very useful. And then you know anytime I think the people that I noticed most kind of working with early on, this is what they were using it for. They were like taking problems that would benefit the team but nobody was bothering to do it and they were using AI to move those forward on the side and then they were checking it in. So there was none of this preciousness about oh this code is bad because I think everybody knew they weren't going to write it themselves, you know. And so I think that like and and this is what I would encourage any manager to do if that you're thinking about how to drive AI adoption on your team is to be like what things would benefit the team and show the team that it has value on an individual level. I think the biggest unlock is mindset, right? Many individual developers right now are feeling this identity threat, this skill threat, this stress of like what does this mean for my job, my job security, the industry and mass layoffs, top down mandates, you know, the the kind of incident level, the reliability issues like these are not helping. And this is the work that Gene and I do on on DOI your career. We have a navigating the eye shift course and module one is just like what is your mindset about this and what do you want your mindset to be? You know, how can you find um a sense of confidence in your skills and yourself that you can navigate this mindset? And so every time I review one of these documents, I see somebody's mindset shift. And I think this is really underdone work. And I think a lot of the problem that like let's like like some organizations are having is because they have tried they've used all the sticks and they're like why are the sticks not working but this person is there having a huge like identity challenge you know and does not have a safe space to work through it. >> Thank you. Thank you. Then on on on the same sort of like on the same area maybe we hear more that to be effective we need to rethink our way of working from the ground up. I mean at least that this is what they're telling me. It's kind of like well if we just rem you know do what we used to do just with AI it's not going to work. Um but this is hard from a social perspective. So I'm wondering it is there an alternative? I mean or this is the only way is to really like you know ground zero and restart from scratch. Mika. >> Yeah. Well, change is hard, right? Like it's it's hard on humans. We know it. Um but I think that the current environment and Kate already said that a little bit. It's stick approaches, right? Carrot and stick approaches, they make it especially hard, right? We see um a lot of media coverage and also CEOs, vocal CEOs and CTOs or whatnot, right? Talking about how AI can do it all, right? Like we can get rid of you. We see this layoffs and so on, right? So we are not only changing in a very positive environment. I think a lot of the changes are happening in a very frightful environment be it true or not right like but you know I think as an organization the first thing that I would do as an organization um as a team right think about you know the foundation that we have and that have to be our values right so our values should actually not change and hopefully the values are around the humans right and that we value the human and the work that they're doing we are valuing and honoring the human experience right and I think this is probably the first thing that I would work on and not the individual that has you know to change their perspectives on on it but really the organization the team should value the developers and and the work that they're doing and I think this is the foundation for people to actually be able to change in a good way and here I'm again saying not you know um changing so that they you know do whatever metric you know target you're setting so that they're reaching that but that they find out that they experiment they talk about you know and shifting their ways and here you said like from the ground up. Yeah, there are drastic changes but we don't have to do that from the ground up and not immediately right. We can have deliberate and evolve gradually right and really take the humans um on the way right on the journey. This is what I do with the with the workshops, right? Whenever I do a workshop, one of the goals is not only learning but also behavioral change, right? And what I have seen over the years is that it only works when you know when you can take the people with on with you on the journey, right? So that you really listen to their friction points, right? To their problems, maybe why they are skeptic about it, right? If I just say, "Oh, you have to use it more." That's not going to happen. Or you do it in that way, right? Very often people not only resist change because out of you know um no reason but they have reasons right they have their way of working and if you solve it with them together right this can really bond strongly that's what I what I probably would do yeah >> well and that shows the the importance of the qualitative aspect of that developer experience uh survey that you mentioned right via DevX or other players in the market right is sure you can have a quantitative number as it relates to developer productivity and experience. But uh that qualitative measurement will really help you understand what those behavioral changes are that are needed. >> Yeah. And I mean I find it because you said Uber, right? I found it funny that first it was like oh we can get rid of all of you. not Uber now but in general like the message came through from some players right and it's like oh maybe it didn't work out that well maybe we are slightly rehiring but that's not nice right like when I work with with teams for example in my workshops the best teams right the best teams are the one that have a very positive strong work culture right that that love their company that treats them well um that have often a lot of freedom but they are the ones that are picking up on ideas right they they they take that learning they you put it into practice. They have discussions with each other, right? This is I mean this is really what um psychological safety also looks like, right? And we know that this is so foundational for everything that we have, right? So yeah. >> Yeah. I mean if you wrote a playbook of like how not to do change management, it would look like a lot of the AI shift like um if the CIA was like, "Oh, we're going to take down this industry." Like this is the playbook they would write. Um, and I think, you know, but but kind of coming back to the main question, one of the things I was reading yesterday is that there's like a U curve in terms of small organizations being able to see gains, large organizations being able to see gains, and the medium-sized ones are having the hardest time. And my take as to why is because of workflow homogeneity, right? And so small teams tend to have homogeneous workflow because they work together and they have this sense of sharing and community and psychological safety and they say this is what's working for me, you try it and their bar for experimentation tends to be lower, right? And large companies tend to have homogeneous workflows just through necessity, right? You got to make that many people effective. You're going to pretty much mandate the way they work. And medium-sized companies have this wild west of like everyone gets to do their own thing. Some things are mandated, some things are not. And like I think that hetrogenity maybe helped them in the before times but now is really holding them back because I do see in my work that AI adoption benefits from homogeneous workflows. Well, Kate, you'd be surprised to learn as well that even the largest enterprises name brands uh that we would know uh still have that heterogeneous uh developer experience and they come to centralize uh to us and other players to centralize and say, "Hey, I want to standardize developer experience across full-time employees and uh third party contractors as well." Um so, it's very much a movement that we're seeing in the market at large to to get that standardization done. And I want to add here something because what I really find important is that on one hand I think consolidation and you know standardization and centralizing things is very important. I worked at Microsoft and our team was responsible for exactly that right but I think it's also so important to leave leeway to the individual right to the person and their needs and how their brain maybe works how their brain wiring is right their habits um and not completely know dictate exactly how you have to do things right so you have to have very and a lot of even large corporations have to do still a lot of work to get their systems right consolidated the technologology solid consolidated, right? But don't like get rigid on the work, you know, the ways people can actually work and interact with tools because there's also research, for example, on neurodeiversity and a lot of engineers are actually neurode divergent, right? That they just work with the IDE for example differently, right? That they have different ways of processing information. So I think if we are getting too prescriptive of how that should actually work for every individual that's also a dangerous dangerous road to go down. >> I mean I think this is where small teams have the edge is that they can have both the homogenity and the experimentation and this is why AI adoption on any kind of larger company becomes a much much harder problem and I think it is really clear today like if you are building an organization from the ground up you would do it in a very different way in this era than you would have done it 5 years ago. But if you have an organization that you built 5 years ago or however long it was, how do you retrofit it onto it given that you know nobody really knows exactly how it's going to look like yet? We're still in the throws of this change. And I do think that that is genuinely a hard problem. And then there's, you know, there's a moral question of how are you going to come at it, right? So to Michaela's point, are you going to prioritize like psychological safety and the things that actually support change management or to my earlier point, right? Are you going to kind of come with the financial statement as the big stick, right? And I think engineers should understand the financial statement, right? But it doesn't mean that I morally agree with that approach. >> Yeah. And I think I need to wrap up. I'm afraid I I wish we had an hour and a half because we only scratched the surface. probably there are there are quite a few fast question that we would go in now and respond to slack but in general like to summarize it I think we realize that this is not an easy problem to solve however there are a couple of things that are emerging one that probably like tooling is a key part of the puzzle but also the way we handle this that is a transformation or a change is equally important and you know probably like not overdoing your metrics as targets is also like a third piece um thank you so for joining. So we will now go in the in the Slack channel respond to all your question and thank you very much for you all to joining this panel. I really appreciate that. >> Was a pleasure. Thanks. Byebye. >> Cheers. >> Bye., The High Cost of AI Immaturity, YouTube, 01kv3sv8d5d16997b18yz6a67n.