Measuring the ai impact on developer experience 🔗 **Source:** [Measuring the ai impact on developer experience](https://www.youtube.com/watch?v=bQqwiuI1XGE) > [!note]+ Document Notes > tags: [developer-experience, ai-tooling, engineering-metrics] > author: Lead Dev > source: Lead Dev > reference: https://www.youtube.com/watch?v=bQqwiuI1XGE > date created: 2025-01-03 16:30 > > # 🏷️ Title: Measuring the AI Impact on Developer Experience > <!-- deep-read:sections start --> ## 🧰 Action Inbox - [H1] Define 3-5 core metrics for AI adoption success — include velocity, quality, and sentiment #metrics - [H2] Establish baseline measurements before AI implementation — capture pre-AI performance data #measurement - [H3] Create communities of practice around AI tool usage — share guardrails and best practices #community ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [DORA 2025 State of AI-Assisted Software Development](https://cloud.google.com/devops/state-of-devops/) - [GitKraken Developer Experience Platform](https://www.gitkraken.com/) ## 📝 Summary - A panel discussion exploring how engineering teams measure AI's impact on developer productivity and experience - Key participants include DORA research leader Nathan Harvey, GitKraken VP Jeremy Castile, and developer metrics expert Bill Harding - Main focus on establishing meaningful metrics beyond adoption rates, including velocity, code quality, and developer sentiment - Emphasis on having clear communication about AI expectations and creating baseline measurements - Discussion covers practical approaches to ROI measurement and avoiding vanity metrics ## 📖 Main Content The panel addresses the challenge of measuring AI's real impact on developer productivity beyond simple adoption metrics. The experts emphasize that while telemetry data (lines of code generated, acceptance rates) shows adoption, it doesn't necessarily indicate improved productivity. Key themes include: **Holistic Measurement Approach**: Teams need a constellation of metrics covering velocity (story points completed, PRs closed), quality (code duplication, review time), and developer experience (friction, burnout). No single metric tells the complete story. **Clear Communication and Mandates**: Organizations with explicit AI adoption stances see amplified benefits across individual effectiveness, organizational performance, and reduced friction. The mandate doesn't need to define specific productivity measures but should set aspirational goals. **Baseline Establishment**: Teams must capture pre-AI performance data to understand actual impact. However, attributing changes solely to AI is challenging in complex organizational environments with multiple variables. **Communities of Practice**: Successful organizations build learning communities around AI practices, sharing guardrails, prompt engineering techniques, and effective usage patterns rather than just focusing on tool selection. ## 🗒️ Notes #### ⏱️ [00:00](https://www.youtube.com/watch?v=bQqwiuI1XGE?t=0) — [H1] Introduction and Panel Setup - Key idea: Discussion on measuring AI impact on developer experience with industry experts - Evidence: "We're going to have a discussion called measuring the AI impact on developer experience" #### ⏱️ [05:30](https://www.youtube.com/watch?v=bQqwiuI1XGE?t=330) — [H2] Getting Feedback from Engineers on AI Tools - Key idea: Leaders get feedback through telemetry and direct communication, but telemetry focuses on adoption not productivity - Evidence: "Telemetry data is focused on things like adoption... it's not necessarily a measure of productivity" #### ⏱️ [08:45](https://www.youtube.com/watch?v=bQqwiuI1XGE?t=525) — [H2] AI Mandate Patterns in Organizations - Key idea: Both top-down mandates and bottom-up adoption are happening, with engineering leaders caught in the middle - Evidence: "We have absolutely seen this both in terms of the tops down mandates... but we also see bottoms up developer adoption" #### ⏱️ [11:20](https://www.youtube.com/watch?v=bQqwiuI1XGE?t=680) — [H1] The Importance of Clear AI Stance - Key idea: Organizations with clear, communicated AI expectations see amplified benefits across multiple performance dimensions - Evidence: "When you have a clear and communicated stance... AI's positive influence on individual effectiveness gets amplified" #### ⏱️ [15:15](https://www.youtube.com/watch?v=bQqwiuI1XGE?t=915) — [H2] Core Metrics Framework for AI Success - Key idea: Teams need 3-5 agreed-upon metrics covering velocity, code quality, and developer experience - Evidence: "At a minimum you want to agree on a core set of metrics... velocity component... code quality component... developer sentiment component" #### ⏱️ [18:30](https://www.youtube.com/watch?v=bQqwiuI1XGE?t=1110) — [H2] Specific Metrics Vary by Context - Key idea: Specific measurements depend on product type, deployment frequency, and organizational role - Evidence: "A web app that deploys every day... is going to have much different priorities than a critical infrastructure app" #### ⏱️ [22:10](https://www.youtube.com/watch?v=bQqwiuI1XGE?t=1330) — [H3] Starting Simple with AI Measurement - Key idea: Teams shouldn't feel behind if just getting started; beginning with simple metrics is acceptable - Evidence: "Don't feel like you're way behind... just start... Keep it simple. And it's okay if you are just getting started" #### ⏱️ [24:45](https://www.youtube.com/watch?v=bQqwiuI1XGE?t=1485) — [H2] Avoiding Meaningless Metrics - Key idea: All metrics are flawed proxies; focus on metrics that drive behavioral change - Evidence: "Every metric is flawed... if you can't answer what behavior would we change if that number doubled... that's probably not a good one" #### ⏱️ [27:20](https://www.youtube.com/watch?v=bQqwiuI1XGE?t=1640) — [H3] Key Quality Indicators for AI Teams - Key idea: Time spent reviewing code and release cadence are particularly indicative metrics for AI-enabled teams - Evidence: "The amount of time spent reviewing code... if you're writing twice as much code but spending four times more hours reviewing" #### ⏱️ [30:15](https://www.youtube.com/watch?v=bQqwiuI1XGE?t=1815) — [H2] Qualitative Metrics: Friction and Burnout - Key idea: DORA tracks friction (undefined but recognizable) and burnout as key qualitative developer experience measures - Evidence: "Friction and burnout... You know friction when you see it... are you cynical? Do you have cynicism while you're at work?" #### ⏱️ [32:40](https://www.youtube.com/watch?v=bQqwiuI1XGE?t=1960) — [H2] Constellation of Metrics Approach - Key idea: Multiple metrics needed like a car dashboard; no single metric provides complete picture - Evidence: "I think of it like a car dashboard... We wouldn't just look at the speedometer" #### ⏱️ [35:20](https://www.youtube.com/watch?v=bQqwiuI1XGE?t=2120) — [H2] AI Beyond Code Generation - Key idea: AI tools expanding across full SDLC from planning to deployment, including management reporting - Evidence: "AI tools for engineering leaders... getting visibility into all these things using AI to help us summarize" #### ⏱️ [38:15](https://www.youtube.com/watch?v=bQqwiuI1XGE?t=2295) — [H1] ROI Justification Through Experimentation - Key idea: Give teams flexibility to experiment with different tools while measuring agreed-upon metrics for comparison - Evidence: "Give teams the flexibility to make their own decisions... see on a team by team basis how outcomes are changing" #### ⏱️ [42:30](https://www.youtube.com/watch?v=bQqwiuI1XGE?t=2550) — [H2] Tools vs. Practices Distinction - Key idea: Success comes from practices built around tools, not the tools themselves; focus on communities of practice - Evidence: "It isn't necessarily the tool... but rather the practices... building communities of practice and communities of learning" #### ⏱️ [44:10](https://www.youtube.com/watch?v=bQqwiuI1XGE?t=2650) — [H3] Business Value Translation - Key idea: Simple value calculators help translate developer productivity gains into business outcomes - Evidence: "Maybe we're paying 40 or 50 bucks per developer... but we are seeing $2,000 per developer in productivity gains monthly" ## 📚 References - [DORA State of DevOps Reports](https://cloud.google.com/devops/state-of-devops/) - [GitKraken Insights Platform](https://www.gitkraken.com/insights) - [Lead Dev Community Slack](https://leaddev.com/community) ## 🗣️ Paraphrase - Engineering leaders need to look beyond AI adoption metrics to understand real productivity impact through combinations of velocity, quality, and developer experience measurements - Organizations that clearly communicate their AI expectations and goals see better results across multiple performance dimensions compared to those without explicit stances - Success with AI tools depends more on the practices and communities built around them than on the specific tools chosen, requiring experimentation and knowledge sharing ## 📰 Newsletter **Measuring AI's Real Impact: Beyond Adoption Metrics** The latest Lead Dev panel reveals a crucial insight: measuring AI tool adoption isn't the same as measuring productivity improvement. Industry experts from DORA, GitKraken, and GitClear shared practical approaches for understanding whether your AI investments are actually paying off. **Key Takeaway**: Establish 3-5 core metrics covering velocity (story points completed), quality (code review time), and developer sentiment before rolling out AI tools. This baseline becomes essential for attribution later. **Practical Application**: Start with simple measurements and build communities of practice around AI usage. The specific tools matter less than the practices your teams develop around them. **Community Connection**: Join the ongoing discussion in the Lead Dev Slack Panel Q&A channel for more specific metric recommendations and implementation strategies. Resources: [DORA Research](https://cloud.google.com/devops/state-of-devops/), [Developer Experience Platform](https://www.gitkraken.com/), [Lead Dev Community](https://leaddev.com/community) ## ✅ Action Items #### DevRel - Developer advocacy: Create AI tool comparison presentations showing real metrics, not just features - Education: Design workshops on establishing baseline measurements before AI adoption - Developer experience: Document common friction points during AI tool rollouts and solutions - Developer marketing: Develop case studies showing ROI calculations for AI tool investments - Community management: Facilitate communities of practice around AI tool usage patterns #### OSPO - Policy & Governance: Develop guidelines for AI tool evaluation and adoption processes - License Compliance: Review AI-generated code licensing implications and scanning approaches - Security Management: Establish review processes for AI-generated code security - Software Inventory: Track AI tools and their usage across development teams - Developer Support: Create training materials on effective AI tool usage patterns - Community Engagement: Share learnings about AI impact measurement with upstream communities - Strategic Alignment: Establish KPIs that connect AI adoption to business outcomes #### Community - Maintainers: Provide templates for measuring AI impact on open source project velocity - Contributors: Create onboarding materials for contributors using AI tools - Sponsors: Develop transparent reporting on how AI tools affect sponsored development work - Ecosystem: Coordinate cross-project standards for AI tool impact measurement #### WordPress - Core: Establish baseline performance metrics before broader AI tool adoption - Core Performance: Monitor AI-generated code impact on WordPress performance benchmarks - Core AI: Develop evaluation frameworks for AI assistance in WordPress development - Community: Create resources for local meetups to discuss AI tool experiences - Training: Develop curriculum on measuring developer productivity with AI tools - Documentation: Document best practices for AI-assisted WordPress development #### The WPCC - Programs: Fund research into AI tool impact measurement in open source contexts - Projects: Support development of shared metrics frameworks for WordPress ecosystem - Funding: Establish criteria for evaluating AI tool ROI in grant applications ## 🧪 Hypotheses - Teams with clear AI adoption mandates will show measurably better outcomes than teams without explicit guidance - Developer sentiment metrics will correlate more strongly with long-term productivity than code generation metrics - Organizations measuring baseline performance before AI adoption will have better ROI attribution than those starting measurement afterward - Communities of practice around AI tools will produce better results than individual team experimentation ### 📊 Methodology - Survey teams with and without clear AI mandates across similar productivity metrics - Longitudinal study tracking developer sentiment alongside traditional velocity metrics - Compare attribution accuracy between teams with pre-AI baselines versus post-implementation measurement - A/B test isolated team learning versus cross-team communities of practice ### ❓ Questions - How long does it take for AI tool adoption to show measurable productivity improvements beyond the initial learning curve? - What's the optimal frequency for surveying developer sentiment about AI tools without creating survey fatigue? - Which specific code quality metrics best predict long-term maintainability when using AI-generated code? ### 🔮 Future Research - Long-term impact studies of AI tool adoption on developer skill development and career growth - Cross-industry comparison of AI impact measurement approaches and their effectiveness - Investigation of AI tool impact on different developer experience levels and specializations ### 💡 Implications - Practice: Teams need to invest in measurement infrastructure before, not after, AI tool rollouts - Policy: Organizations should develop explicit AI adoption stances and communication strategies - Security: Code review processes must adapt to handle increased volume from AI-generated code - Education: Developer training should include effective AI tool usage patterns, not just tool features - Funding: AI tool ROI calculations should include developer experience costs, not just subscription fees ### 🧠 Learning Styles Assessment Reading level estimate: 12th grade #### 👀 Visual learners - Present: Panel format with multiple speakers provides visual variety - Improve: Add dashboard mockups, metric visualization examples, before/after charts #### 🔊 Aural learners - Present: Full audio discussion with natural conversation flow - Improve: Provide chapter markers for specific metric discussions #### 🗣️ Verbal learners - Present: Rich discussion with definitions, examples, and industry terminology - Improve: Glossary of key terms like "DORA metrics," "developer friction," "constellation of metrics" #### 🧩 Kinesthetic learners - Present: References to hands-on experimentation and tool testing - Improve: Provide downloadable metric tracking templates and calculation worksheets #### 🧠 Logical learners - Present: Structured approach to metric selection and ROI calculation - Improve: Decision trees for metric selection, cause-and-effect diagrams for AI impact #### 🧑‍🤝‍🧑 Social learners - Present: Panel discussion format encourages collaborative learning - Improve: Structured discussion guides for team metric selection meetings #### 🧘‍♀️ Solitary learners - Present: Individual reflection opportunities through metric selection exercises - Improve: Self-assessment questionnaires for current AI tool effectiveness #### 🌿 Naturalistic learners - Present: Ecosystem thinking about tool interactions and organizational health - Improve: Analogies to natural systems for understanding metric relationships ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Video has clear speaker identification and structured discussion - Improve: Provide detailed transcript with speaker labels and topic timestamps #### Keyboard navigation - Present: Standard YouTube player controls available - Improve: Ensure any supplementary materials support keyboard navigation #### Contrast and readability - Present: Video player meets standard contrast requirements - Improve: Provide high-contrast transcript formatting option #### Text resizing and re <!-- deep-read:sections end --> ## 📺 Video  --- # 📄 Full Transcript *Auto-generated from YouTube captions · timestamps roughly every 30 seconds · [watch on YouTube](https://www.youtube.com/watch?v=bQqwiuI1XGE)* **[00:08]** Hi everyone, my name is Najla and huge thank you for joining us today. We're going to have a discussion called measuring the imp the AI impact on developer experience and this is a lead dev online panel discussion created in partnership with git kraken. Today's discussion will last around 45 minutes after which both the panelists and myself will head over to the lead dev Slack to answer some of your questions in the panel Q&A channel. We may have time to answer a couple of **[00:39]** questions live. So, please do submit those through the Q&A feature on Zoom. I'll be monitoring it and we'll get to them if we can towards the end. Otherwise, we'll um definitely collect those and talk about them in Slack. So, first let's get started with some introductions. My name is Najla al- Mashu like I mentioned earlier and today I'll be joined by our amazing panelists to share their experiences and insights. So, I'll ask you each to introduce yourselves. Just starting with myself. I am uh currently a freelancer living in **[01:12]** Berlin, Germany. I'm uh a director of engineering at Align where we work with startups to help them uh grow their their businesses. I'm also a coach um a leadership coach working independently here um with clients all over the tech space. So let's start with Nathan. >> Right. Hi. Thank you so much. It's great to see everyone. Uh my name is Nathan Harvey. I lead up the Dora program here with Google Cloud. If you're unfamiliar **[01:43]** with Dora, the quick like 30 second thing on it is is it is research into how do technology driven teams and technology organizations thrive. Dora wants to help teams get better at getting better. >> Great. Thank you, Nathan. >> Of course, >> Jeremy, tell me about you. >> Awesome. Hey, everyone. I'm Jeremy Castile. So, I'm VP of developer research here at Git Kraken. Git Kraken is a developer experience platform. So we help developers be more productive. **[02:14]** We help engineering leaders get visibility and things into things like velocity, code quality, AI impact, which we're going to be chatting about today. I'm super excited about. Um so I've been at Get Kraken for the last three years. Um and before that I led teams at Docker and New Relic. So I've been in the DevOps and dev tooling space for quite some time. Excited for the conversation today. Thank you. And last but not least, Bill. >> Hey everybody. I am Bill Harding. I've **[02:45]** been programming since uh I was a teenager, which uh was a while ago now. I uh in the last five years have been uh really passionate about uh instrumenting uh developer metrics, helping teams understand what it is they're working on and in particular understanding how the new tools that are coming available with uh these various AI agents are measurably changing the outcomes that teams observe. So we've been really active in publishing research over the **[03:16]** last few years. in particular our uh AI code quality research which has been downloaded many many uh times and yeah we're just continuing to use all of our data to inform a better understanding of how teams can get the most from their AI tactics. >> Great. Thank you. I'm so excited to do this panel with E3 specifically. Most of us in tech have been given these mandates that we must adopt AI internally in order to boost engineering **[03:49]** proficiency and productivity. And now we're living with the consequences of all of the decisions that we made. Some are good, some are bad. Um teams have taken very different approaches, but all of us are learning in real time, which I think just makes it really good to to talk through. So, in this panel, I really want to talk about what's working, what isn't, and how we can evolve together a bit more intentionally. And I want to start with Nathan. Um, as engineering leaders, how are we getting **[04:20]** real feedback from the engineers that are using the AI tools day-to-day? >> Yeah, this is a really interesting question. Thanks, Naja. Um, so one of the things that Dora has done this year is we published the 2025 state of AI assisted software development. So I'll be sharing some of the findings that we have there. When it comes to how engineers are or sorry how leaders are getting feedback from engineers, I think there's a couple of different ways that leaders are getting that feedback. First and probably most uh most uh most used **[04:52]** is sort of telemetry data out of these systems. One of the challenges though with that telemetry data is a lot of that telemetry data is focused on things like adoption. For example, if you use an AI agent or an AI assistant or AI assistant and it's generating code for you, a leader might ask, well, you know, how many lines of code were generated by AI or what is our code acceptance rate? Uh, which is a measure of, you know, the AI makes an a suggestion and the engineer makes a choice to accept that **[05:22]** code like how often does that happen? These are good metrics, but it's really important to remember that what these are signaling is adoption. Adoption of the tools. It's not necessarily a measure of productivity. Am I a more productive engineer because I accepted more lines from AI? I don't know. Uh I think we have to think about different sort of measures of productivity. Um and maybe we'll talk about some of those as we go through today. But telemetry is just one way I think that leaders are getting these signals. The other way that leaders can get signals from **[05:53]** developers on how these tools are going and how they're changing the way that people are working is through things like surveys or even just talking to engineers. It's weird. You can learn a lot by just having a conversation with somebody. And I think these things are really important because uh a lot can get missed or or is invisible when we look only at system data. So I think it's always a good idea to mix sort of uh experience reports whether those are through surveys or through going and observing the work or just talking to **[06:24]** engineers together with telemetry that we get out of systems. >> Yeah, that's excellent. And I invite anybody else to jump in if they have a thought. Otherwise, Jeremy, because of your role, I would really love to understand a bit more about how you're seeing AI mandates play out actually on the ground. Um, especially given what Nathan just said about getting signal. Uh, what patterns are you seeing and how **[06:54]** organizations are approaching this moment? >> Yeah, absolutely. So we have spent the last year talking to probably hundreds of engineering teams and so you know obviously there's a spectrum of you know how teams are rolling this out but we have absolutely seen this both in terms of kind of the top tops down mandates um where engineering leaders are getting these mandates to roll out AI across their teams for sure but we also see like bottoms up developer adoption right and I think the engineering leaders are **[07:26]** kind of stuck in that middle area of like man this stuff is coming in fast um and trying to figure out you know how do we how do we make sense of all this um and so yeah absolutely I think to Nathan's point teams are trying to figure out like what are all the different layers of kind of the different types of measurement that we need and what does productivity actually mean right it's you know you know maybe five or 10 years ago you know we used to talk about lines of code it's just that doesn't work in the kind of this new AI era that we're in and so you know **[07:57]** obviously we're going to talk about this, you know, today in this webinar, but developer experience is huge. Uh, getting metrics out of these systems and really being able to kind of like tie all of that together right now is the huge task that's in front of a lot of these engineering teams. >> Yeah, >> I would also say I would also say from a mandate perspective, I'm seeing a lot of really interesting things happening. Um, one organization I was talking with, um, they're a relatively small organization, um, but they're, uh, very technology **[08:29]** forward, very much leaning into AI. And their leadership came out and said, look, the as you're using the AI, it should make you twice as productive. Um, and it very intentionally that leadership did not define what does twice as productive mean? Um, how are you going to get there? They kind of left that to the individual. And really it was kind of this idea that we wanted to sort of inspire everyone to think like look maybe this is helping me a little bit but if it's not if if I don't feel twice as productive I should probably lean in more and learn more **[09:01]** from how to use these tools. And I think that's really really important. Of course, that organization in maybe not a surprise certainly not a surprise to probably this panel there were people in that organization who said uh oh what if I don't hit that 2x uh are there going to be negative consequences for me and so that leadership was very quick to sort of clarify and say no look this is this is we're sort of setting this as an aspirational thing and to give you a good signal have you leaned in enough or not yet and then from a a research **[09:31]** perspective one of the things that we published as part of this year's report with Dora is what we call the Dora AI capabilities model because what we're seeing is that uh everyone is using AI. Everyone in our survey and that we're talking to is using AI but we're seeing high levels of variance in terms of team performance across different organizations. So we wanted to understand what conditions and capabilities really help the adoption of AI amplify uh the impacts of that AI. And one of those things was a clear and **[10:02]** communicated stance. And what we found and and so this is this goes right back to that mandate. Like having a mandate is not a bad thing. When you have a clear and communicated stance on what's what is expected in terms of AI adoption, we see that AI's positive influence on individual effectiveness gets amplified. We see that organizational performance gets amplified. We see that um the neutral effect that AI is having on friction within an organization starts to decrease. So we're decreasing friction **[10:33]** when we have that clear and communicated stance. And we also see that AI's positive influence on software delivery performance also gets amplified. So it's really important to have that clear and communicated stance. It's one of probably the the the earliest foundational things that an organization should look at. Yeah, it seems like the an essential part of getting a team to really maximize its AI potential is setting those goals or choosing those metrics **[11:03]** that that Nathan's alluding to and then making sure that the team understands that measuring this experience they're having as they use it, measuring their sentiment on how well this is working for them. that affirms for the developers that this isn't just about uh getting tools that are going to replace the developers. It's about creating the right combination of tools that are available to developers to help them especially on tasks that they **[11:34]** didn't want to do anyways. And when the developers feel that the manager is participating in iterating towards a AI future that maximizes the well-being of the developers themselves, it's a lot easier to make that mandate into something that actually ends up having a positive outcome because the developers are going to be invested if they feel like you are measuring and understanding how they feel as they try to use the **[12:05]** tools that are available. to them. >> So, Bill, if you can expand a little bit, what specifically are you measuring like on your teams and how, if at all, are you tying that all of that back to business impact just to like kind of button up the whole picture? >> Yeah. Yeah. Well, I I I think that if there was just one takeaway that I would hope people will would leave this webinar with, it is to have a conversation on the team about what are **[12:36]** the three to five metrics that you think for your circumstances define the desirable outcomes that you want to happen as you adopt AI. And I think that productivity based metrics are going to be a subset of the three to five metrics that you choose. But there should also be and it doesn't have to just be five. It can certainly and I think often is more. But I think at a minimum you want to agree on a core set of metrics. And I think that those core set of metrics are ought to have a velocity component. They ought to have a **[13:07]** code quality component. And they absolutely need to have a developer experience or a developer sentiment component so that you have a holistic understanding of not only are these tools helping us get more done in the way that we collaborative we collectively recognize as progress getting things done but are they also not sabotaging our long-term health which a lot of code quality metrics can get into. and are they making the developers happier to uh come to their **[13:40]** job every day? And so I think that the specific uh measurements that get used for that are highly variable because it depends both on the um nature of the product that's being built. A web app that deploys every day uh and where a critical defect often gets fixed in a couple hours is going to have much different priorities than a critical infrastructure app that maybe releases every month and uh if it has a critical defect that's you know stop the **[14:12]** production line we all have to come in and pay attention to this. So I think there uh is a lot of variety on that front. I think there's also a lot of variety depending on the role of who is spearheading the effort to let the teams get the most value out of their AI tooling. I think that a CTO is going to be interested naturally in different metrics than a lead developer or an individual contributor. But I think **[14:43]** that's all the more reason that having the conversation with all these people who can start by agreeing, okay, these are the three to five things that we all agree, we can use as a point of truth when we're comparing what's almost certainly going to be a variety of different approaches that get taken across many teams. If you're working in a big company, uh you're going to have every team as sort of a natural petri dish of different tools that they're trying and different idees that they're **[15:13]** trying. And to be able to use those different teams as data to figure out what the ideal combination of AI and tools are. That is why uh it's very valuable to have those agreed upon metrics so you can figure out who is having the best results and how can we emulate that across the organization. How can we give developers as much information as possible about what has been working for their teammates. Um, **[15:45]** and so I think some of the specific metrics uh are things like story points completed, uh, diff delta on commits, poll requests closed. Uh, there's there's several others. I might list some more in uh, some of the later questions. Uh, but there is a wide variety that we see uh, customers come to us with on GitClar, which has led us to build about 70 different metrics at this point. And when a new customer signs up, it can be a little bit overwhelming almost. I feel bad for **[16:16]** them, but it's a really a product of just how diverse the different circumstances are that customers are starting from that you need a lot of different options. And you want to allow the teams to understand these options as well as possible. pick the ones that apply to them best and uh let their teams experiment and then share what is working best among those experiments. >> Yeah. And one thing I would quickly add **[16:47]** to that as well is like you know at least from you know our experience out chatting with teams like uh you know don't you know I would encourage teams to feel like if you're just getting started on this that's okay like you don't have to feel like you're way behind. And so, um, there are so many, you know, changing things, new tools popping up all the time. Um, and so, you know, just start, I think, is probably the the key takeaway there. And it's okay if you are just getting started on some of the stuff. And there's great resources out there. You know, Nathan **[17:18]** mentioned, right, like the the Dora research, there's tremendous value in that report. Go check it out. Um, but yeah, the important thing is, you know, just get started. Keep it simple. And it's okay if you if you are just getting started. I want to take a question from the audience that is really relevant here. Um, how can we avoid setting up metrics that aren't really going to be meaningful? >> Oh, this is a fun one, I think. Uh, and it's a it's a it's a really great question. So, thanks Beth for asking **[17:49]** that. Um, I have some bad news. You can't like you can't you can't fully like I can't give you advice that will help you 100% avoid setting up metrics that aren't going to provide value for you or aren't going to give you meaningful insights. Um just the fact that you're asking the question gives gives good hope like you're probably on the right path already if you're already thinking about this. But I do think I do think some of the things that are really important is to get very clear on what **[18:20]** outcomes you're expecting. Why are you measuring these things? In fact, in the report uh in the DORA report, we talk a little bit about metrics frameworks and which like what are some of the considerations that you might have as you think about metrics frameworks. You know, we oftentimes, you know, maybe we're thinking about we want to improve developer productivity. So, let's make sure that we define that. But maybe it's product excellence or maybe it's organizational effectiveness or just developer experience. Like all of these things are are very different. Now, they're certainly overlapping. like it **[18:50]** looks like a ven diagram if you take those three things into consideration. And I think it's it is uh I guess there's two things two pieces of advice that I will give that are concrete pieces of advice. The first is that it is important once you've decided on some metrics that you capture a baseline because you're not going to know how much AI has changed that metric if you don't know what it was before you brought AI in. And along with that piece of advice, you have to realize that simply adding AI to the mix like we work **[19:21]** in complex organizations with tons of emergent behavior and you can't just change one thing in your organization like that it's impossible to do that that change is going to have ripple effects and so forth. So you have to have some some understanding that yes maybe this was what it looked like before AI. This is what it looks like now after AI. AI may have had some thing to do with that change. It it may not have. So you're you're probably going to want want to dig deeper. And then the other thing that like the test that I always use when we're thinking about **[19:52]** metrics, like the reason to have metrics as as Bill kind of pointed out there is to have a conversation. And that conversation is going to lead to changing something that we do. So if you look at a metric, I like to ask what what behavior would we change if that number doubled or if that number was cut in half? What would we do differently? And if you can't answer that question, that's a pretty strong signal that the metric actually isn't driving any behavior, isn't driving any change within your organization. So that's probably not a good one. **[20:24]** >> Yeah. The other thing I would add to that is um just really quickly is you know from from the teams that we've been talking to they're the way that they're starting to think about this is almost like every metric is an AI impact metric because as they're rolling out these tools they want to not only measure yes usage adoption and these types of things but what's happening within the codebase. And so being able to bring those things together has been like tremendously important for the teams that we've been talking to. **[20:55]** >> Yeah, absolutely. I think that's why when you can find the metrics that are as close as possible to what uh results as the product growing and the product being more adopted, I think those tend to be uh metrics around the issues that we're closing. Are they valuable issues that we're closing? are they uh issues that are not resulting in a cascade of bugs that follow after we close the issues. I think that uh that issue **[21:26]** tracker uh surface is a rich starting point for a lot of the outcomebased metrics that one could consider. But uh there are certainly uh I I think a set of metrics that tend to be uh especially um indicative of what's happening on an AI team. And I think those are things like the amount of time spent reviewing code. That's something that is very prone to evolve as different uh code generation **[21:59]** practices are used in the organization. Uh if you see that you're writing twice as much code but you're spending four times more uh hours reviewing that code, that's telling you something that uh you probably might want to incorporate into the process of preparing a poll request before it gets posted. Um, release cadence I think is another uh metric that's pretty close to the truth of how rapidly your product is actually **[22:30]** releasing and improving. And I know Dora has been tracking this for years and years and giving us great uh benchmarks that we can use to see how uh our own team's performance compares to the industry. So, those are a couple of the areas that I think have uh often proven especially useful for teams that are looking at their AI usage through the lens of um what are the hard data outcomes that we can measure when we're using system A versus B versus C. **[23:00]** >> Hope you're enjoying this video, but did you know lead has lots more to offer? So webinar, podcasts, I even like articles, conferences, events and most importantly community >> and even thinking about like process tech and people which is how lead website is laid out. I really think this is the number one learning platform for me as an engineering manager. >> Don't forget to check out leadv.com. >> Thank you. There's a lot of questions **[23:30]** about metrics. So, I I want I want to like maybe if you guys want to answer them like a little rapid fire, we can just throw them in because I think they're really interesting and I I really appreciate the audience getting them in. Um, one is can you suggest based on your interviews with companies the top qualitative outcomebased metrics? I think we talked a lot about quantitative metrics and started talking about you know surveys and things like that but maybe someone can summarize **[24:02]** like what is the intention when we when we collect qualitative metrics and what behavior does that shift just to think about like Nathan's terminology here. >> Yeah. One of the qualitative or a maybe two that I'll I'll just mention really quickly that Dora looks into are friction and burnout. Um so how much friction are you experiencing throughout your day? And you know in our survey we actually don't define friction. We kind of leave that to the survey participant to understand what that means. And that's a like it is a good qualitative measure. You know friction when you see **[24:34]** it like I can't get something done because I'm waiting on something else or because this process is is ownorous. Um and so that I think is really important. And then burnout. Uh Bill was talking earlier about that we have to think about the well-being or the developer experience of the people on our team. And so again, trying to understand burnout maybe through something like are you cynical? Do you have cynicism while you're at work? Um are are you able to separate your work life from your home life or do you carry your work with you all the time? Like these are all signals of burnout that you want to watch out for. **[25:04]** >> Yeah. And I would just throw in I think from our perspective um you know developer experience is a core part of who Git Kraken is. We're a you know dev tools company and so this component is massive. So we think about measurements across all the different systems, but really at the center of that is developer experience. So even if all the metrics maybe look great, you really need to make sure that you're pulling in, you know, survey type data from developers to understand the sentiment. How are they feeling about how things are going? And you need to be able to **[25:35]** correlate that across all the other metrics, right? And so that's huge, especially for engineering leaders. I mean, you think about all the things that a good developer experience, all the outcomes that you want, um, you know, things like retention and, you know, burnout, like Nathan mentioned, it's so core to make sure that you're collecting that data as well. >> Thank you. Someone is asking um, what is your take about choosing a **[26:06]** simple but flawed single metric? And the example they gave is number of open PRs. >> All right. Uh I think that every metric is flawed. Like every metric is a proxy for something else. So they're all flawed. We just have to embrace that and understand, you know, as the saying goes like all models are are flawed, some are useful. The same is true about metrics. Uh and then I think in in to be very very clear I don't think any of us I **[26:36]** might be wrong so correct me if I am but I don't think any of us would argue for a single metric uh you have to have what they call in the space paper a constellation of metrics that you're looking at. There have has to be a few things that you're looking at so that you can get sort of different views on the problem that you're trying to address or the thing you're trying to measure. >> Yeah, absolutely. I totally agree. I think of it like a car dashboard, right? We wouldn't just look at the speedometer. We want to know, you know, oil temperature or pressure and these types of things. We need the complete **[27:06]** full picture. And I think that's sort of what we're saying here is not just one single metric or even one category of metrics, right? You need code quality, developer experience, AI impact, productivity. You need to be able to see all of those things. >> Yeah. I think it's a combination of wanting to have propulsive metrics, metrics that are measuring how rapidly you're moving forward, how rapidly you're authoring durable code that gets shipped to production. And you're measuring that uh relative to the um the **[27:39]** stability uh based metrics. How uh much duplication are you introducing as you rapidly deploy and rapidly release code? if you're uh closing twice as many issues as you did last year, but that's because you're copy pasting files all over the place that nobody's going to be able to ultimately manage. Uh just one of those metrics does not tell the valuable story that your team needs to know to make the right decisions as a long-term product. I think it has to be **[28:12]** uh propulsive, augmented by stability and quality. Uh and so that's why uh we are uh shifty in trying to just give one uh answer to the question because any one answer has to be connected to two or three other metrics that help to flesh out the other side of in the case of like open poll requests. Uh you could have questions around like do some open poll requests get abandoned? do uh some open poll requests uh take so long to **[28:44]** review that they ultimately end up being ROI negative like there's just always going to be but also for any given metric which is why uh picking three to five of them is something that uh I think is sort of a minimum required objective for teams that want to get a more holistic understanding of which of their teams are uh doing the best job of using the tools available. All right, great. I know everybody loves **[29:15]** metrics. There's a few more questions in the chat about this, but some of them are really specific. So, I would really would love to bring those to Slack after the the the call and you can get these guys uh to answer more specifically, which will be great. I'm curious to just >> I'm curious to hear from Jeremy beside just writing code where in this whole software development life cycle do you think that AI can genuinely help and **[29:46]** are we identifying the right bottlenecks or are we leaving something on the table? >> Yeah, absolutely. I love this question. Um, this is like really at the core of what we were, you know, what we've been building with Get Kraken Insight. So like you know the dev teams have had their AI tools you know for a number of years now and that's been amazing but what we're seeing now is uh something like kick kraken insights now we're getting into the AI tools for engineering leaders which is awesome right in terms of getting visibility into all these things using AI to help **[30:17]** us summarize some of these metrics and make like you know executive ready kind of you know decision- making and reporting and things like that so it's very exciting we see tools being deployed across across the full SDLC. So certainly in like within the planning space whether it's you know Jira or something else there's tremendous innovation there obviously within the coding and development space we've seen those tools for you know quite some time and even into you know the CI/CD and deployment um space right and so and **[30:49]** just even things like you know seeing teams get really innovative with different MCP tools and now we're getting into this like agentic workflow era so there's no end of like where things are going for sure. I think like AI coding assistance was really just the tip of the iceberg. Um so we're really excited about this for sure. >> Great. Um Bill, I want to hear about when teams **[31:20]** are, you know, they're both testing out different AI tools. They might have different results. um especially as a leader I want to understand how you're comparing understanding the ROI and there was also a question from the audience that's related to this um yeah around how to justify that the AI tool is worth the investment that was made >> so maybe yeah give us some examples there >> yeah so uh so yeah the starting point I **[31:51]** think for understanding whether uh AI investment is going to be worth it is having your uh conversation about what metrics within your company define your success. So again, I think a lot of companies success is defined by are we shipping the highest value issues uh that are in our Jira that basically our product uh people decided as the most important. Are we shipping those and are we happy while we do it? I think that is **[32:22]** uh you know the essence of what you're trying to iterate towards and I think that the best way to do that um after you've chosen your five metrics which you know get clear customers I if I'm just going to pick five that I very commonly see them key in on uh story points completed per uh sprint is a really common one or if they don't use story points just the overall number of issues closed per sprint is uh usually something that they pay a lot of attention into um the rate of durable **[32:53]** code change diff delta is how that gets measured on git clear there's other ways to measure that uh the release cadence that I had alluded to earlier time spent reviewing code and the amount of uh code that's copy pasted or duplicated I I think are all ideas that a lot of uh people evaluating AI teams will find value in considering but ultimately I think you want to give teams the flexib ability to make their own decisions about um what tools seem most promising **[33:26]** to them and you want to let them uh go forth and use those tools and see what the outcomes will be. Uh, and as long as you have a developer analytics tool or a uh, configuration with cloud code potentially that lets you see on a team by team basis how the metrics that you deemed most relevant are comparing on a per developer basis per team. Then you can start getting the truth about was **[33:56]** the team that adopted claude code the one that uh had a spike in the issues that were closed while having no increase in the time to review those issues once they adopted Claude in maybe September. Uh was team B that uh chose instead to go with uh cursor and to use multiple agents. are they uh opening fewer poll requests but getting more high impact tickets closed because they made better decisions about what to work **[34:27]** on and they got those things done quickly. So I love to give teams the flexibility to experiment. I think that is a piece of what makes those developer sentiment evaluations end up being more positive is when the developers feel like they have a voice in how this AI adoption is happening and when they have suggestions from other people about where AI is making the greatest positive difference which oftentimes is in **[34:58]** writing code uh or in writing test code uh in the examples that I've seen since that test code is a specific speific area that a lot of developers aren't enthusiastic about but is very important to the long-term health of a repo. So having a dashboard of metrics that lets you see on a team by team basis how outcomes are changing per team over time as they adopt different tools. I think that is the sort of entry point that a VP or a CTO or even a lead developer can **[35:31]** utilize to start understanding. Are there approaches that over time are consistently ranking higher in the metrics that we collectively decided are the ones that are important to our circumstances? And if so, can we present that information to the teams that aren't yet adopting those uh recommendations and see if their performance changes when they do follow the practices that the teams that uh that have had the best results have **[36:02]** tried. And so I think it should always be uh experimentations that are being evaluated based on the um the data that they generate both code quality, productivity, uh developer happiness. And then those experiments should be promoted and celebrated and uh repeated as the team continues to experiment with what's inevitably going to be a new uh better agent that is uh offered next month. and you know causes **[36:32]** everybody to completely change their IDE and everything about their developer uh experience again. So it's uh it's definitely a moving target which is another reason why I think uh having a constant idea of which metrics are important to your team helps you know whether you are continuing to make progress towards that moving target. I think one of the things that you touched on there, Bill, that is really important um is that it it isn't necessarily the tool that's the tool in and of itself **[37:04]** that's causing those metrics to change. It's the other practices that you've built up around them. And so I think it's it's a a failure mode that I've seen is like, hey, we've distributed three different tools to three different teams. Whichever team is doing best, we're just going to uh align on that tool. it's probably not the tool but rather the practices whether that's they're being very intentional about how they're doing it and so what I'm seeing the most successful organizations doing are is sort of building communities of practice and communities of learning around what are the good practices and **[37:35]** they that might look like things like hackathons that are happening you know the the you mentioned a a cloudmd file like that claw.mmd file helps set up some guardrails for how the agent is going to behave are you setting up similar guardrails for other tools and how do we start sharing those those guardrails? How do we share better context? I think there's a lot of stuff that goes into enablement and really building that adoption. The other thing that we've seen is that there's a lot of um there's always a question of trust. **[38:07]** How much do you trust the output of the AI? And the reality is like adoption feeds trust. So the more you use a thing, the more likely you are to trust it. And the more you trust it, the more likely you are to use it, right? It becomes a nice virtuous cycle. But this is also a pointer for leaders that we've got to give time and space for teams to go learn and experiment and play with these things so that they can start building that trust and understanding, hey, this this works really well when I when I use it like this uh and maybe not so well when I use it like that. And **[38:38]** share those lessons across the organization because many of those lessons are are going to transcend which tool you're using. And it's it's more so how you're using the tools than it is which tool you're using. >> Yeah. And one other quick thing I would add there too is just kind of tying it back to the business outcomes. So we've been spending a lot of time with teams and and sort of how we've been helping them is being able to you know first of all is being able to get these metrics right as kind of the baseline and now how do you translate that as an **[39:08]** engineering leader back to you know business outcomes that you know the the business will will care about. And so where we've been helping them is just, you know, even just taking simple examples. Some of the teams that we work with will have like a kind of a a simple value calculator. And it's like, yes, sure, maybe we're paying, you know, 40 or 50 bucks per developer, uh, you know, for some of these AI coding assistant tools, but we are seeing, you know, $2,000 per developer in productivity gains monthly, right? And so these are the types of things that they're able to translate and you know through faster **[39:40]** velocity and healthier code translating those back into the business that you know helps the business understand you know the value of these AI tools. >> Thank you. I think we have time for one more question to wrap it up. Um I'm kind of torn. I I actually really want to take this audience question um that's related to what you're talking about. How can you try to decouple the **[40:12]** effects of AI from everything else that's happening in the company? For example, major refactorings, hyperrowth, etc. And I think this is a general like evergreen question for experimentation, but I'm really curious how you're talking about attribution for AI tooling. I mean I think this the first option is to just let the natural experiment of different teams use AI different amounts uh factor into your evaluation. **[40:45]** All almost all the modern developer analytics tools allow you to understand how much a team is using AI and so you'll be able to see which teams are maybe not using AI or using AI very little and it's not uncommon that those teams can have great outcomes. And so that is one starting point that you can use is just if you have multiple teams in your company and some of those teams aren't using AI but yet they're still producing uh durable code change at a **[41:18]** rate on par per developer with the teams that are using AI every day. uh that is telling you something about what the real bottom line output uh or outcome of the tool usage is or lack thereof uh lack of positive uh output. Yeah, I think I think this also kind of points back to this idea that you have to have baseline numbers. you might see PR velocity start to change as you're using more AI, but if you have a baseline and a historical view of that, **[41:49]** maybe it's accelerated, but maybe it was already accelerating. So, you haven't actually changed the trajectory of that by introducing AI. But I think a really fun and to every engineer that is listening to this I'm sorry but a really fun experiment you could run is forbid the use of AI for the next iteration from a team and see how that changes both how they feel about the work and how things are going like it's very hard to change just one thing but if we forbid just one thing maybe we get a closer tighter signal on what what do we **[42:21]** actually attribute here back to the use of AI >> that's fun eh it's fun to not to type. Sorry, I should have put that in spare quotes. It's fun. Uh, it can be an interesting experiment. Um, and and maybe you'll learn maybe you'll learn very fast that, you know, instead of this full iteration, two days was enough. We need it back. >> All right. Thank you all so much for the insights on today's panel. I feel like we can go for 3 hours, but we're going **[42:52]** to head now to the lead dev Slack. The channel is called Panel Q&A. So please please head there. We'll copy and paste the remaining questions and we'll continue discussion. Thank you so much for joining and see you next time. >> Thanks everybody.