The AI Capabilities That Amplify Your Org - CourtneyR.dev's 2nd Brain

The AI Capabilities That Amplify Your Org ![rw-book-cover](https://img.youtube.com/vi/ik7YkVRZxxE/maxresdefault.jpg) 🔗 **Source:** [The AI Capabilities That Amplify Your Org](https://www.youtube.com/watch?v=ik7YkVRZxxE) > [!note]+ Document Notes > tags: [AI-assisted-development, platform-engineering, DevOps, developer-productivity] > author: Lead Dev > source: YouTube > reference: https://www.youtube.com/watch?v=ik7YkVRZxxE > date created: 2024-12-28 20:30 > > # 🏷️ Title: The AI capabilities that amplify your org >  ## 🧰 Action Inbox - [H1] Fill out DORA survey when available — annual industry benchmarking survey #measurement - [H2] Assess internal developer platform maturity — use CNCF platform maturity model #platform-engineering - [H3] Measure AI adoption beyond licenses — track code acceptance rates and quality metrics #AI-metrics ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [DORA Research](https://dora.dev) - [2024 State of AI Assisted Software Development](https://dora.dev/research/) ## 📝 Summary - Engineering leaders discuss how AI amplifies organizational capabilities bidirectionally - making good practices better and poor practices worse - DORA research now includes five core metrics (adding rework) and introduces an AI capability model for outcome-focused improvement - Platform engineering emerges as critical foundation for organizational AI success, providing context and automation across the entire SDLC - Key stakeholders are engineering leaders, CTOs, and platform teams navigating AI adoption and ROI measurement - Main risk is investing 100% effort in AI adoption while neglecting underlying platform fundamentals like testing automation and CI/CD ## 📖 Main Content The discussion centers on how AI capabilities amplify organizational performance in both directions - enhancing strengths while exacerbating weaknesses. Eric Maxwell from Google's DORA team explains that their research now focuses on an AI capability model that helps organizations identify which capabilities to develop based on desired outcomes. The bidirectional amplification concept is critical: teams that are user-centric see improved performance with AI, while non-user-centric teams experience substantially worse performance. This pattern applies across code quality, team effectiveness, and product outcomes. Pashant Batra from Harness emphasizes that measuring AI impact requires looking beyond simple adoption metrics to understand actual usage patterns and business outcomes. Organizations need to track not just velocity gains but also quality impacts, rework rates, and long-term maintainability. Platform engineering emerges as the key enabler for organizational-level AI benefits. While individuals and teams can gain productivity from AI tools, achieving organizational performance requires a well-built internal developer platform that provides context, automation, and governance across the entire software delivery lifecycle. ## 🗒️ Notes #### ⏱️ [00:00](https://youtu.be/ik7YkVRZxxE?t=0) — [H1] Introduction and Panel Setup - Key idea: Lead Dev panel on AI capabilities with Google DORA and Harness product leaders - Evidence: "Discussion on how AI capabilities amplify your organization" #### ⏱️ [03:42](https://youtu.be/ik7YkVRZxxE?t=222) — [H2] Pashant Introduction - Software Engineering Insights - Key idea: Harness focuses on empowering engineering leaders with deep insights into team efficiency - Evidence: "Everything begins with measurement and then using those measured metrics to drive actions, drive transformation" #### ⏱️ [05:15](https://youtu.be/ik7YkVRZxxE?t=315) — [H3] Eric Introduction - 10X Technology at Google - Key idea: 10X refers to amplifying companies through technology, not individual engineers - Evidence: "10X is something we talk about quite famously inside Google. It's not really about the engineer. It's more about how can you 10x a company through the use of technology" #### ⏱️ [07:30](https://youtu.be/ik7YkVRZxxE?t=450) — [H1] DORA Framework Overview - Key idea: DORA now includes five core metrics with rework as the newest addition - Evidence: "As of this last year of research 2025, it's now five capabilities or five metrics. We added rework to the list" #### ⏱️ [12:45](https://youtu.be/ik7YkVRZxxE?t=765) — [H2] DORA Capability Model Explanation - Key idea: Metrics are lagging indicators; capabilities on the left drive the outcomes on the right - Evidence: "The metrics are actually the least interesting part of the whole DORA research. What our research really focuses on is the blue box on the right hand side, the outcomes" #### ⏱️ [18:20](https://youtu.be/ik7YkVRZxxE?t=1100) — [H1] AI Capability Model Introduction - Key idea: New AI capability model shows bidirectional amplification effects - Evidence: "If you are doing things well, AI makes those things better. If you are doing things not very well, AI makes things worse" #### ⏱️ [21:15](https://youtu.be/ik7YkVRZxxE?t=1275) — [H3] User-Centric Teams Example - Key idea: AI amplification depends on underlying team characteristics - Evidence: "Teams that are user centric, AI then makes the team performance better. Teams that are not user centric, AI introduction makes team performance measurably worse, substantially worse" #### ⏱️ [24:30](https://youtu.be/ik7YkVRZxxE?t=1470) — [H2] Industry Pressure for AI ROI - Key idea: Engineering leaders face tremendous pressure to show measurable AI returns - Evidence: "Every leader today is saying look I am making investments. Not enabling my teams with the best in breed AI tooling is not an option" #### ⏱️ [28:45](https://youtu.be/ik7YkVRZxxE?t=1725) — [H2] AI Adoption Measurement Challenges - Key idea: Need to measure beyond licenses to actual code acceptance and modification rates - Evidence: "Just having a license and you know once in a while prompting something in the tool does not mean you're effectively using the tool" #### ⏱️ [32:10](https://youtu.be/ik7YkVRZxxE?t=1930) — [H3] Code Provenance Problem - Key idea: Industry lacks proper tagging system for AI-generated vs human-modified code - Evidence: "Truly tagging every line of code like imagine if git had a metadata for every line of code which said this was purely AI generated versus this was AI generated and then edited" #### ⏱️ [35:40](https://youtu.be/ik7YkVRZxxE?t=2140) — [H1] Trust Building Process for AI Tools - Key idea: Three-stage process: adoption, trust-building, then productivity gains - Evidence: "Step one, adopt AI tools, step three, profit. The problem is there's a middle part... This process can take between a month to two months to actually build trust" #### ⏱️ [39:15](https://youtu.be/ik7YkVRZxxE?t=2355) — [H1] Platform Engineering as AI Foundation - Key idea: Internal developer platforms are key to unlocking organizational AI performance - Evidence: "The internal developer platform is the key to unlocking organizational performance with AI... most people have internal developer platforms but almost none of them have well-built ones" #### ⏱️ [42:30](https://youtu.be/ik7YkVRZxxE?t=2550) — [H2] Harness AI Integration Across SDLC - Key idea: AI agents can automate entire software delivery lifecycle beyond just coding - Evidence: "Harness is AI for everything after code... developers are spending 30 to 40% of their time actually developing code, 60 to 70% goes into software delivery" #### ⏱️ [44:50](https://youtu.be/ik7YkVRZxxE?t=2690) — [H3] Platform Context Advantage - Key idea: Platforms with holistic SDLC context enable more powerful AI automation - Evidence: "When you have this holistic view into every action you take as part of your SDLC, the way AI can truly understand and now begin to become your mentor... is just so much more immensely powerful" ## 📚 References - [DORA Research](https://dora.dev) - [AI Capabilities Model Report](https://dora.dev/research/) - [Lead Dev Community Slack](https://leaddev.com/slack) - [CNCF Platform Engineering Maturity Model](https://tag-app-delivery.cncf.io/whitepapers/platform-engineering-maturity-model/) ## 🗣️ Paraphrase - AI acts as an amplifier for existing organizational capabilities, making strong practices stronger and weak practices weaker, particularly around user-centricity and code quality - DORA research has evolved to include rework as a fifth core metric and developed an AI capability model that maps specific capabilities to desired organizational outcomes - Measuring AI success requires tracking adoption, trust-building, and actual business impact rather than just license usage or developer perception of productivity gains ## 📰 Newsletter **AI Amplifies What You Already Do - Make Sure It's Good** The latest research from Google's DORA team reveals a critical insight: AI doesn't just make you better - it makes you *more of what you already are*. If your team is user-focused and has solid engineering practices, AI will accelerate your success. But if you're struggling with fundamentals, AI will amplify those problems too. The key? Build your platform engineering foundation first. Organizations seeing real organizational gains from AI have well-built internal developer platforms that provide context, automation, and governance across the entire software delivery lifecycle. **Practical Steps:** - Assess your platform maturity using the CNCF model - Measure AI adoption beyond licenses - track actual code acceptance rates - Focus on capabilities that drive your desired outcomes, not just tool adoption Resources: [DORA Research](https://dora.dev), [Platform Engineering Guide](https://platformengineering.org), [CNCF Platform Maturity Model](https://tag-app-delivery.cncf.io/whitepapers/platform-engineering-maturity-model/) ## ✅ Action Items #### DevRel - Developer advocacy: Create content explaining bidirectional AI amplification effects, present at engineering conferences - Education: Develop workshops on AI capability assessment, platform engineering fundamentals - Developer experience: Improve documentation for internal AI tools and platform onboarding - Developer marketing: Campaign around "AI amplifies what you do" messaging - Community management: Facilitate discussions on AI adoption challenges in developer forums #### OSPO - Policy & Governance: Establish AI tool evaluation criteria and approval processes - License Compliance: Review AI coding assistant terms and enterprise agreements - Security Management: Assess vulnerabilities in AI-generated code, implement scanning protocols - Software Inventory: Track AI tool usage and generated code provenance - Developer Support: Provide training on responsible AI tool usage and quality practices - Community Engagement: Fund research on AI impact measurement and platform engineering - Strategic Alignment: Define KPIs for AI ROI measurement and platform maturity progression #### Community - Maintainers: Support development of AI capability assessment tools and platform templates - Contributors: Create onboarding programs for AI-assisted development workflows - Sponsors: Fund research into AI amplification effects and measurement methodologies - Ecosystem: Coordinate standards for AI tool interoperability and code provenance tracking #### WordPress - Core: Implement AI-assisted code review processes, establish quality gates for AI-generated contributions - Core Performance: Profile AI-generated code impact on performance budgets and optimization - Core AI: Develop evaluation frameworks for AI tooling integration and quality assessment - Core Program Team: Coordinate AI adoption strategy across teams, track capability development - Community: Create educational content on responsible AI usage in WordPress development - Training: Develop curriculum on AI-assisted WordPress development best practices - Documentation: Use AI to improve documentation quality while maintaining human oversight - Meta: Assess platform engineering maturity for WordPress.org infrastructure #### The WPCC - Programs: Establish fellowships for AI research and platform engineering improvements - Projects: Fund development of AI capability assessment tools for WordPress ecosystem - Funding: Create grants for organizations improving their platform engineering maturity #### FAIR - Technical independence: Develop standards for AI tool evaluation and code provenance tracking - Policies: Establish guidelines for transparent AI usage and quality measurement - Governance artifacts: Create templates for AI governance and platform engineering charters ## 🧪 Hypotheses - Organizations with mature internal developer platforms will see 2-3x greater organizational benefits from AI adoption compared to those with ad-hoc tooling - Teams scoring high on user-centricity metrics will show measurably better outcomes from AI tools within 3-6 months of adoption - Companies investing exclusively in AI tools without improving underlying CI/CD and testing infrastructure will see increased instability metrics - The time to build trust in AI tools (1-2 months) correlates with the quality of internal documentation and platform self-service capabilities ### 📊 Methodology - Longitudinal study tracking DORA metrics before/after AI adoption across organizations with varying platform maturity levels - Survey research measuring user-centricity scores and correlating with AI outcome metrics over 6-month periods - Case study analysis of organizations experiencing AI-related instability vs. those seeing productivity gains ### ❓ Questions - How can organizations systematically assess their readiness for AI amplification before tool adoption? - What specific platform engineering capabilities have the highest correlation with successful organizational AI outcomes? - How might code provenance tracking be implemented at scale without creating developer friction? ### 🔮 Future Research - Development of standardized AI readiness assessment tools for engineering organizations - Investigation into optimal sequencing of platform improvements and AI tool rollouts - Research into developer experience factors that accelerate trust-building with AI tools ### 💡 Implications - Practice: Organizations should assess platform maturity before major AI tool investments - Policy: Need for governance frameworks that address AI tool sprawl and quality control - Security: AI-generated code requires enhanced review processes and vulnerability scanning - Education: Training programs must cover platform engineering alongside AI tool usage - Funding: Budget allocation should prioritize platform foundations over pure AI tool licensing ### 🧠 Learning Styles Assessment Reading level estimate: 12th grade (Flesch-Kincaid) #### 👀 Visual learners - Present: DORA capability model diagrams, platform maturity progression charts - Improve: Add AI amplification effect visualizations, platform architecture examples #### 🔊 Aural learners - Present: 45-minute panel discussion with expert insights - Improve: Create 5-minute audio summary of key amplification concepts #### 🗣️ Verbal learners - Present: Clear definitions of platform engineering, bidirectional amplification - Improve: Add more analogies for AI amplification effects, glossary of terms #### 🧩 Kinesthetic learners - Present: References to hands-on platform assessment tools - Improve: Create interactive AI readiness checklist, platform maturity self-assessment #### 🧠 Logical learners - Present: Clear cause-effect relationships between platform maturity and AI outcomes - Improve: Add decision trees for AI adoption timing, measurement frameworks #### 🧑‍🤝‍🧑 Social learners - Present: Panel discussion format, Slack community engagement - Improve: Create cohort-based AI adoption programs, peer review rubrics #### 🧘‍♀️ Solitary learners - Present: Detailed research reports, self-assessment opportunities - Improve: Add reflection prompts on organization's AI readiness, journaling templates #### 🌿 Naturalistic learners - Present: Systems thinking approach to platform engineering - Improve: Emphasize ecosystem impacts of AI tool choices, sustainability considerations ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Clear heading structure, descriptive link text for research reports - Improve: Add alt text for D --- ## Full Transcript [00:12] Hello, hello and welcome everyone. Thank [00:16] you both so much for joining us live [00:18] today and uh for those that are joining [00:22] in the audience live or those listening [00:23] the recording for a discussion and this [00:27] is a lead dev panel created in [00:29] partnership with harness. It'll last [00:32] roughly 45 minutes after which both [00:34] myself and the panelists will head over [00:37] to the lead dev Slack to answer more of [00:40] your questions in the hashtag I guess we [00:44] say hashtag for it panel Q&A channel and [00:48] to continue discussion if you aren't [00:51] already a Slack member yet for those [00:55] live I'm just shooting you right now the [00:58] Slack invitation it's a wonderful [01:01] community that it's just a way to [01:04] continue discussion with engineering [01:06] leadership and also we welcome the [01:09] voices of course of developers if they [01:11] want to you know [01:14] uh add their thoughts as well because we [01:17] need to be listening to you which will [01:19] be part of the theme of today's and all [01:22] conversations for those of us joining us [01:24] live feel free to just share your [01:26] questions in the zoom and I will ask [01:30] them and then if we have extra questions [01:33] because we have a good community joining [01:34] us today it will continue in the Slack. [01:37] So let's get started. I'm Jennifer [01:38] Rigggins. I'm a technology journalist [01:41] and today I'll be joined by Eric Maxwell [01:44] head of 10X technology at Google and [01:47] Pashant Batra or sometimes B PB uh [01:51] senior director of product management at [01:54] Harness at Harness. Uh pashant can you [01:57] please introduce yourself and what [01:59] you're most excited to talk about today [02:06] you are on mute because you know [02:11] >> had to happen right [02:12] >> had to happen [02:14] >> well good morning good afternoon good [02:16] evening everyone uh I am a senior [02:18] director of product at harness uh I've [02:21] been in the industry for 20 plus years [02:23] as you can imagine over that time I've [02:25] built many different products from uh [02:27] building my first VPN to gateways to [02:30] mobile device management software to um [02:34] CASBY cloud access security brokers to [02:36] my last gig where I was at Nutanix where [02:39] we were building a hybrid cloud platform [02:42] uh and then sort of my five years at [02:43] harness that I've spent I've built a few [02:45] different things uh you know off late [02:47] what I do is I lead one of our product [02:49] lines called software engineering [02:51] insights uh and just as a as a quick [02:54] sort of uh overview of what that means, [02:56] right? So, software engineering insights [02:58] is a product line from harness uh that [03:02] essentially uh aims to empower [03:05] engineering leaders today with deep [03:07] insights into you know how well are [03:10] teams building software, how efficient [03:12] are the processes you're running, how [03:14] productive are teams, how well aligned [03:16] is your execution to the outcomes that [03:17] you're looking for, how well are you [03:19] using AI, right? Everything begins with [03:21] measure measurement and then using those [03:24] measured metrics to drive actions, drive [03:27] transformation, recommend changes, [03:29] right? And so that is what uh this [03:31] product line does. U and I'm really [03:33] excited to sort of uh you know chat with [03:35] Jennifer Eric today and talk about how [03:38] we see the advent of AI in the industry [03:40] and how uh you know modern day platforms [03:42] like harness software engineing insights [03:44] a lot that Eric is working on are truly [03:46] sort of participating in this [03:47] transformation. [03:49] >> Absolutely. Thank you so much. Nice to [03:50] have you. And Eric, can you please [03:52] introduce yourself and then talk about [03:54] your job title because kind of famously [03:57] in the lead dev community, we do not [04:00] love the term 10x engineer. So maybe [04:03] talk about what 10x technology is as [04:05] well [04:06] >> at Google effect. [04:08] >> Yeah. No, no, no problem. Um, so I'll [04:11] just give a little background. I started [04:13] off my career as an engineer and I spent [04:15] most of my time doing back-end uh API [04:18] engineering. mostly working in like the [04:20] the middle layer and and the back end. [04:22] Around 2010 2011, I started really [04:26] getting into automation. Um, not because [04:29] I was trying to do DevOps, but I was one [04:32] of three people at a startup company [04:34] that became webcale. Um, and so we just [04:38] considered that survival, uh, rather [04:40] than putting any, you know, terms or [04:42] tags around it. So, um, I kind of got [04:44] into the game early on and then around [04:47] 2015, I started working at Chef Software [04:50] and that's when I really started working [04:51] with customers and and helping them do [04:55] all the things that I had already had [04:57] like a 5-year jump start on, which was [05:00] pretty cool. Um, and that led me to [05:02] Google. So, I've been here at Google now [05:04] for almost eight years. I'm part of the [05:06] the Dora team. So I'm one of the [05:09] researchers on DORA and I'm also one of [05:12] the authors of of the reports that we [05:14] put out. Um and then so what I wanted to [05:17] do at Google was take the research that [05:20] we do because that really is you know [05:22] where my my passion lies. I really enjoy [05:24] that aspect of it and I wanted to figure [05:26] out how I could take that research and [05:28] bring it to our customers. Um so [05:31] thankfully I had the opportunity to [05:33] start a uh consulting practice on the [05:36] Google cloud consulting side uh which we [05:38] call 10x technology um for better or [05:41] worse but uh so what what we do [05:43] specifically is we we focus on bringing [05:46] door research to our customers. So you [05:48] know we we focus on developer [05:50] experience, how to increase [05:52] productivity, how do you build [05:54] engineering excellence? Of course, a big [05:56] part of that is the cultural aspect and [05:58] then from a technology perspective, we [06:00] focus on, you know, DevOps, AI tooling, [06:04] uh, and of course, lots of platform [06:06] engineering. So, um, nice to be here and [06:09] and to address the the 10X part, 10X is [06:12] something that we talk about quite [06:14] famously inside of Google, actually. Um, [06:17] and so it's not really uh a mention [06:19] about the engineer. It's more about how [06:22] can you 10x a company through the use of [06:25] technology. [06:27] >> That is a beautiful [06:30] disclaimer and explanation. Yes, the 10X [06:32] teams, the 10X organizations is totally [06:36] feasible. Nobody really wants to work [06:38] with someone that selfidentifies as a [06:40] 10X engineer. So, absolutely. [06:42] >> No, no, not at all. So, thank you for [06:44] that clarification. [06:45] So for those that don't know, those in [06:47] Europe know there's the DORA [06:49] environmental regulations completely [06:50] different. Uh DORA Google Google is [06:53] deacronomized. Nathan Harvey has taught [06:56] me that word. Um but it's this industry [06:59] benchmarking annual survey that's [07:01] facilitated by Google. It's most [07:04] associated with its four metrics which [07:05] measure throughput and stability, but [07:07] it's actually well over 100 150 [07:10] questions that allow organizations to [07:12] see where they stand. Uh, as a [07:15] technology journalist, this is one of [07:17] those data sources I really look forward [07:19] to every year and I have a selfish plea [07:22] that I ask you to fill out this [07:24] anonymous survey when it comes around. [07:26] Uh, this is really fascinating data and [07:30] it helps you understand and reflect on [07:33] yourself and we get this overarching [07:35] benchmarking report. I've shared it in [07:37] the chat for those live and it's in the [07:39] webinar resources link so you can get [07:41] this year's report. But Eric, can you [07:44] offer a more official definition of what [07:47] the door study is and how companies can [07:49] use it? [07:51] >> Sure. Yeah. So, um, just one item of [07:54] clarification as of this last year of [07:57] research 2025, it's now five cap or five [08:00] metrics. Um, so we added rework to the [08:03] list, which is pretty cool. It was kind [08:05] of a long time coming. uh we you know we [08:07] had been thinking about it and really [08:09] researching it for two years and uh then [08:12] we finally added it to what we call our [08:14] our core model. So unfortunately we [08:17] haven't had the time to update the [08:18] website but all of that is coming soon. [08:20] So we'll be adding that to the quick [08:22] check on the website uh and we'll be [08:24] adding it to you know all of the [08:25] official documentation. But um yeah, so [08:29] um I I wonder if it would be helpful [08:30] just to kind of show like the the Dora [08:33] capability diagram uh and then I can [08:36] kind of walk through that. [08:39] Um [08:40] >> absolutely. Yeah. So let's share that so [08:43] everyone understands. [08:45] >> So uh it looks like our corporate [08:50] >> Oh, this is something we were having [08:53] technical difficulties before [08:55] >> will not allow me to share my screen. So [08:57] >> that's totally fine if you [08:58] >> can can you share dora.dev? [09:01] >> Absolutely. It would be my pleasure. Let [09:03] me make sure I'm not having nefarious [09:05] things open on my laptop. [09:08] Um here we go. So I'm going to dora.dev [09:13] research [09:14] >> and then I'm going to share my screen. [09:17] Thank you so much. [09:19] Share screen. [09:22] >> Yeah. And if you want to just expand [09:24] that capability model and I can kind of [09:26] talk through it there [09:28] >> like under the capabilities, right? But [09:30] this is [09:31] >> just click on research. [09:34] >> Okay. [09:35] >> Just the top button there. [09:37] >> There we go. [09:38] >> And then scroll down and then click on [09:40] um the uh the the radio button. [09:45] >> Which one? [09:46] >> The radio button at the top there where [09:48] it says detail. [09:50] >> Radio is what? I'm not. [09:52] >> Oh, sorry. So, right above the word [09:54] outcomes. There you go. [09:55] >> Thank you. [09:56] >> Okay, cool. And then just kind of scroll [09:57] down so we can see the whole model or at [10:00] least as much of it as possible. [10:02] >> Cool. [10:03] >> All right. So, so let me I'll just give [10:05] a brief uh explanation. Um the the [10:09] metrics that were mentioned um the now [10:11] five key metrics, those sit in the [10:13] middle, the green box there under [10:15] performance and under software delivery [10:17] specifically. So that's how we measure [10:20] software delivery. So these metrics are [10:23] um globally recognized as the industry [10:25] standard way of measuring software [10:27] delivery performance. Um and in fact [10:29] there are the last time I counted which [10:32] admittedly was maybe five or 6 months [10:34] ago there were 60 third party tools that [10:37] surfaced these metrics for their users. [10:40] Um so just kind of adding to you know [10:43] the giving credence to what I said about [10:45] being kind of industry accepted metrics [10:47] and ways of measuring. Now the [10:49] interesting thing is the the metrics [10:52] have definitely been the thing that have [10:56] maybe how do I say this gotten a lot of [11:00] attention. Uh and so as a result a lot [11:03] of people think that the metrics are [11:06] synonymous with Dora. So when they think [11:08] Dora, they think metrics. [11:10] I personally think that the metrics are [11:12] actually the most the least interesting [11:15] part of the whole Dora research and the [11:17] framework and and whatnot. Uh and and [11:20] I'll kind of explain why I think that. [11:23] So what our research really focuses on [11:25] is the blue box on the right hand side, [11:28] the outcomes. Because let's face it, [11:31] this is what everybody cares about, [11:33] right? All organizations care about the [11:36] organizational performance. So whatever [11:38] the organization is trying to achieve, [11:40] that's what the organization cares [11:41] about, right? I mean that's why the [11:43] organization deploys AI and you know [11:46] does DevOps and builds platforms and you [11:48] know everything else that an [11:49] organization does. They do it to achieve [11:51] their goals and also they want to [11:54] improve the well-being of the people [11:55] that work there. At least most companies [11:57] do. Uh so so this is where we start our [12:00] research. We we start with the outcomes. [12:03] Uh and then so what we do is we ask the [12:05] question okay what are the things that [12:07] will predict higher levels of these [12:09] outcomes that we care about and our [12:11] research has shown that software [12:13] delivery performance and reliability are [12:16] two very strong predictors of these [12:19] outcomes that we care about. [12:21] So and that's where we get the metrics [12:24] because you know if we agree that these [12:26] things are important and that they drive [12:28] the outcomes well we need a way to [12:29] measure these things obviously and so [12:31] that's that's how we came up with um the [12:34] the doraometrics. [12:36] So but that's not where the research [12:38] stops and that's where I think a lot of [12:40] people actually think that you know the [12:42] research stops. it keeps going. And so [12:44] what we want to do is we want to [12:46] understand what are the things that [12:48] organizations can do that will predict [12:50] higher levels of software delivery [12:52] performance and higher levels of [12:54] reliability. And so when we think about [12:57] the metrics, I would encourage folks to [12:59] think about the metrics as just a gauge. [13:02] It they're they're just lagging [13:04] indicators that tell you if you're [13:06] heading in the right direction. And the [13:09] things that you really want to focus on [13:11] are the capabilities on the left hand [13:13] side. And and the way that you do this [13:15] first is obviously by understanding what [13:18] the capabilities are and understanding, [13:20] you know, what do we mean when we say [13:21] continuous integration. Um, as an [13:24] example, we want to make sure that we're [13:26] all sort of talking the same language. I [13:28] I've worked with a lot of customers [13:29] where, you know, you're saying the same [13:32] words, but you mean pretty different [13:34] things, right? So So first, let's ground [13:37] on that. look at what we mean by those [13:39] capabilities and then you want to [13:41] prioritize at the application or service [13:44] level where your strengths and [13:47] weaknesses in these capabilities lie. So [13:50] you need to have like a real truthful [13:52] introspection about you know where are [13:55] the bottlenecks in the process and once [13:57] you've identified the bottlenecks you [13:59] can then prioritize them and then start [14:01] improving them. So the the bulk of our [14:03] research really is is on these [14:05] capabilities here that you see on the [14:07] lefth hand side. [14:09] So I hope that gives a good overview for [14:11] folks. [14:12] >> Absolutely. [14:14] So in the context of this year's report [14:18] which was renamed [14:22] which is something unto itself I think [14:25] we could say is different. It's called [14:27] the state of AI assisted software [14:29] development because that's where we are. [14:33] Uh [14:34] as what's changed? What is different [14:37] right now? What have we learned? Like [14:40] because this year is I think feels like [14:44] a turning point [14:45] >> for the tech industry and for the [14:47] report. No. [14:48] >> Yeah, definitely. Um I I think it kind [14:51] of follows along the theme of the [14:52] deacroning that we did uh two years ago [14:55] for the 10-year anniversary. And the [14:57] reason that we did that is because we [15:00] felt that folks were getting too hung up [15:03] on the term DevOps. [15:05] And you know those of us that have been [15:07] in the industry when we say the word [15:10] DevOps we understand what that means. We [15:12] understand that it is you know a [15:14] cultural and a process and an [15:17] organizational transformation. [15:19] But a lot of people conflate that with [15:22] technology. And when you're trying to [15:25] have these conversations with people who [15:27] are not technologists, [15:30] a lot of times they just kind of shut [15:32] down, you know, when they hear trigger [15:34] words, if you will, you know, and then [15:36] DevOps is kind of one of them where we [15:38] would have these conversations and I [15:40] would use the word DevOps and they would [15:41] say, "Okay, well, you know, our [15:44] pipelines are fine. We don't want to [15:45] focus on our pipelines. We want to focus [15:46] on delivering code faster. you're like, [15:49] "Yes, that's what I'm talking about." So [15:51] anyway, so we we made the uh we made the [15:53] decision then to to to deacronym and [15:56] also then as part of that, the name of [15:58] the report was the state of DevOps [16:00] report. Um so it just seemed natural [16:03] that we evolved that remove DevOps from [16:06] the title and really start focusing on [16:09] as you mentioned the the changing [16:11] landscape that we see in the industry [16:13] today. [16:15] >> Okay. and not just a title. What is [16:18] different this year? What has changed? [16:21] Because last year it was kind of like, [16:22] well, [16:24] AI has assisted the quality of docs [16:27] quite a bit, but it's kind of made [16:30] things less stable. So, what is the [16:33] unique about the one that came out in [16:35] November? [16:36] >> Yeah. Well, I mean, obviously there's a [16:38] major focus on AI in the entirety of the [16:42] report. So, not only did we expand the [16:44] research that we uh started three year [16:47] two years prior um and really started [16:51] digging into uh AI at a much deeper [16:53] level. And by the way, just to kind of [16:55] back up for a second, for those of [16:57] people that haven't really been [16:58] following our research, the what we do [17:00] is we we study something, you know, in a [17:03] given year and when we discover [17:05] something that's interesting, exciting, [17:07] something that we want to understand [17:08] more, we continue to explore those [17:11] topics and just go deeper and deeper and [17:13] deeper. So, this is the third year of of [17:15] that that AI uh pursuit. And it was [17:18] really apparent to us that this was [17:20] something that we needed to go allin on [17:22] because the industry is going allin on [17:24] this. And it really has changed the the [17:27] paradigm for what's happening at the um [17:32] you know at the the entire organization [17:34] from idea to implementation. And so we [17:38] wanted to get a deeper understanding of [17:41] you know how is AI affecting developers [17:44] in their day-to-day? What are things [17:45] that developers are using? How are they [17:47] using these things? What impacts is AI [17:50] having? And one of the the impacts that [17:52] we did discover uh again was that it is [17:56] increasing the instability [17:58] which is not a good thing. Um and also [18:01] increasing rework. So so we we did see [18:04] that. And then another major change or [18:07] major thing that we added to the report [18:09] this year is we create so that [18:11] capability guide um or map rather that [18:14] you were just displaying on the screen. [18:16] We expanded that and we created an AI [18:19] capability model. So what we wanted to [18:22] understand so what we talked about just [18:24] a minute ago is the capabilities that [18:27] will drive software delivery [18:28] performance. What we wanted to [18:30] understand was when you when you have AI [18:35] what are the capabilities that you can [18:38] can focus on to drive specific outcomes [18:42] that you're looking for. So there we go. [18:46] Um, so what this does is this allows you [18:49] to start from the right hand side. We [18:51] always start from the right hand side [18:52] and work our way to the left. Uh, and [18:55] and so you can look at this map and and [18:57] say, "Okay, I want to improve, you know, [19:00] uh, my team performance as an example. [19:03] So you can hover over team performance [19:05] and what it does is it will tell you and [19:07] highlight the uh the capabilities that [19:11] you need to focus on when you have when [19:14] you're working with AI. And so those [19:18] capabilities will amplify the impact of [19:22] AI on those outcomes that you care [19:24] about. Now it's really important to [19:27] understand that when I say the word [19:28] amplify, this is a birectional pathway. [19:32] So it doesn't just make things better, [19:35] it also makes things worse. And let me [19:38] explain. [19:39] If you are doing things well, AI makes [19:43] those things better. If you are doing [19:47] things not very well, AI makes things [19:51] worse. So this team performance is a [19:54] really great example. When you dig into [19:56] our research, what it shows is that [19:58] teams that are user centric, [20:02] AI then makes the team performance [20:05] better. [20:06] Teams that are not user centric, [20:10] AI introduction into the team makes team [20:13] performance measurably worse, [20:16] substantially worse. So I think that's [20:18] really important to to understand is is [20:21] this this amplification goes both ways. [20:23] So anyway, so we created this model to [20:26] help people focus on the outcomes that [20:28] they're looking for and really try to [20:30] dissect and understand, you know, [20:33] perhaps why aren't we getting the [20:35] results that we expected? Like code [20:38] quality as an example is a big one that [20:40] customers, you know, frequently I [20:42] frequently hear from people, hey, you [20:44] know, these tools are great. They're [20:46] helping us write code quickly, but a lot [20:48] of times the quality just isn't there. [20:50] Well, what our research shows is that in [20:52] order to get high levels of code [20:54] quality, you need to have internal data [20:57] that is accessible to your AI tools. So, [21:00] you know, if you're using code assist, [21:02] they need to be trained on your internal [21:04] repos in order to really produce good [21:07] quality code. as an example. [21:10] >> And as you can see, it doesn't just [21:12] affect code quality, it affects [21:14] individual effectiveness, it affects [21:17] product performance, [21:19] >> which makes sense because we've always [21:21] known garbage in garbage out data, [21:23] right? [21:25] >> That's right. Yep. And and so if you're [21:27] interested in [21:28] >> garbage, but something trained on the [21:30] world is not trained on your code. These [21:33] are difference. [21:35] >> That's right. Yeah, that's that's very [21:37] correct. Um, and then lastly, what I'll [21:39] say about this is um, we've also been [21:42] from a Dora research perspective, we've [21:44] been trying to put out more content and [21:46] and give more to the community. And so [21:48] we actually just recently released an [21:50] entire report on this model. Um, and so [21:54] that's the AI capabilities model that [21:56] you see there on the left hand side. Um [21:59] it is almost a hundred pages that goes [22:02] very in-depth into each of those [22:04] capabilities and the outcomes that they [22:05] drive. Shows all of the research, all of [22:08] the data. Um and so that's uh that's a [22:11] really great place uh to go if you want [22:13] to learn more about this model [22:16] >> and everything we're talking about is [22:18] not just this model and these things but [22:21] what it actually is in reality. So [22:24] pashant with that in mind like what are [22:28] you how does this ring true compared [22:30] with like what's because harness is a [22:33] company that uses AI you are still [22:34] technology company and what your [22:37] organization is experienced and what you [22:39] find your clients are going through. [22:41] Yeah. [22:43] uh loved the overview veric I think Dora [22:45] has been the golden standard for the [22:47] industry for a while right for any [22:50] leader as we have gone into a services [22:52] architecture uh if you are the owner of [22:55] a service uh and you're responsible for [22:58] the uptime the the health of these [23:00] services one of the key measures of this [23:02] is you know as as you um as as you own [23:05] the teams that build these services [23:07] maintain these services is the is the is [23:09] are the durometrics right so um and and [23:12] AI I sort of uh is uh is a is a is a is [23:17] a big event that's happening in the [23:18] industry. Um and there are many ways to [23:21] measure AI, many ways to understand AI, [23:23] but Dora continues to be one of those [23:25] golden metrics that absolutely continues [23:26] to matter, right? Because at the end of [23:28] the day, AI or or not, uh you are [23:30] measuring software delivery and that's [23:32] what Dora helps you measure, right? So I [23:34] think uh from my perspective you know I [23:36] spend a lot of time talking to a lot of [23:38] our customers who are um um on the hook [23:42] in so many ways right if if you imagine [23:44] being a SVP of engineering a CIO a CTO [23:48] today there is tremendous pressure on uh [23:52] us as leaders to to truly basically [23:55] embrace AI uh and show results show ROI [23:59] right money is no longer cheap over the [24:01] last many years money has become [24:03] expensive Right? Everything must be [24:05] proven. The ROI must be shown. Right? [24:07] And AI is such a transformational [24:09] technology right now that everybody is [24:11] is is expecting magic to happen. But [24:14] that magic needs to be proven. It needs [24:16] to be shown by hard numbers. Right? So [24:18] every uh leader today is saying look I [24:21] am making investments. Not enabling my [24:23] teams with the best in breed AI tooling [24:25] is not an option. Some teams are signing [24:28] up for one vendor. Most customers that I [24:31] talk to are have an open vendor policy. [24:33] Does not mean anybody can bring anything [24:35] but it means that as a best practice [24:38] they want to enable more and more coding [24:40] assistance for their users. See what you [24:43] know their developers love within sort [24:45] of an enterprise contract with the right [24:47] guard rails and and so on and so forth, [24:50] right? Um and as these investments begin [24:52] to happen, the the obvious question now [24:54] for the that engine leaders have to [24:56] answer back to the board, back to the [24:57] CFO is like we made all these [24:59] investments. What is the ROI? What tool [25:01] is worth investing more in? What tool is [25:04] truly providing us value? And so on and [25:05] so forth, right? And so um so it comes [25:09] down to measuring what tools developers [25:13] are using. It comes down to measuring [25:16] with the tools that these developers are [25:18] using. What impact are you actually [25:20] seeing in your SDLC? There is obviously [25:22] a very easy perception. Everybody feels [25:25] more productive, right? All of a sudden [25:26] you can prompt and code is written for [25:28] you, right? And test cases are written [25:30] for you and documentation can be [25:31] generated. That is a massively powerful [25:33] thing to happen. So there is an obvious [25:35] perception that my productivity has [25:37] significantly enhanced. [25:39] And yet as an organization to to [25:41] everything that Eric was mentioning um [25:44] it is not necessary that if everybody [25:46] begins to use tool on their own that as [25:48] an organization you're actually thriving [25:50] and you are moving towards becoming that [25:52] 10x organization that is not guaranteed [25:53] right so as leaders again you want to [25:56] begin to measure you want to begin to [25:58] understand that as an organization not [25:59] as an individual not as one team but as [26:02] an organization how well have we [26:03] embraced AI what is the impact I'm [26:05] seeing what is the positive impact but [26:08] sometimes what is the negative impact or [26:10] sometimes what is flatlined where I [26:11] expected obvious gains but I but I'm not [26:13] seeing those gains. So you want to begin [26:15] to measure. So I think Eric said [26:16] something where metrics are laggered [26:18] indicators uh and might be very less [26:20] interesting as compared to the actual [26:21] capabilities you want to develop. But [26:22] guess what right um unless until you [26:25] have those gauges that tell you where is [26:27] the slowdown where is that you should [26:29] focus on you don't know what capability [26:30] to start improving as an organization. [26:33] So installing that gauge becomes sort of [26:34] that you know first step you take begin [26:36] to measure things uh that opens you know [26:39] your mind that gives you eyesight into [26:42] what are the different bottlenecks [26:44] you're you're you're observing [26:45] especially again as you as you invest in [26:47] AI like I'll give a very crude example [26:50] uh and Pandora research also speaks to [26:52] this is right uh what what AI is doing [26:54] is that it's generating tons of code [26:56] right it's so easy to write code now [26:58] every engineer is creating more and more [27:00] and more code but that more code is [27:03] leading to first of all review [27:04] bottlenecks. If you have not as an [27:06] organization considered what does it [27:08] mean to review this massive amount of [27:10] code that is being generated now and [27:11] rethink your review processes there is a [27:13] review bottleneck. If the quality of the [27:15] code for whatever reasons because you [27:17] didn't train your models well enough [27:18] because you've not enabled your [27:19] engineers well enough if the quality of [27:21] that code is not great it's leading to a [27:23] lot of rework you have a rework [27:24] bottleneck right so being able to [27:26] identify these things and then finding [27:28] the capabilities you have to invest in [27:29] is the way I see sort of the industry uh [27:32] you know truly sort of panning on this [27:35] >> actually when we [27:37] those that are watching this have gone [27:39] through the process of signing up for a [27:41] lead dev webinar and there's quite an [27:44] openended a question about like what you [27:46] hope to get out of joining this [27:48] conversation. And the most common [27:52] response was [27:54] to summarize it because everyone writes [27:56] their own ideas was people were looking [27:59] for effective measurable ways to boost [28:01] productivity or to increase engineering [28:03] efficiency for technical teams. So this [28:06] does come down to the meas the metrics. [28:09] So I mean I we I definitely hope people [28:13] fill out Dora sometimes [28:15] uh once a year at least maybe more but [28:18] it is not a quick survey by any means. [28:23] It is quite detailed and thorough and [28:25] important in itself. But what are [28:28] especially when we have this whole idea [28:30] of empowering developers with autonomy, [28:33] but then this creates this whole AI [28:35] sprawl and maybe shadow AI because we [28:39] know if you try to ban something that [28:41] they want to use that it's not that [28:42] expensive. They'll just take out their [28:44] credit cards. Maybe we need to control [28:46] that. So, how can you measure the AI [28:50] adoption where the bottlenecks are and [28:52] the impact on your developers? What are [28:54] some of the key metrics? [28:57] >> Is that a question for me, Jennifer? [28:59] >> For either of you? Yeah. What have you [29:01] been finding that is some powerful [29:03] metrics either for your team or your [29:04] customers? [29:06] >> Yeah, I can take a first crack, Eric, [29:07] and then you can add to that. [29:08] >> Sure. [29:08] >> Yeah, please. [29:09] >> Um, [29:10] >> so I think uh yeah, I think uh the first [29:12] thing like what does it mean to measure [29:14] AI, right? Um there are two broad [29:16] categories. What is the adoption of [29:18] these tools and then what is the impact [29:20] of those tools and we'll we'll come to [29:21] both. Right? So to measure adoption of [29:25] these tools obviously you want to start [29:27] looking into saying a how many users are [29:30] using these tools what kinds of tools [29:33] but then using this tool does not [29:35] guarantee that you're actually using it. [29:37] Right? Just having a license and you [29:40] know once in a while prompting something [29:41] in the tool does not mean you're [29:42] effectively using the tool. So then you [29:44] have to go within these tools and begin [29:46] to understand how much of the code that [29:48] you're actually uh committing to your [29:50] repositories is actually coming from [29:52] these AI assistants, right? Uh [29:54] interestingly that is a hard problem to [29:56] solve today. The state [29:58] >> how do you measure that? Like are you [30:00] just asking developers or are you using [30:02] tools to measure the percentage of code [30:06] that AI was involved? Because maybe [30:08] developers don't feel safe saying [30:09] whether or not they're using AI or not. [30:11] Side note, they are. [30:14] I think every yeah fortunately every [30:16] coding assistant on the market today uh [30:18] exposes uh data as well as APIs for uh [30:22] you know customers to gather insights [30:24] for you know per user per team per [30:26] organization. How much prompting are [30:29] users doing? How many lines of code is [30:32] is an is an agent generating? How much [30:34] of that line how much of that code being [30:36] generated is being accepted by the users [30:38] and actually becoming part of their [30:39] code. Right? Uh so so there's a lot of [30:41] data that these assistants provide that [30:43] for example a tool like SEI we talk to a [30:46] cloud code we talk to a windsurf a [30:48] cursor a Gemini code it says right [30:49] different tools and pull in this data so [30:52] that provides that common visibility of [30:54] adoption and usage of these tools to our [30:57] customers um but the but the hard part [31:00] is once that data is accepted right uh [31:02] let's say I prompted winds and winds [31:05] generated code and I accepted that code [31:07] there is no way to guarantee that that [31:09] code was then not modified or lines of [31:11] code were not deleted by the developers. [31:13] Right? And that's where there is a [31:14] little bit of a flux in the industry [31:16] today where truly measuring uh truly [31:18] tagging every line of code like imagine [31:20] if git had a metadata for every line of [31:22] code which said this was purely AI [31:24] generated versus this was AI generated [31:26] and then erit versus you know some of [31:27] the code that was only manually created. [31:30] Uh so that level of sort of [31:31] instrumentation for code does not exist [31:34] uh today but I have no doubt that we all [31:35] heading towards that uh if we have to [31:37] deeply measure this. That's just the [31:38] measurement of adoption and usage. And [31:41] then there is the impact part. And if [31:42] you really think of impact part, what [31:44] you what you want to measure is [31:45] obviously like like why am I using [31:47] because I want to be faster. I want to [31:48] be generating more code. Uh what is the [31:51] improvement in your code velocity is the [31:53] first thing you want to measure. And [31:54] this is this is expected to be typically [31:56] very positive gains, right? But velocity [31:59] often can come [32:00] >> velocity velocity at what point? [32:04] the you could you could look at [32:06] >> yeah you could look at velocity at a [32:08] granular level in terms of the PRs that [32:10] developers are generating are am I [32:11] creating a lot more PRs than I used to [32:14] am I writing a lot more lines of code [32:15] than I used to but more importantly from [32:17] a business perspective am I actually [32:19] resolving more bugs and am I delivering [32:20] more features and capabilities in the [32:22] product right has the level of features [32:24] that I was delivering has that actually [32:26] improved so that is feature velocity PR [32:28] velocity bug velocity that tools like [32:30] SEI help you measure uh but to complete [32:33] that point Jennifer I think velocity can [32:35] often come at the cost of quality right [32:36] especially in the age of AI uh if you're [32:39] not reviewing this code very well then [32:41] often the number of bugs coming is is [32:43] high but also like Eric mentioned it [32:46] might be leading to a lot of rework how [32:48] do you measure the rework that is being [32:50] created because of all this uh you know [32:53] AI code that that is coming in and then [32:55] finally it's not just quality of the [32:57] code now but it's the long-term [32:58] maintainability of the code you know [33:00] it's it's uh it's AI is many times not [33:02] explainable [33:03] A lot of times the code that is written [33:06] uh if if you know an year later [33:08] developers had to go fix things and AI [33:10] was not fixing it, chances are you might [33:12] not understand what's in there. What is [33:13] the long-term maintainability of this [33:15] code? What is the quality of this code? [33:16] Right? So there are many ways to sort of [33:18] think about this problem uh that leaders [33:20] today are looking to measure and that's [33:21] where again tools like SEI software [33:23] engines insights come in and try to [33:25] provide those uh those insights. [33:29] >> Okay. So, so, um, c can I respond to to [33:32] that one? I I have some some thoughts on [33:34] this, and I'll and I'll start by, uh, [33:36] summoning one of my favorite South Park [33:38] episodes, the underpants gnomes, if you [33:40] all remember that, where it was like [33:42] step one, collect underpants, step [33:43] three, profit. So, I think a lot of [33:47] customers and organizations treat AI in [33:50] in a very similar way where they think [33:52] step one, adopt AI tools, step three, [33:55] profit. the the the problem is is that [33:59] there's that there's a middle part. Um [34:01] so there's adopting the tool and then [34:04] you have to build trust in in the tool [34:07] with the developers and only until you [34:10] have built trust will you then start to [34:12] see developer productivity. And and [34:15] there's some additional nuance here [34:16] because the way that you measure [34:18] adoption is going to be different than [34:20] the way that you measure trust both from [34:22] a metrics perspective and also at the [34:25] the level at which you measure it. So [34:27] most of these will be at the individual [34:29] level. So when you're adopting the tool, [34:31] you want to see are people using the [34:33] tool? Are they using the licenses? You [34:35] know, are they firing it up? You know, [34:37] yada yada. When you get into trust, you [34:40] need to understand, you know, things [34:42] like the percentage of of code [34:44] committed, lines of code committed from [34:46] AI to see whether they're they are [34:48] actually having trust in the tool. This [34:51] process can take between a month to two [34:52] months to actually build trust in these [34:55] systems. And only after you've done that [34:57] are you going to be able to reach [34:59] developer productivity. Developer [35:02] productivity though is now measured at [35:04] the team or the application level. I was [35:07] going to say developers tend to really [35:09] not like well probably any human being [35:11] to be monitored and measured at level [35:14] especially in a less secure job market. [35:18] >> Yeah. And and it's just not helpful, [35:20] right? It doesn't really actually tell [35:21] you anything that's important. You know, [35:24] it would it gives you data, but does the [35:26] data matter? And I would argue very [35:28] strongly that it doesn't. So in addition [35:30] to all of the things that pashant [35:32] mentioned around velocity and ticket [35:34] closures um you can also use dorometrics [35:37] as a measure of productivity but of [35:39] course again that's at the application [35:41] or service level. But that brings us [35:43] into a new and and interesting situation [35:46] because a lot of companies are adopting [35:49] AI. They're building trust in AI but [35:52] they're not getting productivity out of [35:54] AI. And and a lot of times when you zoom [35:58] in and you really get try to get an [35:59] understanding of why that is, what I'm [36:02] seeing is that companies are investing [36:05] what seems like over 100% of all of [36:08] their time, resources, and effort into [36:11] AI adoption, [36:13] but they're not continuing to focus on [36:16] the fundamentals of the underlying [36:18] platforms and the underlying pipelines. [36:21] You know, and like Pashant mentioned, [36:23] you're now producing a lot more code, [36:26] and a lot of times it's code that you [36:28] don't even fully understand, but yet [36:31] companies haven't spent the time to [36:33] improve their testing automation, their [36:37] uh, you know, their documentation [36:39] quality, their continuous integration. [36:42] And so now what you have these systems [36:44] that are fragile and already not doing [36:47] well. I mean, I don't know about you, [36:48] Pashant, but I've literally never worked [36:50] with a customer in the last 10 years [36:52] when I asked them the question, "Is your [36:54] testing automation great?" None of them [36:57] say yes. [36:58] >> And now there's more code to [37:01] >> Now there's more. And we're not even [37:02] talking about agents yet, right? [37:05] >> We're just talking about code assist [37:06] tools for the most part. When you start [37:09] getting into agents now, it's like one [37:12] developer to, you know, how many a how [37:15] many developers is an agent? It could [37:18] almost be an infinite amount. And so [37:20] unless you have these underlying [37:22] fundamentals in place and and really one [37:25] of the keys to this is that and and this [37:27] is shown in our research from last year [37:30] is the the internal developer platform [37:33] is the key to unlocking organizational [37:37] performance with AI. So you know you can [37:41] get and pashant was talking about this [37:43] earlier you can get individual [37:45] performance you can get team performance [37:48] but to really get organizational [37:50] performance you need a well-built [37:53] internal developer platform and most [37:56] people that I interact with they have [37:58] internal developer platforms but almost [38:01] none of them have well-built and and you [38:05] know properly built internal developer [38:07] platforms. So, I I definitely see that [38:09] as as something that is a is a friction [38:12] point now and it's just going to get [38:14] exponentially worse the more that that [38:16] organizations adopt uh agents to do the [38:20] bulk of the work. [38:22] >> Absolutely. And uh for those that are [38:24] joining us live, I've enacted a poll, if [38:27] that's the verb, uh which best describes [38:30] where your organization is on the [38:31] platform engineering journey. And we've [38:33] got a well hopefully uh well we did but [38:37] then our budgets were cut and then [38:39] following the cloudnative computing [38:41] foundations platform maturity model [38:44] we've got so please do answer if you [38:46] want uh we're talking about provisional [38:48] which is custom processes by request ad [38:50] hoc totally appropriate in a startup [38:54] setting where you've got less than 50 [38:56] engineers you still have that close [38:58] context and all totally fine then we've [39:00] maybe got the operational maybe you [39:02] start to standardized tooling centrally [39:04] tracked by a dedicated team. Then maybe [39:07] you have the reach the level of [39:09] scalability with platform as a product. [39:11] You're treating your [39:13] internal developers like customers. You [39:15] have the self-service mechanism. So [39:18] because you can't have a platform team [39:20] scale to the amount of developers you [39:22] have. Uh you have insights metrics. We [39:25] clearly like metrics on the call. And [39:27] then the goal the end goal of your [39:30] golden path you're [39:33] is to have this reach this level of [39:35] optimization where you're balancing both [39:37] qualitative and quantitative metrics. [39:39] You have integrated managed services [39:43] enabled ecosystem and then you still [39:46] have that like platform as a product [39:48] which might be even having internal [39:50] marketing or demo days things like that. [39:53] So uh while we see where our audience is [39:58] on that uh I would recommend making it [40:00] wide and I wrote long responses uh [40:04] pashant so how are you finding as a [40:07] platform company and you're not the only [40:09] one by any means and there is this idea [40:12] that maybe you start out by building a [40:15] little something or a wiki to start but [40:18] as it evolves can you offer some use [40:22] cases is of how like we're really [40:24] finding platform engineering as a [40:26] conduit of AI. So [40:29] >> yeah, absolutely. I think um one of the [40:32] taglines we use at harness is that [40:33] harness is AI for everything after code, [40:36] right? If you really think about SDLC, [40:39] um it's it's almost funny to think about [40:41] the fact that, you know, developers [40:43] today are spending 30 to 40% of their [40:45] time actually developing code, you know, [40:48] designing things, writing specs, [40:50] actually writing code in git uh and and [40:53] working on on the on the truly thing [40:55] that sort of uh you know gets them [40:57] excited and and gets their juices [40:58] flowing. And fortunately unfortunately [41:00] 60 to 70% of our time as developers [41:02] today goes into so much of the software [41:05] delivery aspect that is just necessary [41:08] friction to deliver software. It is [41:10] building of my software. It is running [41:12] you know a battery of different kinds of [41:14] test suites. It is you know making sure [41:16] my software is secure. Running code [41:18] vulnerability scans running container [41:20] scans running cloud scans running API [41:22] scans. It's about then being able to [41:25] deploy this thing. Uh being able to [41:27] handle deployment failures. It's about [41:29] managing cloud cost. A huge part of [41:31] today's engineing organizations is that [41:33] every time I deploy I I update a new [41:35] feature, it has incrementally led to [41:37] significant more cloud cost that we are [41:39] incurring [41:40] >> and AI more we would assume [41:42] >> and you need governance. You need [41:44] controls to sort of cap that in some in [41:46] some capacity because the easiest thing [41:47] for anybody to do is to make sure my [41:49] service is healthy. I'm going to spit [41:51] spin up 50 instances of this container. [41:53] Sure. But now you're burning money, [41:55] right? So how do you sort of um so a [41:58] platform essentially allows you to [42:00] manage all these different parts of [42:03] SDLC. It's so much more than just [42:05] writing code, right? Um and uh harness [42:09] for example today is in is has [42:11] introduced AI into every aspect of this [42:13] this delivery life cycle, right? So we [42:15] have a DevOps agent as an example that [42:17] helps you build better pipelines, debug [42:21] those pipelines, create pipelines [42:23] automatically for the code that you want [42:24] to deploy. Um we have uh a dev sec ops [42:28] agent that automatically for every [42:30] service that you're deploying go goes to [42:31] the code, runs the different scans, [42:34] identifies vulnerabilities, but then [42:36] because well AI uh is now actually [42:39] creating PRs to fix those [42:41] vulnerabilities, right? So how do you [42:43] build bring in that level of automation [42:45] where you know automatically you [42:46] discover vulnerabilities but then you [42:47] automatically fix those vulnerabilities. [42:49] We have a cloud cost agent that is [42:51] constantly left shifting the notion of [42:53] cost every time you're building [42:55] something new. How do you left shift a [42:56] construct of cost and and be able to [42:58] provide early indicators to say that you [43:00] know here are some improvements you can [43:02] make to to better control the cost [43:03] you're running. Um we have a S sur agent [43:07] that is part of SDLC now that every time [43:09] you deploy something it is constantly [43:12] looking at the service that you've [43:14] deployed the uptime it is looking for [43:15] signatures uh how healthy is the service [43:18] looking if it's not automatically create [43:20] incidents automatically diagnose those [43:22] incidents right so there is just so much [43:25] you can do as a platform right and and I [43:27] think to bring this home uh one thing [43:29] that I want to call out is the real [43:31] power of platforms is that it has [43:33] context text to every in in your [43:36] organization for you as a as an account. [43:38] It understands everything you built, [43:40] everything you deployed, every feature [43:42] flag you changed, every um metric that [43:45] you're measuring, every incident that [43:47] happened, every cloud cost uh action [43:49] that you're taking. When you have this [43:51] holistic view into every action you take [43:54] as part of your SDLC, the way AI can [43:57] truly understand and now begin to become [43:59] your mentor, begin to make [44:01] recommendations, begin to optimize all [44:02] of this is just so much more immensely [44:04] powerful than individual tools you might [44:06] bring in, right? And that is where [44:07] platforms really begin to shine because [44:09] it has a true semantic understanding of [44:11] everything you do from an SDLC [44:13] standpoint. that that's such an amazing [44:16] point that I have also been really [44:18] trying to pound the table around and I [44:20] refer to the IDP uh or the platform as [44:23] the center of the universe for [44:26] everything that's happening in the [44:27] organization because you have the [44:30] documentation, you have all of the the [44:32] previous uh uh things that have been [44:34] tried and and that have worked and maybe [44:36] not worked. You have all of the the [44:38] engineering metrics. You you have all of [44:40] the code, all of the documentation, the [44:42] service catalog, all of the golden [44:44] paths. I mean, it's it's wild. And [44:48] having that context allows you to do so [44:51] much more in a fully automated way. [44:56] >> Absolutely. And with the searchable [44:59] context as well, which I think is very [45:01] important, people [45:03] and especially developers maybe don't [45:05] want to ask people the first time they [45:07] want to do something. They do want to be [45:09] self-service. So that especially if [45:11] they're new to an organization so that [45:13] when they're asking a question, it's an [45:15] important one, not a who owns that or [45:18] how do I find this? Because guess what? [45:22] it's frustration is and finding things [45:25] is like 20% of a developer's day and I [45:28] assume anyone joining a company not just [45:30] someone in a technical role. So for [45:33] those that are on this call, uh the [45:37] platform journey is [45:40] really all over the place, I would say. [45:43] We've definitely have a varying amount [45:45] of people and on this platform journey, [45:48] I wish you luck because I think we've [45:50] found very clearly with this year's [45:52] Storm Report, it is the best way and the [45:58] most successful way. the these two [46:00] trends have converged and it's the way [46:02] to deliver AI now and then uh good luck [46:06] on your journey. Uh we will continue [46:09] this conversation in Slack and thank you [46:11] so much to our active listeners and [46:13] those listening afterwards and [46:14] especially thank you so much to Pashant [46:17] and Eric. Good luck on your what is a [46:21] journey not a destination with AI. [46:23] Thanks y'all. [46:24] >> Thanks everyone. [46:36] I'm on the  [[3. Resources/Readwise/Full Document Contents/Articles/The AI Capabilities That Amplify Your Org|Full Document Contents]] ## Document Notes --- tags: #AI-assisted-development #platform-engineering #DevOps #developer-productivity author: Lead Dev source: YouTube reference: https://www.youtube.com/watch?v=ik7YkVRZxxE date created: 2024-12-28 20:30 --- # 🏷️ Title: The AI capabilities that amplify your org ## 🧰 Action Inbox - [H1] Fill out DORA survey when available — annual industry benchmarking survey #measurement - [H2] Assess internal developer platform maturity — use CNCF platform maturity model #platform-engineering - [H3] Measure AI adoption beyond licenses — track code acceptance rates and quality metrics #AI-metrics ## 🧭 Table of Contents - [[#📝 Summary]] - [[#📖 Main Content]] - [[#🗒️ Notes]] - [[#📚 References]] - [[#🗣️ Paraphrase]] - [[#📰 Newsletter]] - [[#✅ Action Items]] - [[#🧪 Hypotheses]] - [[#📊 Methodology]] - [[#❓ Questions]] - [[#🔮 Future Research]] - [[#💡 Implications]] - [[#🧠 Learning Styles Assessment]] - [[#📊 Accessibility Assessment]] - [[#🌍 DEIB Assessment]] - [[#✅ Logical Validity]] - [[#🛡️ DARVO Analysis]] - [[#🎯 Conclusion]] ### 🔗 References - [DORA Research](https://dora.dev) - [2024 State of AI Assisted Software Development](https://dora.dev/research/) ## 📝 Summary - Engineering leaders discuss how AI amplifies organizational capabilities bidirectionally - making good practices better and poor practices worse - DORA research now includes five core metrics (adding rework) and introduces an AI capability model for outcome-focused improvement - Platform engineering emerges as critical foundation for organizational AI success, providing context and automation across the entire SDLC - Key stakeholders are engineering leaders, CTOs, and platform teams navigating AI adoption and ROI measurement - Main risk is investing 100% effort in AI adoption while neglecting underlying platform fundamentals like testing automation and CI/CD ## 📖 Main Content The discussion centers on how AI capabilities amplify organizational performance in both directions - enhancing strengths while exacerbating weaknesses. Eric Maxwell from Google's DORA team explains that their research now focuses on an AI capability model that helps organizations identify which capabilities to develop based on desired outcomes. The bidirectional amplification concept is critical: teams that are user-centric see improved performance with AI, while non-user-centric teams experience substantially worse performance. This pattern applies across code quality, team effectiveness, and product outcomes. Pashant Batra from Harness emphasizes that measuring AI impact requires looking beyond simple adoption metrics to understand actual usage patterns and business outcomes. Organizations need to track not just velocity gains but also quality impacts, rework rates, and long-term maintainability. Platform engineering emerges as the key enabler for organizational-level AI benefits. While individuals and teams can gain productivity from AI tools, achieving organizational performance requires a well-built internal developer platform that provides context, automation, and governance across the entire software delivery lifecycle. ## 🗒️ Notes #### ⏱️ [00:00](https://youtu.be/ik7YkVRZxxE?t=0) — [H1] Introduction and Panel Setup - Key idea: Lead Dev panel on AI capabilities with Google DORA and Harness product leaders - Evidence: "Discussion on how AI capabilities amplify your organization" #### ⏱️ [03:42](https://youtu.be/ik7YkVRZxxE?t=222) — [H2] Pashant Introduction - Software Engineering Insights - Key idea: Harness focuses on empowering engineering leaders with deep insights into team efficiency - Evidence: "Everything begins with measurement and then using those measured metrics to drive actions, drive transformation" #### ⏱️ [05:15](https://youtu.be/ik7YkVRZxxE?t=315) — [H3] Eric Introduction - 10X Technology at Google - Key idea: 10X refers to amplifying companies through technology, not individual engineers - Evidence: "10X is something we talk about quite famously inside Google. It's not really about the engineer. It's more about how can you 10x a company through the use of technology" #### ⏱️ [07:30](https://youtu.be/ik7YkVRZxxE?t=450) — [H1] DORA Framework Overview - Key idea: DORA now includes five core metrics with rework as the newest addition - Evidence: "As of this last year of research 2025, it's now five capabilities or five metrics. We added rework to the list" #### ⏱️ [12:45](https://youtu.be/ik7YkVRZxxE?t=765) — [H2] DORA Capability Model Explanation - Key idea: Metrics are lagging indicators; capabilities on the left drive the outcomes on the right - Evidence: "The metrics are actually the least interesting part of the whole DORA research. What our research really focuses on is the blue box on the right hand side, the outcomes" #### ⏱️ [18:20](https://youtu.be/ik7YkVRZxxE?t=1100) — [H1] AI Capability Model Introduction - Key idea: New AI capability model shows bidirectional amplification effects - Evidence: "If you are doing things well, AI makes those things better. If you are doing things not very well, AI makes things worse" #### ⏱️ [21:15](https://youtu.be/ik7YkVRZxxE?t=1275) — [H3] User-Centric Teams Example - Key idea: AI amplification depends on underlying team characteristics - Evidence: "Teams that are user centric, AI then makes the team performance better. Teams that are not user centric, AI introduction makes team performance measurably worse, substantially worse" #### ⏱️ [24:30](https://youtu.be/ik7YkVRZxxE?t=1470) — [H2] Industry Pressure for AI ROI - Key idea: Engineering leaders face tremendous pressure to show measurable AI returns - Evidence: "Every leader today is saying look I am making investments. Not enabling my teams with the best in breed AI tooling is not an option" #### ⏱️ [28:45](https://youtu.be/ik7YkVRZxxE?t=1725) — [H2] AI Adoption Measurement Challenges - Key idea: Need to measure beyond licenses to actual code acceptance and modification rates - Evidence: "Just having a license and you know once in a while prompting something in the tool does not mean you're effectively using the tool" #### ⏱️ [32:10](https://youtu.be/ik7YkVRZxxE?t=1930) — [H3] Code Provenance Problem - Key idea: Industry lacks proper tagging system for AI-generated vs human-modified code - Evidence: "Truly tagging every line of code like imagine if git had a metadata for every line of code which said this was purely AI generated versus this was AI generated and then edited" #### ⏱️ [35:40](https://youtu.be/ik7YkVRZxxE?t=2140) — [H1] Trust Building Process for AI Tools - Key idea: Three-stage process: adoption, trust-building, then productivity gains - Evidence: "Step one, adopt AI tools, step three, profit. The problem is there's a middle part... This process can take between a month to two months to actually build trust" #### ⏱️ [39:15](https://youtu.be/ik7YkVRZxxE?t=2355) — [H1] Platform Engineering as AI Foundation - Key idea: Internal developer platforms are key to unlocking organizational AI performance - Evidence: "The internal developer platform is the key to unlocking organizational performance with AI... most people have internal developer platforms but almost none of them have well-built ones" #### ⏱️ [42:30](https://youtu.be/ik7YkVRZxxE?t=2550) — [H2] Harness AI Integration Across SDLC - Key idea: AI agents can automate entire software delivery lifecycle beyond just coding - Evidence: "Harness is AI for everything after code... developers are spending 30 to 40% of their time actually developing code, 60 to 70% goes into software delivery" #### ⏱️ [44:50](https://youtu.be/ik7YkVRZxxE?t=2690) — [H3] Platform Context Advantage - Key idea: Platforms with holistic SDLC context enable more powerful AI automation - Evidence: "When you have this holistic view into every action you take as part of your SDLC, the way AI can truly understand and now begin to become your mentor... is just so much more immensely powerful" ## 📚 References - [DORA Research](https://dora.dev) - [AI Capabilities Model Report](https://dora.dev/research/) - [Lead Dev Community Slack](https://leaddev.com/slack) - [CNCF Platform Engineering Maturity Model](https://tag-app-delivery.cncf.io/whitepapers/platform-engineering-maturity-model/) ## 🗣️ Paraphrase - AI acts as an amplifier for existing organizational capabilities, making strong practices stronger and weak practices weaker, particularly around user-centricity and code quality - DORA research has evolved to include rework as a fifth core metric and developed an AI capability model that maps specific capabilities to desired organizational outcomes - Measuring AI success requires tracking adoption, trust-building, and actual business impact rather than just license usage or developer perception of productivity gains ## 📰 Newsletter **AI Amplifies What You Already Do - Make Sure It's Good** The latest research from Google's DORA team reveals a critical insight: AI doesn't just make you better - it makes you *more of what you already are*. If your team is user-focused and has solid engineering practices, AI will accelerate your success. But if you're struggling with fundamentals, AI will amplify those problems too. The key? Build your platform engineering foundation first. Organizations seeing real organizational gains from AI have well-built internal developer platforms that provide context, automation, and governance across the entire software delivery lifecycle. **Practical Steps:** - Assess your platform maturity using the CNCF model - Measure AI adoption beyond licenses - track actual code acceptance rates - Focus on capabilities that drive your desired outcomes, not just tool adoption Resources: [DORA Research](https://dora.dev), [Platform Engineering Guide](https://platformengineering.org), [CNCF Platform Maturity Model](https://tag-app-delivery.cncf.io/whitepapers/platform-engineering-maturity-model/) ## ✅ Action Items #### DevRel - Developer advocacy: Create content explaining bidirectional AI amplification effects, present at engineering conferences - Education: Develop workshops on AI capability assessment, platform engineering fundamentals - Developer experience: Improve documentation for internal AI tools and platform onboarding - Developer marketing: Campaign around "AI amplifies what you do" messaging - Community management: Facilitate discussions on AI adoption challenges in developer forums #### OSPO - Policy & Governance: Establish AI tool evaluation criteria and approval processes - License Compliance: Review AI coding assistant terms and enterprise agreements - Security Management: Assess vulnerabilities in AI-generated code, implement scanning protocols - Software Inventory: Track AI tool usage and generated code provenance - Developer Support: Provide training on responsible AI tool usage and quality practices - Community Engagement: Fund research on AI impact measurement and platform engineering - Strategic Alignment: Define KPIs for AI ROI measurement and platform maturity progression #### Community - Maintainers: Support development of AI capability assessment tools and platform templates - Contributors: Create onboarding programs for AI-assisted development workflows - Sponsors: Fund research into AI amplification effects and measurement methodologies - Ecosystem: Coordinate standards for AI tool interoperability and code provenance tracking #### WordPress - Core: Implement AI-assisted code review processes, establish quality gates for AI-generated contributions - Core Performance: Profile AI-generated code impact on performance budgets and optimization - Core AI: Develop evaluation frameworks for AI tooling integration and quality assessment - Core Program Team: Coordinate AI adoption strategy across teams, track capability development - Community: Create educational content on responsible AI usage in WordPress development - Training: Develop curriculum on AI-assisted WordPress development best practices - Documentation: Use AI to improve documentation quality while maintaining human oversight - Meta: Assess platform engineering maturity for WordPress.org infrastructure #### The WPCC - Programs: Establish fellowships for AI research and platform engineering improvements - Projects: Fund development of AI capability assessment tools for WordPress ecosystem - Funding: Create grants for organizations improving their platform engineering maturity #### FAIR - Technical independence: Develop standards for AI tool evaluation and code provenance tracking - Policies: Establish guidelines for transparent AI usage and quality measurement - Governance artifacts: Create templates for AI governance and platform engineering charters ## 🧪 Hypotheses - Organizations with mature internal developer platforms will see 2-3x greater organizational benefits from AI adoption compared to those with ad-hoc tooling - Teams scoring high on user-centricity metrics will show measurably better outcomes from AI tools within 3-6 months of adoption - Companies investing exclusively in AI tools without improving underlying CI/CD and testing infrastructure will see increased instability metrics - The time to build trust in AI tools (1-2 months) correlates with the quality of internal documentation and platform self-service capabilities ### 📊 Methodology - Longitudinal study tracking DORA metrics before/after AI adoption across organizations with varying platform maturity levels - Survey research measuring user-centricity scores and correlating with AI outcome metrics over 6-month periods - Case study analysis of organizations experiencing AI-related instability vs. those seeing productivity gains ### ❓ Questions - How can organizations systematically assess their readiness for AI amplification before tool adoption? - What specific platform engineering capabilities have the highest correlation with successful organizational AI outcomes? - How might code provenance tracking be implemented at scale without creating developer friction? ### 🔮 Future Research - Development of standardized AI readiness assessment tools for engineering organizations - Investigation into optimal sequencing of platform improvements and AI tool rollouts - Research into developer experience factors that accelerate trust-building with AI tools ### 💡 Implications - Practice: Organizations should assess platform maturity before major AI tool investments - Policy: Need for governance frameworks that address AI tool sprawl and quality control - Security: AI-generated code requires enhanced review processes and vulnerability scanning - Education: Training programs must cover platform engineering alongside AI tool usage - Funding: Budget allocation should prioritize platform foundations over pure AI tool licensing ### 🧠 Learning Styles Assessment Reading level estimate: 12th grade (Flesch-Kincaid) #### 👀 Visual learners - Present: DORA capability model diagrams, platform maturity progression charts - Improve: Add AI amplification effect visualizations, platform architecture examples #### 🔊 Aural learners - Present: 45-minute panel discussion with expert insights - Improve: Create 5-minute audio summary of key amplification concepts #### 🗣️ Verbal learners - Present: Clear definitions of platform engineering, bidirectional amplification - Improve: Add more analogies for AI amplification effects, glossary of terms #### 🧩 Kinesthetic learners - Present: References to hands-on platform assessment tools - Improve: Create interactive AI readiness checklist, platform maturity self-assessment #### 🧠 Logical learners - Present: Clear cause-effect relationships between platform maturity and AI outcomes - Improve: Add decision trees for AI adoption timing, measurement frameworks #### 🧑‍🤝‍🧑 Social learners - Present: Panel discussion format, Slack community engagement - Improve: Create cohort-based AI adoption programs, peer review rubrics #### 🧘‍♀️ Solitary learners - Present: Detailed research reports, self-assessment opportunities - Improve: Add reflection prompts on organization's AI readiness, journaling templates #### 🌿 Naturalistic learners - Present: Systems thinking approach to platform engineering - Improve: Emphasize ecosystem impacts of AI tool choices, sustainability considerations ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Clear heading structure, descriptive link text for research reports - Improve: Add alt text for D --- ## Full Transcript [00:12] Hello, hello and welcome everyone. Thank [00:16] you both so much for joining us live [00:18] today and uh for those that are joining [00:22] in the audience live or those listening [00:23] the recording for a discussion and this [00:27] is a lead dev panel created in [00:29] partnership with harness. It'll last [00:32] roughly 45 minutes after which both [00:34] myself and the panelists will head over [00:37] to the lead dev Slack to answer more of [00:40] your questions in the hashtag I guess we [00:44] say hashtag for it panel Q&A channel and [00:48] to continue discussion if you aren't [00:51] already a Slack member yet for those [00:55] live I'm just shooting you right now the [00:58] Slack invitation it's a wonderful [01:01] community that it's just a way to [01:04] continue discussion with engineering [01:06] leadership and also we welcome the [01:09] voices of course of developers if they [01:11] want to you know [01:14] uh add their thoughts as well because we [01:17] need to be listening to you which will [01:19] be part of the theme of today's and all [01:22] conversations for those of us joining us [01:24] live feel free to just share your [01:26] questions in the zoom and I will ask [01:30] them and then if we have extra questions [01:33] because we have a good community joining [01:34] us today it will continue in the Slack. [01:37] So let's get started. I'm Jennifer [01:38] Rigggins. I'm a technology journalist [01:41] and today I'll be joined by Eric Maxwell [01:44] head of 10X technology at Google and [01:47] Pashant Batra or sometimes B PB uh [01:51] senior director of product management at [01:54] Harness at Harness. Uh pashant can you [01:57] please introduce yourself and what [01:59] you're most excited to talk about today [02:06] you are on mute because you know [02:11] >> had to happen right [02:12] >> had to happen [02:14] >> well good morning good afternoon good [02:16] evening everyone uh I am a senior [02:18] director of product at harness uh I've [02:21] been in the industry for 20 plus years [02:23] as you can imagine over that time I've [02:25] built many different products from uh [02:27] building my first VPN to gateways to [02:30] mobile device management software to um [02:34] CASBY cloud access security brokers to [02:36] my last gig where I was at Nutanix where [02:39] we were building a hybrid cloud platform [02:42] uh and then sort of my five years at [02:43] harness that I've spent I've built a few [02:45] different things uh you know off late [02:47] what I do is I lead one of our product [02:49] lines called software engineering [02:51] insights uh and just as a as a quick [02:54] sort of uh overview of what that means, [02:56] right? So, software engineering insights [02:58] is a product line from harness uh that [03:02] essentially uh aims to empower [03:05] engineering leaders today with deep [03:07] insights into you know how well are [03:10] teams building software, how efficient [03:12] are the processes you're running, how [03:14] productive are teams, how well aligned [03:16] is your execution to the outcomes that [03:17] you're looking for, how well are you [03:19] using AI, right? Everything begins with [03:21] measure measurement and then using those [03:24] measured metrics to drive actions, drive [03:27] transformation, recommend changes, [03:29] right? And so that is what uh this [03:31] product line does. U and I'm really [03:33] excited to sort of uh you know chat with [03:35] Jennifer Eric today and talk about how [03:38] we see the advent of AI in the industry [03:40] and how uh you know modern day platforms [03:42] like harness software engineing insights [03:44] a lot that Eric is working on are truly [03:46] sort of participating in this [03:47] transformation. [03:49] >> Absolutely. Thank you so much. Nice to [03:50] have you. And Eric, can you please [03:52] introduce yourself and then talk about [03:54] your job title because kind of famously [03:57] in the lead dev community, we do not [04:00] love the term 10x engineer. So maybe [04:03] talk about what 10x technology is as [04:05] well [04:06] >> at Google effect. [04:08] >> Yeah. No, no, no problem. Um, so I'll [04:11] just give a little background. I started [04:13] off my career as an engineer and I spent [04:15] most of my time doing back-end uh API [04:18] engineering. mostly working in like the [04:20] the middle layer and and the back end. [04:22] Around 2010 2011, I started really [04:26] getting into automation. Um, not because [04:29] I was trying to do DevOps, but I was one [04:32] of three people at a startup company [04:34] that became webcale. Um, and so we just [04:38] considered that survival, uh, rather [04:40] than putting any, you know, terms or [04:42] tags around it. So, um, I kind of got [04:44] into the game early on and then around [04:47] 2015, I started working at Chef Software [04:50] and that's when I really started working [04:51] with customers and and helping them do [04:55] all the things that I had already had [04:57] like a 5-year jump start on, which was [05:00] pretty cool. Um, and that led me to [05:02] Google. So, I've been here at Google now [05:04] for almost eight years. I'm part of the [05:06] the Dora team. So I'm one of the [05:09] researchers on DORA and I'm also one of [05:12] the authors of of the reports that we [05:14] put out. Um and then so what I wanted to [05:17] do at Google was take the research that [05:20] we do because that really is you know [05:22] where my my passion lies. I really enjoy [05:24] that aspect of it and I wanted to figure [05:26] out how I could take that research and [05:28] bring it to our customers. Um so [05:31] thankfully I had the opportunity to [05:33] start a uh consulting practice on the [05:36] Google cloud consulting side uh which we [05:38] call 10x technology um for better or [05:41] worse but uh so what what we do [05:43] specifically is we we focus on bringing [05:46] door research to our customers. So you [05:48] know we we focus on developer [05:50] experience, how to increase [05:52] productivity, how do you build [05:54] engineering excellence? Of course, a big [05:56] part of that is the cultural aspect and [05:58] then from a technology perspective, we [06:00] focus on, you know, DevOps, AI tooling, [06:04] uh, and of course, lots of platform [06:06] engineering. So, um, nice to be here and [06:09] and to address the the 10X part, 10X is [06:12] something that we talk about quite [06:14] famously inside of Google, actually. Um, [06:17] and so it's not really uh a mention [06:19] about the engineer. It's more about how [06:22] can you 10x a company through the use of [06:25] technology. [06:27] >> That is a beautiful [06:30] disclaimer and explanation. Yes, the 10X [06:32] teams, the 10X organizations is totally [06:36] feasible. Nobody really wants to work [06:38] with someone that selfidentifies as a [06:40] 10X engineer. So, absolutely. [06:42] >> No, no, not at all. So, thank you for [06:44] that clarification. [06:45] So for those that don't know, those in [06:47] Europe know there's the DORA [06:49] environmental regulations completely [06:50] different. Uh DORA Google Google is [06:53] deacronomized. Nathan Harvey has taught [06:56] me that word. Um but it's this industry [06:59] benchmarking annual survey that's [07:01] facilitated by Google. It's most [07:04] associated with its four metrics which [07:05] measure throughput and stability, but [07:07] it's actually well over 100 150 [07:10] questions that allow organizations to [07:12] see where they stand. Uh, as a [07:15] technology journalist, this is one of [07:17] those data sources I really look forward [07:19] to every year and I have a selfish plea [07:22] that I ask you to fill out this [07:24] anonymous survey when it comes around. [07:26] Uh, this is really fascinating data and [07:30] it helps you understand and reflect on [07:33] yourself and we get this overarching [07:35] benchmarking report. I've shared it in [07:37] the chat for those live and it's in the [07:39] webinar resources link so you can get [07:41] this year's report. But Eric, can you [07:44] offer a more official definition of what [07:47] the door study is and how companies can [07:49] use it? [07:51] >> Sure. Yeah. So, um, just one item of [07:54] clarification as of this last year of [07:57] research 2025, it's now five cap or five [08:00] metrics. Um, so we added rework to the [08:03] list, which is pretty cool. It was kind [08:05] of a long time coming. uh we you know we [08:07] had been thinking about it and really [08:09] researching it for two years and uh then [08:12] we finally added it to what we call our [08:14] our core model. So unfortunately we [08:17] haven't had the time to update the [08:18] website but all of that is coming soon. [08:20] So we'll be adding that to the quick [08:22] check on the website uh and we'll be [08:24] adding it to you know all of the [08:25] official documentation. But um yeah, so [08:29] um I I wonder if it would be helpful [08:30] just to kind of show like the the Dora [08:33] capability diagram uh and then I can [08:36] kind of walk through that. [08:39] Um [08:40] >> absolutely. Yeah. So let's share that so [08:43] everyone understands. [08:45] >> So uh it looks like our corporate [08:50] >> Oh, this is something we were having [08:53] technical difficulties before [08:55] >> will not allow me to share my screen. So [08:57] >> that's totally fine if you [08:58] >> can can you share dora.dev? [09:01] >> Absolutely. It would be my pleasure. Let [09:03] me make sure I'm not having nefarious [09:05] things open on my laptop. [09:08] Um here we go. So I'm going to dora.dev [09:13] research [09:14] >> and then I'm going to share my screen. [09:17] Thank you so much. [09:19] Share screen. [09:22] >> Yeah. And if you want to just expand [09:24] that capability model and I can kind of [09:26] talk through it there [09:28] >> like under the capabilities, right? But [09:30] this is [09:31] >> just click on research. [09:34] >> Okay. [09:35] >> Just the top button there. [09:37] >> There we go. [09:38] >> And then scroll down and then click on [09:40] um the uh the the radio button. [09:45] >> Which one? [09:46] >> The radio button at the top there where [09:48] it says detail. [09:50] >> Radio is what? I'm not. [09:52] >> Oh, sorry. So, right above the word [09:54] outcomes. There you go. [09:55] >> Thank you. [09:56] >> Okay, cool. And then just kind of scroll [09:57] down so we can see the whole model or at [10:00] least as much of it as possible. [10:02] >> Cool. [10:03] >> All right. So, so let me I'll just give [10:05] a brief uh explanation. Um the the [10:09] metrics that were mentioned um the now [10:11] five key metrics, those sit in the [10:13] middle, the green box there under [10:15] performance and under software delivery [10:17] specifically. So that's how we measure [10:20] software delivery. So these metrics are [10:23] um globally recognized as the industry [10:25] standard way of measuring software [10:27] delivery performance. Um and in fact [10:29] there are the last time I counted which [10:32] admittedly was maybe five or 6 months [10:34] ago there were 60 third party tools that [10:37] surfaced these metrics for their users. [10:40] Um so just kind of adding to you know [10:43] the giving credence to what I said about [10:45] being kind of industry accepted metrics [10:47] and ways of measuring. Now the [10:49] interesting thing is the the metrics [10:52] have definitely been the thing that have [10:56] maybe how do I say this gotten a lot of [11:00] attention. Uh and so as a result a lot [11:03] of people think that the metrics are [11:06] synonymous with Dora. So when they think [11:08] Dora, they think metrics. [11:10] I personally think that the metrics are [11:12] actually the most the least interesting [11:15] part of the whole Dora research and the [11:17] framework and and whatnot. Uh and and [11:20] I'll kind of explain why I think that. [11:23] So what our research really focuses on [11:25] is the blue box on the right hand side, [11:28] the outcomes. Because let's face it, [11:31] this is what everybody cares about, [11:33] right? All organizations care about the [11:36] organizational performance. So whatever [11:38] the organization is trying to achieve, [11:40] that's what the organization cares [11:41] about, right? I mean that's why the [11:43] organization deploys AI and you know [11:46] does DevOps and builds platforms and you [11:48] know everything else that an [11:49] organization does. They do it to achieve [11:51] their goals and also they want to [11:54] improve the well-being of the people [11:55] that work there. At least most companies [11:57] do. Uh so so this is where we start our [12:00] research. We we start with the outcomes. [12:03] Uh and then so what we do is we ask the [12:05] question okay what are the things that [12:07] will predict higher levels of these [12:09] outcomes that we care about and our [12:11] research has shown that software [12:13] delivery performance and reliability are [12:16] two very strong predictors of these [12:19] outcomes that we care about. [12:21] So and that's where we get the metrics [12:24] because you know if we agree that these [12:26] things are important and that they drive [12:28] the outcomes well we need a way to [12:29] measure these things obviously and so [12:31] that's that's how we came up with um the [12:34] the doraometrics. [12:36] So but that's not where the research [12:38] stops and that's where I think a lot of [12:40] people actually think that you know the [12:42] research stops. it keeps going. And so [12:44] what we want to do is we want to [12:46] understand what are the things that [12:48] organizations can do that will predict [12:50] higher levels of software delivery [12:52] performance and higher levels of [12:54] reliability. And so when we think about [12:57] the metrics, I would encourage folks to [12:59] think about the metrics as just a gauge. [13:02] It they're they're just lagging [13:04] indicators that tell you if you're [13:06] heading in the right direction. And the [13:09] things that you really want to focus on [13:11] are the capabilities on the left hand [13:13] side. And and the way that you do this [13:15] first is obviously by understanding what [13:18] the capabilities are and understanding, [13:20] you know, what do we mean when we say [13:21] continuous integration. Um, as an [13:24] example, we want to make sure that we're [13:26] all sort of talking the same language. I [13:28] I've worked with a lot of customers [13:29] where, you know, you're saying the same [13:32] words, but you mean pretty different [13:34] things, right? So So first, let's ground [13:37] on that. look at what we mean by those [13:39] capabilities and then you want to [13:41] prioritize at the application or service [13:44] level where your strengths and [13:47] weaknesses in these capabilities lie. So [13:50] you need to have like a real truthful [13:52] introspection about you know where are [13:55] the bottlenecks in the process and once [13:57] you've identified the bottlenecks you [13:59] can then prioritize them and then start [14:01] improving them. So the the bulk of our [14:03] research really is is on these [14:05] capabilities here that you see on the [14:07] lefth hand side. [14:09] So I hope that gives a good overview for [14:11] folks. [14:12] >> Absolutely. [14:14] So in the context of this year's report [14:18] which was renamed [14:22] which is something unto itself I think [14:25] we could say is different. It's called [14:27] the state of AI assisted software [14:29] development because that's where we are. [14:33] Uh [14:34] as what's changed? What is different [14:37] right now? What have we learned? Like [14:40] because this year is I think feels like [14:44] a turning point [14:45] >> for the tech industry and for the [14:47] report. No. [14:48] >> Yeah, definitely. Um I I think it kind [14:51] of follows along the theme of the [14:52] deacroning that we did uh two years ago [14:55] for the 10-year anniversary. And the [14:57] reason that we did that is because we [15:00] felt that folks were getting too hung up [15:03] on the term DevOps. [15:05] And you know those of us that have been [15:07] in the industry when we say the word [15:10] DevOps we understand what that means. We [15:12] understand that it is you know a [15:14] cultural and a process and an [15:17] organizational transformation. [15:19] But a lot of people conflate that with [15:22] technology. And when you're trying to [15:25] have these conversations with people who [15:27] are not technologists, [15:30] a lot of times they just kind of shut [15:32] down, you know, when they hear trigger [15:34] words, if you will, you know, and then [15:36] DevOps is kind of one of them where we [15:38] would have these conversations and I [15:40] would use the word DevOps and they would [15:41] say, "Okay, well, you know, our [15:44] pipelines are fine. We don't want to [15:45] focus on our pipelines. We want to focus [15:46] on delivering code faster. you're like, [15:49] "Yes, that's what I'm talking about." So [15:51] anyway, so we we made the uh we made the [15:53] decision then to to to deacronym and [15:56] also then as part of that, the name of [15:58] the report was the state of DevOps [16:00] report. Um so it just seemed natural [16:03] that we evolved that remove DevOps from [16:06] the title and really start focusing on [16:09] as you mentioned the the changing [16:11] landscape that we see in the industry [16:13] today. [16:15] >> Okay. and not just a title. What is [16:18] different this year? What has changed? [16:21] Because last year it was kind of like, [16:22] well, [16:24] AI has assisted the quality of docs [16:27] quite a bit, but it's kind of made [16:30] things less stable. So, what is the [16:33] unique about the one that came out in [16:35] November? [16:36] >> Yeah. Well, I mean, obviously there's a [16:38] major focus on AI in the entirety of the [16:42] report. So, not only did we expand the [16:44] research that we uh started three year [16:47] two years prior um and really started [16:51] digging into uh AI at a much deeper [16:53] level. And by the way, just to kind of [16:55] back up for a second, for those of [16:57] people that haven't really been [16:58] following our research, the what we do [17:00] is we we study something, you know, in a [17:03] given year and when we discover [17:05] something that's interesting, exciting, [17:07] something that we want to understand [17:08] more, we continue to explore those [17:11] topics and just go deeper and deeper and [17:13] deeper. So, this is the third year of of [17:15] that that AI uh pursuit. And it was [17:18] really apparent to us that this was [17:20] something that we needed to go allin on [17:22] because the industry is going allin on [17:24] this. And it really has changed the the [17:27] paradigm for what's happening at the um [17:32] you know at the the entire organization [17:34] from idea to implementation. And so we [17:38] wanted to get a deeper understanding of [17:41] you know how is AI affecting developers [17:44] in their day-to-day? What are things [17:45] that developers are using? How are they [17:47] using these things? What impacts is AI [17:50] having? And one of the the impacts that [17:52] we did discover uh again was that it is [17:56] increasing the instability [17:58] which is not a good thing. Um and also [18:01] increasing rework. So so we we did see [18:04] that. And then another major change or [18:07] major thing that we added to the report [18:09] this year is we create so that [18:11] capability guide um or map rather that [18:14] you were just displaying on the screen. [18:16] We expanded that and we created an AI [18:19] capability model. So what we wanted to [18:22] understand so what we talked about just [18:24] a minute ago is the capabilities that [18:27] will drive software delivery [18:28] performance. What we wanted to [18:30] understand was when you when you have AI [18:35] what are the capabilities that you can [18:38] can focus on to drive specific outcomes [18:42] that you're looking for. So there we go. [18:46] Um, so what this does is this allows you [18:49] to start from the right hand side. We [18:51] always start from the right hand side [18:52] and work our way to the left. Uh, and [18:55] and so you can look at this map and and [18:57] say, "Okay, I want to improve, you know, [19:00] uh, my team performance as an example. [19:03] So you can hover over team performance [19:05] and what it does is it will tell you and [19:07] highlight the uh the capabilities that [19:11] you need to focus on when you have when [19:14] you're working with AI. And so those [19:18] capabilities will amplify the impact of [19:22] AI on those outcomes that you care [19:24] about. Now it's really important to [19:27] understand that when I say the word [19:28] amplify, this is a birectional pathway. [19:32] So it doesn't just make things better, [19:35] it also makes things worse. And let me [19:38] explain. [19:39] If you are doing things well, AI makes [19:43] those things better. If you are doing [19:47] things not very well, AI makes things [19:51] worse. So this team performance is a [19:54] really great example. When you dig into [19:56] our research, what it shows is that [19:58] teams that are user centric, [20:02] AI then makes the team performance [20:05] better. [20:06] Teams that are not user centric, [20:10] AI introduction into the team makes team [20:13] performance measurably worse, [20:16] substantially worse. So I think that's [20:18] really important to to understand is is [20:21] this this amplification goes both ways. [20:23] So anyway, so we created this model to [20:26] help people focus on the outcomes that [20:28] they're looking for and really try to [20:30] dissect and understand, you know, [20:33] perhaps why aren't we getting the [20:35] results that we expected? Like code [20:38] quality as an example is a big one that [20:40] customers, you know, frequently I [20:42] frequently hear from people, hey, you [20:44] know, these tools are great. They're [20:46] helping us write code quickly, but a lot [20:48] of times the quality just isn't there. [20:50] Well, what our research shows is that in [20:52] order to get high levels of code [20:54] quality, you need to have internal data [20:57] that is accessible to your AI tools. So, [21:00] you know, if you're using code assist, [21:02] they need to be trained on your internal [21:04] repos in order to really produce good [21:07] quality code. as an example. [21:10] >> And as you can see, it doesn't just [21:12] affect code quality, it affects [21:14] individual effectiveness, it affects [21:17] product performance, [21:19] >> which makes sense because we've always [21:21] known garbage in garbage out data, [21:23] right? [21:25] >> That's right. Yep. And and so if you're [21:27] interested in [21:28] >> garbage, but something trained on the [21:30] world is not trained on your code. These [21:33] are difference. [21:35] >> That's right. Yeah, that's that's very [21:37] correct. Um, and then lastly, what I'll [21:39] say about this is um, we've also been [21:42] from a Dora research perspective, we've [21:44] been trying to put out more content and [21:46] and give more to the community. And so [21:48] we actually just recently released an [21:50] entire report on this model. Um, and so [21:54] that's the AI capabilities model that [21:56] you see there on the left hand side. Um [21:59] it is almost a hundred pages that goes [22:02] very in-depth into each of those [22:04] capabilities and the outcomes that they [22:05] drive. Shows all of the research, all of [22:08] the data. Um and so that's uh that's a [22:11] really great place uh to go if you want [22:13] to learn more about this model [22:16] >> and everything we're talking about is [22:18] not just this model and these things but [22:21] what it actually is in reality. So [22:24] pashant with that in mind like what are [22:28] you how does this ring true compared [22:30] with like what's because harness is a [22:33] company that uses AI you are still [22:34] technology company and what your [22:37] organization is experienced and what you [22:39] find your clients are going through. [22:41] Yeah. [22:43] uh loved the overview veric I think Dora [22:45] has been the golden standard for the [22:47] industry for a while right for any [22:50] leader as we have gone into a services [22:52] architecture uh if you are the owner of [22:55] a service uh and you're responsible for [22:58] the uptime the the health of these [23:00] services one of the key measures of this [23:02] is you know as as you um as as you own [23:05] the teams that build these services [23:07] maintain these services is the is the is [23:09] are the durometrics right so um and and [23:12] AI I sort of uh is uh is a is a is a is [23:17] a big event that's happening in the [23:18] industry. Um and there are many ways to [23:21] measure AI, many ways to understand AI, [23:23] but Dora continues to be one of those [23:25] golden metrics that absolutely continues [23:26] to matter, right? Because at the end of [23:28] the day, AI or or not, uh you are [23:30] measuring software delivery and that's [23:32] what Dora helps you measure, right? So I [23:34] think uh from my perspective you know I [23:36] spend a lot of time talking to a lot of [23:38] our customers who are um um on the hook [23:42] in so many ways right if if you imagine [23:44] being a SVP of engineering a CIO a CTO [23:48] today there is tremendous pressure on uh [23:52] us as leaders to to truly basically [23:55] embrace AI uh and show results show ROI [23:59] right money is no longer cheap over the [24:01] last many years money has become [24:03] expensive Right? Everything must be [24:05] proven. The ROI must be shown. Right? [24:07] And AI is such a transformational [24:09] technology right now that everybody is [24:11] is is expecting magic to happen. But [24:14] that magic needs to be proven. It needs [24:16] to be shown by hard numbers. Right? So [24:18] every uh leader today is saying look I [24:21] am making investments. Not enabling my [24:23] teams with the best in breed AI tooling [24:25] is not an option. Some teams are signing [24:28] up for one vendor. Most customers that I [24:31] talk to are have an open vendor policy. [24:33] Does not mean anybody can bring anything [24:35] but it means that as a best practice [24:38] they want to enable more and more coding [24:40] assistance for their users. See what you [24:43] know their developers love within sort [24:45] of an enterprise contract with the right [24:47] guard rails and and so on and so forth, [24:50] right? Um and as these investments begin [24:52] to happen, the the obvious question now [24:54] for the that engine leaders have to [24:56] answer back to the board, back to the [24:57] CFO is like we made all these [24:59] investments. What is the ROI? What tool [25:01] is worth investing more in? What tool is [25:04] truly providing us value? And so on and [25:05] so forth, right? And so um so it comes [25:09] down to measuring what tools developers [25:13] are using. It comes down to measuring [25:16] with the tools that these developers are [25:18] using. What impact are you actually [25:20] seeing in your SDLC? There is obviously [25:22] a very easy perception. Everybody feels [25:25] more productive, right? All of a sudden [25:26] you can prompt and code is written for [25:28] you, right? And test cases are written [25:30] for you and documentation can be [25:31] generated. That is a massively powerful [25:33] thing to happen. So there is an obvious [25:35] perception that my productivity has [25:37] significantly enhanced. [25:39] And yet as an organization to to [25:41] everything that Eric was mentioning um [25:44] it is not necessary that if everybody [25:46] begins to use tool on their own that as [25:48] an organization you're actually thriving [25:50] and you are moving towards becoming that [25:52] 10x organization that is not guaranteed [25:53] right so as leaders again you want to [25:56] begin to measure you want to begin to [25:58] understand that as an organization not [25:59] as an individual not as one team but as [26:02] an organization how well have we [26:03] embraced AI what is the impact I'm [26:05] seeing what is the positive impact but [26:08] sometimes what is the negative impact or [26:10] sometimes what is flatlined where I [26:11] expected obvious gains but I but I'm not [26:13] seeing those gains. So you want to begin [26:15] to measure. So I think Eric said [26:16] something where metrics are laggered [26:18] indicators uh and might be very less [26:20] interesting as compared to the actual [26:21] capabilities you want to develop. But [26:22] guess what right um unless until you [26:25] have those gauges that tell you where is [26:27] the slowdown where is that you should [26:29] focus on you don't know what capability [26:30] to start improving as an organization. [26:33] So installing that gauge becomes sort of [26:34] that you know first step you take begin [26:36] to measure things uh that opens you know [26:39] your mind that gives you eyesight into [26:42] what are the different bottlenecks [26:44] you're you're you're observing [26:45] especially again as you as you invest in [26:47] AI like I'll give a very crude example [26:50] uh and Pandora research also speaks to [26:52] this is right uh what what AI is doing [26:54] is that it's generating tons of code [26:56] right it's so easy to write code now [26:58] every engineer is creating more and more [27:00] and more code but that more code is [27:03] leading to first of all review [27:04] bottlenecks. If you have not as an [27:06] organization considered what does it [27:08] mean to review this massive amount of [27:10] code that is being generated now and [27:11] rethink your review processes there is a [27:13] review bottleneck. If the quality of the [27:15] code for whatever reasons because you [27:17] didn't train your models well enough [27:18] because you've not enabled your [27:19] engineers well enough if the quality of [27:21] that code is not great it's leading to a [27:23] lot of rework you have a rework [27:24] bottleneck right so being able to [27:26] identify these things and then finding [27:28] the capabilities you have to invest in [27:29] is the way I see sort of the industry uh [27:32] you know truly sort of panning on this [27:35] >> actually when we [27:37] those that are watching this have gone [27:39] through the process of signing up for a [27:41] lead dev webinar and there's quite an [27:44] openended a question about like what you [27:46] hope to get out of joining this [27:48] conversation. And the most common [27:52] response was [27:54] to summarize it because everyone writes [27:56] their own ideas was people were looking [27:59] for effective measurable ways to boost [28:01] productivity or to increase engineering [28:03] efficiency for technical teams. So this [28:06] does come down to the meas the metrics. [28:09] So I mean I we I definitely hope people [28:13] fill out Dora sometimes [28:15] uh once a year at least maybe more but [28:18] it is not a quick survey by any means. [28:23] It is quite detailed and thorough and [28:25] important in itself. But what are [28:28] especially when we have this whole idea [28:30] of empowering developers with autonomy, [28:33] but then this creates this whole AI [28:35] sprawl and maybe shadow AI because we [28:39] know if you try to ban something that [28:41] they want to use that it's not that [28:42] expensive. They'll just take out their [28:44] credit cards. Maybe we need to control [28:46] that. So, how can you measure the AI [28:50] adoption where the bottlenecks are and [28:52] the impact on your developers? What are [28:54] some of the key metrics? [28:57] >> Is that a question for me, Jennifer? [28:59] >> For either of you? Yeah. What have you [29:01] been finding that is some powerful [29:03] metrics either for your team or your [29:04] customers? [29:06] >> Yeah, I can take a first crack, Eric, [29:07] and then you can add to that. [29:08] >> Sure. [29:08] >> Yeah, please. [29:09] >> Um, [29:10] >> so I think uh yeah, I think uh the first [29:12] thing like what does it mean to measure [29:14] AI, right? Um there are two broad [29:16] categories. What is the adoption of [29:18] these tools and then what is the impact [29:20] of those tools and we'll we'll come to [29:21] both. Right? So to measure adoption of [29:25] these tools obviously you want to start [29:27] looking into saying a how many users are [29:30] using these tools what kinds of tools [29:33] but then using this tool does not [29:35] guarantee that you're actually using it. [29:37] Right? Just having a license and you [29:40] know once in a while prompting something [29:41] in the tool does not mean you're [29:42] effectively using the tool. So then you [29:44] have to go within these tools and begin [29:46] to understand how much of the code that [29:48] you're actually uh committing to your [29:50] repositories is actually coming from [29:52] these AI assistants, right? Uh [29:54] interestingly that is a hard problem to [29:56] solve today. The state [29:58] >> how do you measure that? Like are you [30:00] just asking developers or are you using [30:02] tools to measure the percentage of code [30:06] that AI was involved? Because maybe [30:08] developers don't feel safe saying [30:09] whether or not they're using AI or not. [30:11] Side note, they are. [30:14] I think every yeah fortunately every [30:16] coding assistant on the market today uh [30:18] exposes uh data as well as APIs for uh [30:22] you know customers to gather insights [30:24] for you know per user per team per [30:26] organization. How much prompting are [30:29] users doing? How many lines of code is [30:32] is an is an agent generating? How much [30:34] of that line how much of that code being [30:36] generated is being accepted by the users [30:38] and actually becoming part of their [30:39] code. Right? Uh so so there's a lot of [30:41] data that these assistants provide that [30:43] for example a tool like SEI we talk to a [30:46] cloud code we talk to a windsurf a [30:48] cursor a Gemini code it says right [30:49] different tools and pull in this data so [30:52] that provides that common visibility of [30:54] adoption and usage of these tools to our [30:57] customers um but the but the hard part [31:00] is once that data is accepted right uh [31:02] let's say I prompted winds and winds [31:05] generated code and I accepted that code [31:07] there is no way to guarantee that that [31:09] code was then not modified or lines of [31:11] code were not deleted by the developers. [31:13] Right? And that's where there is a [31:14] little bit of a flux in the industry [31:16] today where truly measuring uh truly [31:18] tagging every line of code like imagine [31:20] if git had a metadata for every line of [31:22] code which said this was purely AI [31:24] generated versus this was AI generated [31:26] and then erit versus you know some of [31:27] the code that was only manually created. [31:30] Uh so that level of sort of [31:31] instrumentation for code does not exist [31:34] uh today but I have no doubt that we all [31:35] heading towards that uh if we have to [31:37] deeply measure this. That's just the [31:38] measurement of adoption and usage. And [31:41] then there is the impact part. And if [31:42] you really think of impact part, what [31:44] you what you want to measure is [31:45] obviously like like why am I using [31:47] because I want to be faster. I want to [31:48] be generating more code. Uh what is the [31:51] improvement in your code velocity is the [31:53] first thing you want to measure. And [31:54] this is this is expected to be typically [31:56] very positive gains, right? But velocity [31:59] often can come [32:00] >> velocity velocity at what point? [32:04] the you could you could look at [32:06] >> yeah you could look at velocity at a [32:08] granular level in terms of the PRs that [32:10] developers are generating are am I [32:11] creating a lot more PRs than I used to [32:14] am I writing a lot more lines of code [32:15] than I used to but more importantly from [32:17] a business perspective am I actually [32:19] resolving more bugs and am I delivering [32:20] more features and capabilities in the [32:22] product right has the level of features [32:24] that I was delivering has that actually [32:26] improved so that is feature velocity PR [32:28] velocity bug velocity that tools like [32:30] SEI help you measure uh but to complete [32:33] that point Jennifer I think velocity can [32:35] often come at the cost of quality right [32:36] especially in the age of AI uh if you're [32:39] not reviewing this code very well then [32:41] often the number of bugs coming is is [32:43] high but also like Eric mentioned it [32:46] might be leading to a lot of rework how [32:48] do you measure the rework that is being [32:50] created because of all this uh you know [32:53] AI code that that is coming in and then [32:55] finally it's not just quality of the [32:57] code now but it's the long-term [32:58] maintainability of the code you know [33:00] it's it's uh it's AI is many times not [33:02] explainable [33:03] A lot of times the code that is written [33:06] uh if if you know an year later [33:08] developers had to go fix things and AI [33:10] was not fixing it, chances are you might [33:12] not understand what's in there. What is [33:13] the long-term maintainability of this [33:15] code? What is the quality of this code? [33:16] Right? So there are many ways to sort of [33:18] think about this problem uh that leaders [33:20] today are looking to measure and that's [33:21] where again tools like SEI software [33:23] engines insights come in and try to [33:25] provide those uh those insights. [33:29] >> Okay. So, so, um, c can I respond to to [33:32] that one? I I have some some thoughts on [33:34] this, and I'll and I'll start by, uh, [33:36] summoning one of my favorite South Park [33:38] episodes, the underpants gnomes, if you [33:40] all remember that, where it was like [33:42] step one, collect underpants, step [33:43] three, profit. So, I think a lot of [33:47] customers and organizations treat AI in [33:50] in a very similar way where they think [33:52] step one, adopt AI tools, step three, [33:55] profit. the the the problem is is that [33:59] there's that there's a middle part. Um [34:01] so there's adopting the tool and then [34:04] you have to build trust in in the tool [34:07] with the developers and only until you [34:10] have built trust will you then start to [34:12] see developer productivity. And and [34:15] there's some additional nuance here [34:16] because the way that you measure [34:18] adoption is going to be different than [34:20] the way that you measure trust both from [34:22] a metrics perspective and also at the [34:25] the level at which you measure it. So [34:27] most of these will be at the individual [34:29] level. So when you're adopting the tool, [34:31] you want to see are people using the [34:33] tool? Are they using the licenses? You [34:35] know, are they firing it up? You know, [34:37] yada yada. When you get into trust, you [34:40] need to understand, you know, things [34:42] like the percentage of of code [34:44] committed, lines of code committed from [34:46] AI to see whether they're they are [34:48] actually having trust in the tool. This [34:51] process can take between a month to two [34:52] months to actually build trust in these [34:55] systems. And only after you've done that [34:57] are you going to be able to reach [34:59] developer productivity. Developer [35:02] productivity though is now measured at [35:04] the team or the application level. I was [35:07] going to say developers tend to really [35:09] not like well probably any human being [35:11] to be monitored and measured at level [35:14] especially in a less secure job market. [35:18] >> Yeah. And and it's just not helpful, [35:20] right? It doesn't really actually tell [35:21] you anything that's important. You know, [35:24] it would it gives you data, but does the [35:26] data matter? And I would argue very [35:28] strongly that it doesn't. So in addition [35:30] to all of the things that pashant [35:32] mentioned around velocity and ticket [35:34] closures um you can also use dorometrics [35:37] as a measure of productivity but of [35:39] course again that's at the application [35:41] or service level. But that brings us [35:43] into a new and and interesting situation [35:46] because a lot of companies are adopting [35:49] AI. They're building trust in AI but [35:52] they're not getting productivity out of [35:54] AI. And and a lot of times when you zoom [35:58] in and you really get try to get an [35:59] understanding of why that is, what I'm [36:02] seeing is that companies are investing [36:05] what seems like over 100% of all of [36:08] their time, resources, and effort into [36:11] AI adoption, [36:13] but they're not continuing to focus on [36:16] the fundamentals of the underlying [36:18] platforms and the underlying pipelines. [36:21] You know, and like Pashant mentioned, [36:23] you're now producing a lot more code, [36:26] and a lot of times it's code that you [36:28] don't even fully understand, but yet [36:31] companies haven't spent the time to [36:33] improve their testing automation, their [36:37] uh, you know, their documentation [36:39] quality, their continuous integration. [36:42] And so now what you have these systems [36:44] that are fragile and already not doing [36:47] well. I mean, I don't know about you, [36:48] Pashant, but I've literally never worked [36:50] with a customer in the last 10 years [36:52] when I asked them the question, "Is your [36:54] testing automation great?" None of them [36:57] say yes. [36:58] >> And now there's more code to [37:01] >> Now there's more. And we're not even [37:02] talking about agents yet, right? [37:05] >> We're just talking about code assist [37:06] tools for the most part. When you start [37:09] getting into agents now, it's like one [37:12] developer to, you know, how many a how [37:15] many developers is an agent? It could [37:18] almost be an infinite amount. And so [37:20] unless you have these underlying [37:22] fundamentals in place and and really one [37:25] of the keys to this is that and and this [37:27] is shown in our research from last year [37:30] is the the internal developer platform [37:33] is the key to unlocking organizational [37:37] performance with AI. So you know you can [37:41] get and pashant was talking about this [37:43] earlier you can get individual [37:45] performance you can get team performance [37:48] but to really get organizational [37:50] performance you need a well-built [37:53] internal developer platform and most [37:56] people that I interact with they have [37:58] internal developer platforms but almost [38:01] none of them have well-built and and you [38:05] know properly built internal developer [38:07] platforms. So, I I definitely see that [38:09] as as something that is a is a friction [38:12] point now and it's just going to get [38:14] exponentially worse the more that that [38:16] organizations adopt uh agents to do the [38:20] bulk of the work. [38:22] >> Absolutely. And uh for those that are [38:24] joining us live, I've enacted a poll, if [38:27] that's the verb, uh which best describes [38:30] where your organization is on the [38:31] platform engineering journey. And we've [38:33] got a well hopefully uh well we did but [38:37] then our budgets were cut and then [38:39] following the cloudnative computing [38:41] foundations platform maturity model [38:44] we've got so please do answer if you [38:46] want uh we're talking about provisional [38:48] which is custom processes by request ad [38:50] hoc totally appropriate in a startup [38:54] setting where you've got less than 50 [38:56] engineers you still have that close [38:58] context and all totally fine then we've [39:00] maybe got the operational maybe you [39:02] start to standardized tooling centrally [39:04] tracked by a dedicated team. Then maybe [39:07] you have the reach the level of [39:09] scalability with platform as a product. [39:11] You're treating your [39:13] internal developers like customers. You [39:15] have the self-service mechanism. So [39:18] because you can't have a platform team [39:20] scale to the amount of developers you [39:22] have. Uh you have insights metrics. We [39:25] clearly like metrics on the call. And [39:27] then the goal the end goal of your [39:30] golden path you're [39:33] is to have this reach this level of [39:35] optimization where you're balancing both [39:37] qualitative and quantitative metrics. [39:39] You have integrated managed services [39:43] enabled ecosystem and then you still [39:46] have that like platform as a product [39:48] which might be even having internal [39:50] marketing or demo days things like that. [39:53] So uh while we see where our audience is [39:58] on that uh I would recommend making it [40:00] wide and I wrote long responses uh [40:04] pashant so how are you finding as a [40:07] platform company and you're not the only [40:09] one by any means and there is this idea [40:12] that maybe you start out by building a [40:15] little something or a wiki to start but [40:18] as it evolves can you offer some use [40:22] cases is of how like we're really [40:24] finding platform engineering as a [40:26] conduit of AI. So [40:29] >> yeah, absolutely. I think um one of the [40:32] taglines we use at harness is that [40:33] harness is AI for everything after code, [40:36] right? If you really think about SDLC, [40:39] um it's it's almost funny to think about [40:41] the fact that, you know, developers [40:43] today are spending 30 to 40% of their [40:45] time actually developing code, you know, [40:48] designing things, writing specs, [40:50] actually writing code in git uh and and [40:53] working on on the on the truly thing [40:55] that sort of uh you know gets them [40:57] excited and and gets their juices [40:58] flowing. And fortunately unfortunately [41:00] 60 to 70% of our time as developers [41:02] today goes into so much of the software [41:05] delivery aspect that is just necessary [41:08] friction to deliver software. It is [41:10] building of my software. It is running [41:12] you know a battery of different kinds of [41:14] test suites. It is you know making sure [41:16] my software is secure. Running code [41:18] vulnerability scans running container [41:20] scans running cloud scans running API [41:22] scans. It's about then being able to [41:25] deploy this thing. Uh being able to [41:27] handle deployment failures. It's about [41:29] managing cloud cost. A huge part of [41:31] today's engineing organizations is that [41:33] every time I deploy I I update a new [41:35] feature, it has incrementally led to [41:37] significant more cloud cost that we are [41:39] incurring [41:40] >> and AI more we would assume [41:42] >> and you need governance. You need [41:44] controls to sort of cap that in some in [41:46] some capacity because the easiest thing [41:47] for anybody to do is to make sure my [41:49] service is healthy. I'm going to spit [41:51] spin up 50 instances of this container. [41:53] Sure. But now you're burning money, [41:55] right? So how do you sort of um so a [41:58] platform essentially allows you to [42:00] manage all these different parts of [42:03] SDLC. It's so much more than just [42:05] writing code, right? Um and uh harness [42:09] for example today is in is has [42:11] introduced AI into every aspect of this [42:13] this delivery life cycle, right? So we [42:15] have a DevOps agent as an example that [42:17] helps you build better pipelines, debug [42:21] those pipelines, create pipelines [42:23] automatically for the code that you want [42:24] to deploy. Um we have uh a dev sec ops [42:28] agent that automatically for every [42:30] service that you're deploying go goes to [42:31] the code, runs the different scans, [42:34] identifies vulnerabilities, but then [42:36] because well AI uh is now actually [42:39] creating PRs to fix those [42:41] vulnerabilities, right? So how do you [42:43] build bring in that level of automation [42:45] where you know automatically you [42:46] discover vulnerabilities but then you [42:47] automatically fix those vulnerabilities. [42:49] We have a cloud cost agent that is [42:51] constantly left shifting the notion of [42:53] cost every time you're building [42:55] something new. How do you left shift a [42:56] construct of cost and and be able to [42:58] provide early indicators to say that you [43:00] know here are some improvements you can [43:02] make to to better control the cost [43:03] you're running. Um we have a S sur agent [43:07] that is part of SDLC now that every time [43:09] you deploy something it is constantly [43:12] looking at the service that you've [43:14] deployed the uptime it is looking for [43:15] signatures uh how healthy is the service [43:18] looking if it's not automatically create [43:20] incidents automatically diagnose those [43:22] incidents right so there is just so much [43:25] you can do as a platform right and and I [43:27] think to bring this home uh one thing [43:29] that I want to call out is the real [43:31] power of platforms is that it has [43:33] context text to every in in your [43:36] organization for you as a as an account. [43:38] It understands everything you built, [43:40] everything you deployed, every feature [43:42] flag you changed, every um metric that [43:45] you're measuring, every incident that [43:47] happened, every cloud cost uh action [43:49] that you're taking. When you have this [43:51] holistic view into every action you take [43:54] as part of your SDLC, the way AI can [43:57] truly understand and now begin to become [43:59] your mentor, begin to make [44:01] recommendations, begin to optimize all [44:02] of this is just so much more immensely [44:04] powerful than individual tools you might [44:06] bring in, right? And that is where [44:07] platforms really begin to shine because [44:09] it has a true semantic understanding of [44:11] everything you do from an SDLC [44:13] standpoint. that that's such an amazing [44:16] point that I have also been really [44:18] trying to pound the table around and I [44:20] refer to the IDP uh or the platform as [44:23] the center of the universe for [44:26] everything that's happening in the [44:27] organization because you have the [44:30] documentation, you have all of the the [44:32] previous uh uh things that have been [44:34] tried and and that have worked and maybe [44:36] not worked. You have all of the the [44:38] engineering metrics. You you have all of [44:40] the code, all of the documentation, the [44:42] service catalog, all of the golden [44:44] paths. I mean, it's it's wild. And [44:48] having that context allows you to do so [44:51] much more in a fully automated way. [44:56] >> Absolutely. And with the searchable [44:59] context as well, which I think is very [45:01] important, people [45:03] and especially developers maybe don't [45:05] want to ask people the first time they [45:07] want to do something. They do want to be [45:09] self-service. So that especially if [45:11] they're new to an organization so that [45:13] when they're asking a question, it's an [45:15] important one, not a who owns that or [45:18] how do I find this? Because guess what? [45:22] it's frustration is and finding things [45:25] is like 20% of a developer's day and I [45:28] assume anyone joining a company not just [45:30] someone in a technical role. So for [45:33] those that are on this call, uh the [45:37] platform journey is [45:40] really all over the place, I would say. [45:43] We've definitely have a varying amount [45:45] of people and on this platform journey, [45:48] I wish you luck because I think we've [45:50] found very clearly with this year's [45:52] Storm Report, it is the best way and the [45:58] most successful way. the these two [46:00] trends have converged and it's the way [46:02] to deliver AI now and then uh good luck [46:06] on your journey. Uh we will continue [46:09] this conversation in Slack and thank you [46:11] so much to our active listeners and [46:13] those listening afterwards and [46:14] especially thank you so much to Pashant [46:17] and Eric. Good luck on your what is a [46:21] journey not a destination with AI. [46:23] Thanks y'all. [46:24] >> Thanks everyone. [46:36] I'm on the ## Highlights - essentially uh aims to empower engineering leaders today with deep insights into you know how well are teams building software, how efficient are the processes you're running, how productive are teams, how well aligned is your execution to the outcomes that you're looking for, how well are you using AI, right? Everything begins with measure measurement and then using those measured metrics to drive actions, drive transformation, recommend changes, right? And so that is what uh this ([View Highlight](https://read.readwise.io/read/01kgg2hxf3w2a0e4xsj3gms7z9)) - Tags: 👻 ai highlighted - improving them. So the the bulk of our research really is is on these capabilities here that you see on the lefth hand side. So I hope that gives a good overview for folks. >> Absolutely. So in the context of this year's report which was renamed which is something unto itself I think we could say is different. It's called the state of AI assisted software development because that's where we are. ([View Highlight](https://read.readwise.io/read/01kgg2hxhmckd4x45k95grf1gy)) - Tags: 👻 ai highlighted - you know at the the entire organization from idea to implementation. And so we wanted to get a deeper understanding of you know how is AI affecting developers in their day-to-day? What are things that developers are using? How are they using these things? What impacts is AI having? And one of the the impacts that we did discover uh again was that it is increasing the instability which is not a good thing. Um and also ([View Highlight](https://read.readwise.io/read/01kgg2hxm4dw6mab4fz1znz40b)) - Tags: 👻 ai highlighted - increasing rework. So so we we did see that. And then another major change or major thing that we added to the report this year is we create so that capability guide um or map rather that you were just displaying on the screen. We expanded that and we created an AI capability model. So what we wanted to understand so what we talked about just a minute ago is the capabilities that will drive software delivery performance. What we wanted to ([View Highlight](https://read.readwise.io/read/01kgg2hxpsh88gcz5q9z8fjdv4)) - Tags: 👻 ai highlighted - dissect and understand, you know, perhaps why aren't we getting the results that we expected? Like code quality as an example is a big one that customers, you know, frequently I frequently hear from people, hey, you know, these tools are great. They're helping us write code quickly, but a lot of times the quality just isn't there. Well, what our research shows is that in order to get high levels of code quality, you need to have internal data that is accessible to your AI tools. So, ([View Highlight](https://read.readwise.io/read/01kgg2hxs9dbd2e1bc8te7y6am)) - Tags: 👻 ai highlighted - an organization how well have we embraced AI what is the impact I'm seeing what is the positive impact but sometimes what is the negative impact or sometimes what is flatlined where I expected obvious gains but I but I'm not seeing those gains. So you want to begin to measure. So I think Eric said something where metrics are laggered indicators uh and might be very less interesting as compared to the actual capabilities you want to develop. But guess what right um unless until you have those gauges that tell you where is the slowdown where is that you should focus on you don't know what capability ([View Highlight](https://read.readwise.io/read/01kgg2hxz3q4e387g51cn4syw9)) - Tags: 👻 ai highlighted - been finding that is some powerful metrics either for your team or your customers? >> Yeah, I can take a first crack, Eric, and then you can add to that. >> Sure. >> Yeah, please. >> Um, >> so I think uh yeah, I think uh the first thing like what does it mean to measure AI, right? Um there are two broad categories. What is the adoption of these tools and then what is the impact of those tools and we'll we'll come to both. Right? So to measure adoption of these tools obviously you want to start looking into saying a how many users are ([View Highlight](https://read.readwise.io/read/01kgg2hy25fq41nrf4js54dr2d)) - Tags: 👻 ai highlighted - to all of the things that pashant mentioned around velocity and ticket closures um you can also use dorometrics as a measure of productivity but of course again that's at the application or service level. But that brings us into a new and and interesting situation because a lot of companies are adopting AI. They're building trust in AI but they're not getting productivity out of AI. And and a lot of times when you zoom in and you really get try to get an understanding of why that is, what I'm ([View Highlight](https://read.readwise.io/read/01kgg2hybw28s7qdhhx0p19fjq)) - Tags: 👻 ai highlighted ## New Highlights added 2026-02-28 at 7:49 PM - essentially uh aims to empower engineering leaders today with deep insights into you know how well are teams building software, how efficient are the processes you're running, how productive are teams, how well aligned is your execution to the outcomes that you're looking for, how well are you using AI, right? Everything begins with measure measurement and then using those measured metrics to drive actions, drive transformation, recommend changes, right? And so that is what uh this ([View Highlight](https://read.readwise.io/read/01kgg2hxf3w2a0e4xsj3gms7z9)) - Tags: 👻 ai highlighted - improving them. So the the bulk of our research really is is on these capabilities here that you see on the lefth hand side. So I hope that gives a good overview for folks. >> Absolutely. So in the context of this year's report which was renamed which is something unto itself I think we could say is different. It's called the state of AI assisted software development because that's where we are. ([View Highlight](https://read.readwise.io/read/01kgg2hxhmckd4x45k95grf1gy)) - Tags: 👻 ai highlighted - you know at the the entire organization from idea to implementation. And so we wanted to get a deeper understanding of you know how is AI affecting developers in their day-to-day? What are things that developers are using? How are they using these things? What impacts is AI having? And one of the the impacts that we did discover uh again was that it is increasing the instability which is not a good thing. Um and also ([View Highlight](https://read.readwise.io/read/01kgg2hxm4dw6mab4fz1znz40b)) - Tags: 👻 ai highlighted - increasing rework. So so we we did see that. And then another major change or major thing that we added to the report this year is we create so that capability guide um or map rather that you were just displaying on the screen. We expanded that and we created an AI capability model. So what we wanted to understand so what we talked about just a minute ago is the capabilities that will drive software delivery performance. What we wanted to ([View Highlight](https://read.readwise.io/read/01kgg2hxpsh88gcz5q9z8fjdv4)) - Tags: 👻 ai highlighted - dissect and understand, you know, perhaps why aren't we getting the results that we expected? Like code quality as an example is a big one that customers, you know, frequently I frequently hear from people, hey, you know, these tools are great. They're helping us write code quickly, but a lot of times the quality just isn't there. Well, what our research shows is that in order to get high levels of code quality, you need to have internal data that is accessible to your AI tools. So, ([View Highlight](https://read.readwise.io/read/01kgg2hxs9dbd2e1bc8te7y6am)) - Tags: 👻 ai highlighted - an organization how well have we embraced AI what is the impact I'm seeing what is the positive impact but sometimes what is the negative impact or sometimes what is flatlined where I expected obvious gains but I but I'm not seeing those gains. So you want to begin to measure. So I think Eric said something where metrics are laggered indicators uh and might be very less interesting as compared to the actual capabilities you want to develop. But guess what right um unless until you have those gauges that tell you where is the slowdown where is that you should focus on you don't know what capability ([View Highlight](https://read.readwise.io/read/01kgg2hxz3q4e387g51cn4syw9)) - Tags: 👻 ai highlighted - been finding that is some powerful metrics either for your team or your customers? >> Yeah, I can take a first crack, Eric, and then you can add to that. >> Sure. >> Yeah, please. >> Um, >> so I think uh yeah, I think uh the first thing like what does it mean to measure AI, right? Um there are two broad categories. What is the adoption of these tools and then what is the impact of those tools and we'll we'll come to both. Right? So to measure adoption of these tools obviously you want to start looking into saying a how many users are ([View Highlight](https://read.readwise.io/read/01kgg2hy25fq41nrf4js54dr2d)) - Tags: 👻 ai highlighted - to all of the things that pashant mentioned around velocity and ticket closures um you can also use dorometrics as a measure of productivity but of course again that's at the application or service level. But that brings us into a new and and interesting situation because a lot of companies are adopting AI. They're building trust in AI but they're not getting productivity out of AI. And and a lot of times when you zoom in and you really get try to get an understanding of why that is, what I'm ([View Highlight](https://read.readwise.io/read/01kgg2hybw28s7qdhhx0p19fjq)) - Tags: 👻 ai highlighted