How to Measure the Business Impact of AI - CourtneyR.dev's 2nd Brain

How to Measure the Business Impact of AI ![rw-book-cover](https://img.youtube.com/vi/_-j4VBobl-s/maxresdefault.jpg) 🔗 **Source:** [How to Measure the Business Impact of AI](https://www.youtube.com/watch?v=_-j4VBobl-s) > [!note]+ Document Notes > tags: #artificial-intelligence #developer-productivity #business-metrics #engineering-management #roi-measurement > author: Lead Dev > source: YouTube > reference: https://www.youtube.com/watch?v=_-j4VBobl-s > date created: 2024-12-19 14:30 > > # 🏷️ Title: How to measure the business impact of AI >  ## 🧰 Action Inbox - [H1] Establish baseline metrics before implementing AI tools — prevent inability to measure ROI #artificial-intelligence - [H2] Implement cohort analysis comparing high vs low AI adopters — measure actual productivity impact #developer-productivity - [H3] Create psychological safety for honest burnout reporting — prevent AI-induced overwork #engineering-management ## 🧭 Table of Contents - [[#📝 Summary]] - [[#📖 Main Content]] - [[#🗒️ Notes]] - [[#📚 References]] - [[#🗣️ Paraphrase]] - [[#📰 Newsletter]] - [[#✅ Action Items]] - [[#🧪 Hypotheses]] - [[#📊 Methodology]] - [[#❓ Questions]] - [[#🔮 Future Research]] - [[#💡 Implications]] - [[#🧠 Learning Styles Assessment]] - [[#📊 Accessibility Assessment]] - [[#🌍 DEIB Assessment]] - [[#✅ Logical Validity]] - [[#🛡️ DARVO Analysis]] - [[#🎯 Conclusion]] ### 🔗 References - [Lead Dev](https://leaddev.com/) - [GitKraken](https://www.gitkraken.com/) ## 📝 Summary - Engineering leaders discuss measuring AI's business impact beyond adoption metrics - Traditional DORA metrics remain relevant but need context for AI-driven changes - Three-layer approach: culture change, system impact, customer satisfaction - Burnout risks increase as AI enables faster development cycles - ROI measurement requires tying technical metrics to business priorities ## 📖 Main Content Organizations are moving beyond AI experimentation to demanding measurable returns on investment. The panel explores how engineering leaders can demonstrate AI's business value while addressing the complex challenges of measuring productivity, quality, and developer experience in AI-enhanced workflows. The discussion reveals that traditional metrics like DORA (Deployment frequency, Lead time, Change failure rate, Recovery time) remain foundational but require new context. AI adoption creates initial productivity drops before improvements, making attribution difficult when multiple organizational changes occur simultaneously. ## 🗒️ Notes #### ⏱️ [00:00](https://youtu.be/_-j4VBobl-s?t=0) — [H1] Panel introduction and context - Key idea: AI has shifted from experimentation to expectation in organizations - Evidence: "Leadership's no longer asking whether we are using AI they're actually asking what's we are delivering with it" #### ⏱️ [05:30](https://youtu.be/_-j4VBobl-s?t=330) — [H2] Three-layer ROI framework - Key idea: Stacia proposes culture, system impact, and customer satisfaction layers - Evidence: "80% of my developers are using AI is it good enough then no it's like similar to saying that 80% of people are using gym membership" #### ⏱️ [08:15](https://youtu.be/_-j4VBobl-s?t=495) — [H3] Understanding adoption resistance - Key idea: Non-adopters reveal important systemic issues - Evidence: "Those 20% in that situation... diving into those that are resistant is very important especially to understand the cultural impact" #### ⏱️ [12:20](https://youtu.be/_-j4VBobl-s?t=740) — [H2] Baseline measurement challenges - Key idea: Most organizations lack pre-AI baselines for comparison - Evidence: "The answer is going to be don't know come back in 6 months and we probably should have had that conversation you know at the beginning of 2025" #### ⏱️ [15:45](https://youtu.be/_-j4VBobl-s?t=945) — [H1] DORA metrics remain foundational - Key idea: Traditional engineering metrics still apply but need AI context - Evidence: "All of This AI adoption is going to make us do all the things that we should have been doing in the first place" #### ⏱️ [18:30](https://youtu.be/_-j4VBobl-s?t=1110) — [H2] Change failure rate as key indicator - Key idea: Speed increases may hide quality bottlenecks downstream - Evidence: "Change failure rate is a particularly interesting one because we know the speed of releasing code is increasing" #### ⏱️ [22:10](https://youtu.be/_-j4VBobl-s?t=1330) — [H3] Developer experience measurement - Key idea: SPACE framework addresses satisfaction and well-being beyond velocity - Evidence: "Speed will be will the same but if as a part of it we are also improving developer experience" #### ⏱️ [25:45](https://youtu.be/_-j4VBobl-s?t=1545) — [H1] Attribution challenges in metrics - Key idea: Separating AI impact from other organizational changes is difficult - Evidence: "If the changes that we see in this metrics if they are because of AI or because of other things that are happening" #### ⏱️ [28:20](https://youtu.be/_-j4VBobl-s?t=1700) — [H2] Cohort analysis for measuring impact - Key idea: Compare high vs low AI adopters to isolate effects - Evidence: "For people who were using AI like so so not not that deep like shallow cycle time was like flat during the 12 months but for people who did more of AI adoption there was a drop two to three times" #### ⏱️ [32:15](https://youtu.be/_-j4VBobl-s?t=1935) — [H3] Proxy measures for cognitive load - Key idea: Use measurable indicators like PR review time and bug reports - Evidence: "You could say customer reported bugs. You could say the amount of time it takes to review PRs" #### ⏱️ [35:40](https://youtu.be/_-j4VBobl-s?t=2140) — [H1] Communication with executives - Key idea: Tie technical metrics to business priorities CFOs understand - Evidence: "You need to tie to what what business cares and have the storytelling from that point of view" #### ⏱️ [38:55](https://youtu.be/_-j4VBobl-s?t=2335) — [H1] AI burnout risks - Key idea: Excitement and capability of AI tools leads to overwork - Evidence: "People who are really going to the agentic coding... they get into into this doamine kind of excitement when they can't stop working" #### ⏱️ [41:30](https://youtu.be/_-j4VBobl-s?t=2490) — [H2] Monitoring work patterns for burnout - Key idea: Use telemetry data to identify unhealthy work patterns - Evidence: "Looking out for those kind of signals... are commits being made at weird hours of the day PR being submitted at weird hours" ## 📚 References - [DORA Metrics Research](https://dora.dev/) - [SPACE Framework Paper](https://queue.acm.org/detail.cfm?id=3454124) - [Lead Dev Community](https://leaddev.com/) ## 🗣️ Paraphrase - AI measurement requires a layered approach starting with cultural adoption, moving to system metrics, and ending with customer impact - Traditional DORA metrics remain valuable but need context for AI-specific challenges like attribution and initial productivity drops - Burnout risks increase as AI tools enable addictive productivity cycles that require active management and monitoring ## 📰 Newsletter **Measuring AI's Real Impact Beyond the Hype** Engineering leaders are grappling with a new challenge: demonstrating AI's business value beyond adoption statistics. A recent Lead Dev panel revealed that while 80% developer adoption sounds impressive, it's like gym membership numbers—usage doesn't guarantee results. The key insight? Layer your measurement approach. Start with cultural change (depth of adoption), move to system metrics (DORA fundamentals), and culminate with customer satisfaction. Traditional metrics like deployment frequency and change failure rate remain crucial, but require new context for AI-enhanced workflows. **Practical takeaway**: Implement cohort analysis comparing high vs low AI adopters. One organization saw 2-3x cycle time improvements for deep adopters while shallow users remained flat. **Watch out for**: AI-induced burnout. The excitement of enhanced productivity can create addictive work patterns requiring active monitoring of commit patterns and work hours. Resources: [DORA Research](https://dora.dev/), [SPACE Framework](https://queue.acm.org/detail.cfm?id=3454124), [Lead Dev Community](https://leaddev.com/) ## ✅ Action Items #### DevRel - Developer advocacy: Create AI productivity measurement workshops for engineering teams - Education: Develop training on balancing AI productivity with quality metrics - Developer experience: Document AI tool onboarding friction points and solutions - Community management: Facilitate discussions on AI measurement best practices #### OSPO - Policy & Governance: Establish guidelines for AI-generated code review processes - Security Management: Implement scanning for vulnerabilities in AI-generated code - Software Inventory: Track AI-generated code ratios in SBOM documentation - Community Engagement: Support measurement frameworks for open source AI adoption #### Community - Maintainers: Provide templates for measuring AI impact on project velocity - Contributors: Create onboarding materials for AI-assisted contribution workflows - Sponsors: Develop transparency reports on AI tooling ROI investments #### WordPress - Core: Implement baseline measurements before broader AI tool adoption - Core Performance: Monitor AI-generated code impact on performance budgets - Community: Create measurement guidelines for local WordPress communities - Training: Develop curriculum on measuring developer productivity ethically - Documentation: Update contribution guidelines for AI-assisted documentation ## 🧪 Hypotheses - Organizations implementing cohort analysis will achieve 2-3x better AI ROI measurement clarity - Teams without baseline metrics will struggle to demonstrate AI value for 6+ months - Developer burnout will increase 40% in organizations lacking AI usage monitoring - Companies tying AI metrics to business priorities will see higher executive buy-in ### 📊 Methodology - Study design: Longitudinal cohort study comparing high vs low AI adopters - Data sources: DORA metrics, developer surveys, commit pattern analysis - Experiments: A/B test measurement frameworks across engineering teams ### ❓ Questions - How can organizations retrospectively establish baselines for AI impact measurement? - What proxy metrics best predict developer burnout in AI-enhanced workflows? - How do we separate AI impact from other concurrent organizational changes? ### 🔮 Future Research - Long-term sustainability of AI-enhanced productivity gains - Correlation between AI adoption depth and code quality metrics - Cross-industry benchmarks for AI productivity measurement ### 💡 Implications - Practice: Requires new measurement frameworks combining technical and human metrics - Policy: Need governance around AI-generated code attribution and quality - Security: AI code requires enhanced review processes and vulnerability scanning - Education: Developers need training on sustainable AI-assisted work practices - Funding: ROI measurement complexity may delay AI investment decisions ### 🧠 Learning Styles Assessment Reading level estimate: 12th grade #### 👀 Visual learners - Present: Panel format with multiple speakers providing varied perspectives - Improve: Add measurement framework diagrams and ROI calculation flowcharts #### 🔊 Aural learners - Present: Video format with discussion-based learning - Improve: Provide audio-only version with speaker identification #### 🗣️ Verbal learners - Present: Rich definitions of DORA, SPACE, and measurement terminology - Improve: Add glossary with measurement framework comparisons #### 🧩 Kinesthetic learners - Present: Actionable cohort analysis and measurement implementations - Improve: Provide hands-on measurement toolkit and templates #### 🧠 Logical learners - Present: Three-layer framework and systematic measurement approach - Improve: Add decision trees for metric selection and causation analysis #### 🧑‍🤝‍🧑 Social learners - Present: Panel discussion format with multiple expert perspectives - Improve: Add discussion prompts for team measurement conversations #### 🧘‍♀️ Solitary learners - Present: Individual reflection on measurement challenges and solutions - Improve: Provide self-assessment tools for measurement maturity #### 🌿 Naturalistic learners - Present: Discussion of organizational ecosystems and measurement environments - Improve: Add analogies connecting measurement to natural system indicators ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Video has spoken content but lacks descriptive elements - Improve: Add transcript with speaker identification and contextual descriptions #### Keyboard navigation - Present: Standard YouTube player controls - Improve: Ensure all interactive elements are keyboard accessible #### Contrast and readability - Present: Video format may have contrast issues - Improve: Provide high-contrast transcript and text alternatives #### Text resizing and reflow - Present: YouTube player supports standard browser zoom - Improve: Ensure transcript text reflows properly at all zoom levels #### Media and captions - Present: YouTube auto-captions available but may be inaccurate - Improve: Professional captions with speaker identification and technical term spelling #### Forms and interactions - Present: Standard YouTube interaction patterns - Improve: Ensure Q&A submission forms meet accessibility standards ### 🌍 DEIB Assessment #### Cultural breadth - Present: Panel includes diverse professional backgrounds and perspectives - Improve: Include international perspectives on AI measurement practices #### Inclusive language - Present: Technical discussion uses accessible explanations - Improve: Define more technical terms and avoid industry-specific idioms #### Neurodiversity support - Present: Clear section structure with defined topics - Improve: Add visual agenda and topic timestamps for better navigation #### Representation - Present: Mixed gender panel with varied professional roles - Improve: Include perspectives from different company sizes and industries #### Psychological safety - Present: Discussion acknowledges burnout and well-being concerns - Improve: More explicit guidance on creating safe measurement environments ### ✅ Logical Validity #### Strong reasoning - Deductive strength: Three-layer framework logically connects culture to business outcomes - Inductive strength: Cohort analysis evidence supports differentiated AI impact claims - Causal reasoning: Clear attribution challenges acknowledged and addressed - Analogical strength: Gym membership analogy effectively illustrates adoption vs. value gap #### Fallacies - False precision: Tendency to treat metrics as more accurate than they are - Post hoc reasoning: Risk of attributing all changes to AI adoption #### Tone of voice - Overall tone: Professional, collaborative, and evidence-based with appropriate caution about measurement limitations ### 🛡️ DARVO Analysis - Deny: No denial patterns observed - Attack: No attacks on measurement critics - Reverse victim/offender: No victim/offender reversal present - Emotional cues and rhetoric: Balanced discussion acknowledging both benefits and risks ## 🎯 Conclusion Measuring AI's business impact requires moving beyond simple adoption metrics to a layered approach connecting cultural change, system improvements, and customer outcomes. While traditional DORA metrics remain foundational, organizations need new frameworks for attribution, baseline establishment, and burnout prevention. Success depends on tying technical measurements to business priorities while maintaining psychological safety for honest reporting about AI's real effects on teams and productivity. ## 🧱 Refactor Appendix ### 📚 Literature Split Pad Purpose: Stage CEE units that become Literature notes. Split level: H3 ### [H1] Three-layer AI ROI framework provides systematic measurement approach > [!quote] [How to measure the business impact of AI](https://youtu.be/_-j4VBobl-s) by [[Lead Dev]] via [[YouTube]]. Published 2024. Accessed 2024-12-19. > "The first layer is culture, as you start having behavioral change this is already a good sign towards early signal of ROI." Stacia's framework progresses from cultural adoption through system metrics to customer impact, providing structure for complex AI measurement challenges. [view highlight](https://youtu.be/_-j4VBobl-s#H1) #xref #articleseed ID:rw_ai_roi_framework ### [H2] Cohort analysis reveals differentiated AI adoption impact > [!quote] [How to measure the business impact of AI](https://youtu.be/_-j4VBobl-s) > "For people who did more of AI adoption there was a drop two to three times in cycle time compared to shallow adopters." Comparing high vs low AI adopters provides clearer attribution than organization-wide before/after comparisons during periods of multiple changes. [view highlight]( --- ## Full Transcript [00:12] uh and a huge thank you for joining us [00:14] today for the discussion on how to [00:16] measure the business impact of AI. This [00:19] is a lead dev online panel discussion [00:21] creating partnership with geek cranken. [00:24] The discussion will last roughly around [00:26] f 45 minutes after which both myself and [00:29] the panelists will head over to the lead [00:31] dev slack to answer some of your [00:34] questions in the panel Q&A channel. We [00:37] may also have some time to answer a [00:39] couple of questions live. So please do [00:41] submit those through the Q&A feature in [00:43] Zoom and we'll get to them if we can [00:46] towards the end. So let's get started [00:48] with some introductions. My name is [00:50] Habser and today I will be joined by uh [00:54] Stacia Vernon and Jennifer to share [00:57] their expectations, experience and [00:59] insights. I will also uh ask each of you [01:02] the panelists to introduce yourselves. [01:04] We can start with Stacia please. [01:07] >> Hi, my name is Staca and I'm VP [01:10] engineering at Git Kraken. Um, Git [01:12] Kraken is a developer experience [01:14] platform that is trusted by millions of [01:18] developers and as we're improving the [01:21] developer experience, we're also giving [01:22] a way for managers to measure the impact [01:25] of all those changes. Super excited to [01:28] be here. Super passionate about all this [01:30] topic. [01:32] >> Thank you, Jennifer. [01:34] >> Yeah. Hey y'all. I'm Jennifer against [01:36] may know me because sometimes I'm [01:38] hosting some of these uh with lead dev [01:42] or my by line on lead dev. I've been a [01:46] working writer since 2003. Oh no, she [01:50] can't. She looks so young. And I'm a [01:53] freelance journalist. In the last like [01:55] decade or so, I've specialized in the [01:57] developer experience. So a lot of that [02:00] is measuring it because it's engineering [02:03] as a science and you can't improve what [02:05] you can't measure. [02:08] Vernon go. [02:11] Greetings everyone. My name is Vernon. I [02:13] am a senior engineering manager. I'm [02:15] based in the UK. Um and my I've been in [02:18] the software industry sim similar to [02:20] Jenny. Jennifer I've been in the [02:22] software industry for 20 years. But you [02:24] that means you started when you were one [02:26] year old. Is that correct? Um so my my [02:30] area of expertise and the thing that I [02:31] love the most is software testing and [02:33] quality. So super duper interested in [02:37] this uh conversation because I think [02:38] that topic is going to come up. So yeah [02:41] that's me [02:42] >> most definitely. Thank you very much you [02:45] all. Uh AI engineer has shifted from [02:47] experimentation to expectation. [02:50] Leadership's no longer asking whether we [02:52] are using AI they're actually asking [02:54] what's we are delivering with it. [02:57] Depression nice demonstrate real [02:58] business impact. The problem is that [03:00] traditional metrics were not viewed to [03:03] AI powered workflows and adoption [03:06] doesn't equal value. Um so productivity [03:10] gains might affect quality, risk, cost [03:13] and developer experience in complex way. [03:16] So the real question becomes what should [03:18] we measure and how we should connect [03:20] those signals to outcomes leadership [03:22] actually really cares. Today we are [03:25] going to explore those topics. How to [03:27] move beyond vanity metrics, balance [03:30] productivity with quality and risk, and [03:33] build a practical measurement framework [03:35] that helps us clearly communicate AI [03:38] impact to business and leadership. So [03:41] we're going to go through some questions [03:43] um and we're going to share a lot of [03:45] like a good topics, insights, experience [03:47] today. We're going to start with like uh [03:50] the first question about moving beyond [03:52] experimentation. [03:54] Uh so organizations are moving beyond AI [03:58] pilots but not all towards scale. Some [04:01] are deepening analysis, some are seeing [04:03] siloed adoption and only a few are [04:05] really scaling on that topic. What [04:08] changes when leadership start asking [04:11] show me the return of investment? What [04:14] should engineering leaders prepare for [04:17] that moment? Um we can start Estasia [04:22] maybe. Yeah. Yeah. This is this is good [04:25] one. And this is the time when when [04:27] leadership start asking show me ra this [04:30] is the time when we are going from [04:32] excitement piloting all of this positive [04:35] stuff towards the accountability. [04:38] So now we need to start showing the [04:39] results and um return of investment I [04:43] would I would categorize this in three [04:45] layers. The first one is is culture. um [04:48] when it when I'm saying like 80% of my [04:51] developers are using um AI is it good [04:55] enough then no it's it's like similar to [04:59] uh saying that 80% of people are using [05:02] gym membership does this mean that they [05:04] have muscle already or or no but what is [05:07] critical in as a part of the building [05:09] the culture is um the depth of adoption [05:13] so when you start having this behavioral [05:15] change this is already a good sign [05:17] towards like early signal of of the uh [05:20] of of the uh ROI. So the first layer of [05:24] of the cake is the culture. as you start [05:26] changing seeing the changes on the [05:28] culture now uh hopefully we'll start [05:31] seeing the good changes in the u in the [05:34] in the system do we see impact on on [05:37] velocity on quality uh something like [05:41] Dora or is is amazing thing to to to [05:44] measure but at the same time engineering [05:47] team could be running like 100 miles per [05:49] hour in absolutely wrong direction. What [05:52] really matters is um are we are we [05:55] having the impact on customers? Do we [05:57] have more satisfied customers with a [06:00] with a product and this is kind of the [06:03] layered cakes that we need to address in [06:05] order to talk about ROI? But sometimes [06:08] it's hard to start with the big thing. [06:11] So yeah, all three are super critical [06:14] talking about uh ROI impact. [06:18] But one more thing that I'd like to [06:20] mention is um ROI has denominator and [06:23] this is cost. Um right now the the cost [06:28] um is relatively uh not too high as it [06:31] should be. So it will be interesting [06:33] what that denominator will be converting [06:36] over a period of time and how that [06:38] change that we are getting through all [06:40] the layers are actually [06:43] normalized by the cost. [06:47] Absolutely. I agree. What I thought was [06:49] interesting is and we're not sure we [06:50] hear different numbers. So 80% was just [06:53] number you're throwing out there. This [06:54] idea that 80% of developers are using [06:56] AI. Those 20% [06:59] in that situation even if it's 95% [07:03] I think exploring the 5% through [07:06] qualitative conversations not pressure [07:08] because we know it developers really [07:10] react strongly if it's an AI mandate. It [07:13] should be a well-communicated [07:17] decision that we are as a business are [07:20] committing to AI because but not saying [07:23] you must use AI but I think diving into [07:27] those that are resistant is very [07:30] important especially to understand the [07:32] cultural impact like you said because if [07:35] they're reticent to adopt AI it could be [07:37] for reasons that maybe you're using [07:40] tools that are inaccessible to that [07:42] percentage of your workforce that maybe [07:45] cannot see as well or maybe it's like [07:48] bad with color blindness or there could [07:51] be different things or it could be the [07:53] real fear of them losing their jobs or [07:57] something that they're reticent or they [08:00] could be somewhere between a junior and [08:02] a senior and just not finding that [08:04] impact yet because that's where we're [08:06] not seeing as much AI impact. People [08:08] with over five years experience seem to [08:13] find impact because they can pick and [08:15] choose really quickly of what works, [08:18] what doesn't because they have this [08:19] concept. And then junior engineers have [08:21] this perception [08:23] that they're really using AI well, but [08:26] then it's not measurably better in other [08:28] ways like quality and security and [08:31] reliability. So I think [08:34] splitting adoption into longevity in a [08:37] company and in their career is very [08:40] interesting but also understanding who's [08:42] not adopting is a fascinating indicator [08:45] to look at. Vernon, [08:48] >> thank you. [08:50] >> Uh man, I think [08:54] a couple of things are going to change. [08:55] I think one thing that's going to change [08:57] is [08:59] I think the the leaders of our [09:01] organizations are asking this question [09:03] and that's reasonable. I think I think [09:05] the problem is we probably should have [09:07] had some baselines to compare to before [09:12] and so we won't have them and so the and [09:14] so the answer is going to be don't know [09:16] come back in 6 months and we probably [09:18] should have had that conversation you [09:20] know at the beginning of 2025 so we say [09:22] well what are we even doing now and and [09:24] I think I think the second thing is [09:26] there this [09:28] I have a general hunch right which is [09:33] all of This AI adoption is going to make [09:37] us do all the things that we should have [09:41] been doing in the first place [09:44] like you know are you using pipeline [09:47] definitively are you measuring the right [09:49] things all those all those questions I [09:51] don't think they change I just think the [09:53] use of all these tools mean that oh all [09:57] that it's like so I'll I'll use a [09:59] personal example so I I got diagnosed [10:01] with type 2 diabetes a couple of years [10:03] ago [10:04] And the reason you get type two diabetes [10:06] in my in my framing anyway is all the [10:09] stuff that you know you're supposed to [10:10] do to stay healthy, if you don't do that [10:13] stuff, you get type two diabetes. So [10:15] it's like it's like walking, not eating [10:17] lots of processed food and sugar, you [10:19] know, visceral fat, all of this kind of [10:21] thing. And I think the software [10:22] equivalent is going to be the same. It's [10:25] going to be things like CICD, are you [10:27] measuring the right things, etc., etc., [10:30] etc. we know all this stuff but now [10:31] we're going to be forced to actually do [10:33] it because otherwise we won't be able to [10:34] able to answer this question. [10:38] >> Definitely. Well, very good. Very good. [10:40] Thank you very much. Uh I feel like we [10:42] also mentioned about metrics things that [10:44] we have been doing before and how to [10:47] move towards this new reality and then [10:49] it comes the question about metrics and [10:51] the metric gap. uh we know many managers [10:55] they lack clear AI impact metrics that [10:58] helps them a lot to understand the goals [11:00] and how to move forward. Why do you [11:03] think traditional engineering metrics [11:04] fall short in measuring AI impact and [11:07] where do they break down? [11:12] >> Um Jennifer [11:14] >> I was going to say could we say that all [11:16] engineering metrics have always fallen [11:19] short except at a few exceptional [11:21] companies. So that [11:24] AI is AI is just more of everything. [11:28] There's some other factors in like we're [11:30] working with both humans and machines [11:32] that may or may not be doing things [11:34] without our consent and doing thinking [11:36] and all. [11:38] But just like with Vernon said, this is [11:42] bringing up this whole idea that well, [11:45] you're not going to be successful with [11:46] AI if you didn't have the engineering [11:48] best practices in place. If you haven't [11:50] done the things the Door of Report has [11:51] been saying for a decade, [11:55] you're not going to be great. But then [11:58] also 70% of Fortune 500 companies have [12:01] failed to move to the cloud. So there's [12:04] got to be somewhere in there. But if [12:06] you're not measuring change, again, [12:08] engineering is a science. So I'm just [12:10] guessing that they haven't been that [12:12] much. But I just thought like maybe we [12:15] should if you don't know Dora, it's [12:20] just thought let's like talk about those [12:22] four metrics. So Dora, I I don't work [12:26] for Google. I just love data as a [12:29] journalist. And Dora is an annual [12:31] survey. It's free and anonymous to [12:36] enter or respond to. And it has about [12:40] 150 questions I want to say [12:42] that get to all these core aspects of [12:45] your team and lets you kind of benchmark [12:48] against other [12:50] engineering forward organizations or [12:53] companies in general now because they're [12:54] all tech companies, aren't they? And [12:56] then the core four or maybe five are [13:00] change lead time, deployment frequency, [13:03] change failure rate, and then meantime [13:06] to recovery or now it's called failed [13:08] deployment recovery. So those are like [13:11] some really good ones to be constantly [13:13] monitoring and I think they're some of [13:15] the simpler to monitor with tools. [13:19] So do that. But I think like change [13:23] failure rate is a particularly [13:24] interesting one because [13:28] we know the speed of releasing code is [13:30] increasing. [13:32] But that doesn't mean they're not just [13:34] shoving bottlenecks down further. [13:37] Maybe it's like a snake trying to eat a [13:39] rat because the bottleneck doesn't [13:40] really work in this metaphor. Um just [13:43] keeps shoving down and choking on it. [13:45] And [13:47] so this rate of rework and percentage of [13:51] time spent on new capabilities is a [13:53] really interesting comparison because if [13:56] you're just spending more time on code [13:57] reviews, [13:59] if you're just spending more time in the [14:01] first six months [14:04] reworking something versus six months [14:07] after or later than six months, if [14:11] you're trying to find out what went [14:13] wrong all the time, that's just a new [14:15] level of frustration for developers [14:17] because cannot stress it's not [14:19] interesting them writing more code [14:22] because that was never the most [14:23] interesting [14:25] well developers have always wanted to [14:27] write code but that was never the [14:28] bottleneck to speed so percentage of [14:33] code percentage of code written by AI is [14:36] at least interesting I'm not sure if it [14:38] tells us anything about productivity but [14:40] it's an interesting number but there's a [14:44] lot more interesting things about both [14:46] the AI generated code and what the code [14:48] developers are creating right now if [14:50] they're creating code who knows [14:53] Jen 100% agree with you on on the DORA [14:57] metrics it's it's traditional one [14:59] lasting for for for a long time already [15:03] and this is definitely what we need to [15:04] be measuring um as we are driving into [15:08] more and more of AI adoption [15:11] one uh element that I'd like to add here [15:14] is that um dorometrics is assuming the [15:16] signal it's speed and think of this that [15:19] you're you're going on a car without AC [15:22] like with with like something that is [15:24] very hard to manage and you're still [15:27] moving with 50 kilometers per hour right [15:31] um but with AI there is a chance of [15:34] putting this AC like nice uh audio [15:37] system so the experience of the [15:39] developer could be could be improving so [15:42] speed will be will the same but if as a [15:45] part of it we are also improving [15:47] developer experience and we can get this [15:49] through space metrics through um uh [15:53] through voice of developer and trying to [15:55] understand what's satisfaction or very [15:58] often dissatisfaction I think this is um [16:02] um this is interesting element to to [16:04] keep an eye on one more thing on the [16:07] dorometrics um yeah we should have the [16:12] baseline at the at the very beginning [16:14] but the thing is that um AI wasn't [16:18] happening at moment one and then [16:21] everyone just start adopting at at at [16:24] the pace and it was such a clean system [16:26] that nobody nothing else was happening [16:29] in in the engineer no process other [16:31] process changes no like changes on on on [16:34] the things that we are working on so [16:36] with with dora metrics um it's we also [16:40] need to address the question of [16:42] attribution ution if the changes that we [16:45] see um in the in this metrics if they [16:48] are because of AI or because of other [16:50] things that are happening um and then [16:54] through DOR now we have this now it's [16:56] not 50 kilometers per hour now it's now [16:58] it's 100 kilometers per hour that that [17:00] that we are we are moving um how [17:03] sustainable it is so we can see like a [17:06] good spike it's it's it's going up but [17:09] is it because of future burnout like [17:13] this um there's this post about AI [17:16] vampire um is it sustainable longer term [17:20] even if like everything is good from the [17:22] door metrics we have good speed we have [17:24] good quality um not that much of the [17:26] incidents that are happening after that [17:28] but are we losing people after that and [17:31] we as leaders also need to take care of [17:33] that too [17:36] >> nice thank you Vernon do you have to do [17:38] you want to add something we can jump to [17:41] the next question as well [17:42] >> only only that I think what Stalia said [17:45] about burnout [17:47] is if if it if if traditional metrics [17:51] fall down anywhere it's that because I [17:53] think the consequence of AI is that [17:56] we're all doing more [17:58] and so measuring that side of things you [18:02] know we're measuring the the health of [18:05] our systems I think we need again we're [18:07] going to be forced to measure the health [18:08] of our humans that are maintaining the [18:10] systems because They're probably [18:11] working. They're probably redlinining [18:14] and they probably don't even realize it. [18:15] So, [18:17] >> absolutely. And just because Stashia [18:19] mentioned the space framework and as [18:21] journalists I like to define things and [18:23] clarify because we love an acronym. [18:25] We're just as bad as the UN with them. [18:28] uh space stand is like the brings [18:30] together all these different [18:32] interdimensions of developer [18:34] productivity and it's an acronym that [18:36] stands for satisfaction well-being like [18:39] like Vernon said which I don't I think [18:41] is probably worryingly the one that [18:44] companies are measuring the least [18:46] performance [18:48] historically measured really poorly in [18:50] all companies and annually activity [18:54] communication and collaboration and [18:56] efficiency and flow which I think these [18:58] are these were measurements pre the AI [19:02] of it all but they're even more [19:06] impactful and we shared it in the chat [19:09] the space framework like academic paper [19:12] but there are many questions [19:15] to get to these answers of how your [19:17] developers are feeling because we know [19:19] happy developers are more productive [19:24] >> thank you and I would like to uh go [19:26] deeper from that one like developer [19:29] developer experience and then um we have [19:31] been talking about metrics and then how [19:33] AI intensifies if you're going doing [19:36] really well or if you're not doing so [19:37] well as a company and now um the title [19:41] of this presentation is like well you're [19:43] talking with executive and CFO so I want [19:45] to bring that perspective into this [19:47] questions uh and then regarding to [19:50] development experience as a business [19:52] metric and talking to the CFO and [19:55] executives [19:56] AI can reduce cognitive load and [19:58] friction as we discussed but also can [20:00] increase initially. How do we quantify [20:04] improvements in the development the [20:06] development experience in a way that [20:08] resonates with the CFO and these [20:11] equities like talking the language that [20:13] they actually understand. [20:18] >> Yeah. In this case we need to um there's [20:22] no oneizefits all. So you need you [20:26] every company is different in the in [20:28] this form and we as leaders we need to [20:31] understand what are the priorities for [20:33] for a company right now and um it it [20:37] could be that um improve customer [20:39] satisfaction because stability is not [20:41] there. It could be that we want to have [20:43] more experimentation. We it could be [20:45] that we want to deliver more more [20:48] things. So these are the different [20:50] signals that we should be capturing um [20:52] that that executives do care and then we [20:56] need to start translating to the uh [20:59] language of of engineering metrics. Um [21:03] so something like [21:07] faster um a faster uh cycle time. That [21:11] means that now we can we can have uh [21:13] more more experiments that are running [21:15] in a permit like when you're trying to [21:16] do product market fit or launch some [21:19] some some new type of functionality then [21:21] these are the metrics that you should be [21:23] connecting that now we can do much more [21:26] a AB AB um um tests and and experiments [21:31] sometimes um companies do care about the [21:35] the um the capacity and like how much we [21:39] can do like with with less. [21:42] >> So again like this is something that [21:43] that we could convert. So some advice [21:47] from me is um really be be really really [21:50] aware what are the top priorities for [21:53] the company and then try to um organize [21:58] your [21:59] um your initiatives that will be helping [22:02] to address those those constraints in [22:05] the system and pick up the right metrics [22:09] to articulate the impact. And as Veron [22:12] was mentioning before that we need to [22:14] have the baseline. Um so this is like [22:17] before doing the changes you need to [22:18] know the baseline and you're setting the [22:20] target. I want this to improve for the [22:24] reasons that company needs that. What [22:27] I'm curious about is that baseline, [22:29] Vernon. [22:31] Everything's moving so fast and this [22:33] pressure to adopt AI is so fast and [22:36] you're right, a lot of baselines weren't [22:38] there and a lot of things were [22:40] undermeasured. [22:41] So, how do you balance that like absurd [22:44] speed [22:46] with the need to stop and get a [22:48] baseline? [22:51] >> Yeah, I I'm I'm with you. I don't think [22:53] stopping is realistic like this. So I [22:56] think it would be a matter of the only [22:59] thing that I can I can think of um to [23:02] try and mitigate that is [23:06] we often use tools like Jira. So there [23:09] may be data lurking around in there that [23:12] you can perhaps mine and reverse [23:13] engineer and say well you know maybe I [23:16] can figure out what was what's [23:17] historically what does our lead time [23:19] look like and kind of use that as a as a [23:22] comparison to what's going on now. But [23:25] if you don't have that data, you're [23:26] gonna just have you're just gonna have [23:27] to start from where you are now and look [23:31] for look for the trends going forwards, [23:33] I think, because yeah, I think you're [23:35] right. I mean, generally speaking, we [23:38] haven't, you know, as an industry, you [23:39] haven't done a great job of measuring [23:41] these things. And then for this [23:43] question, if you're talking about [23:44] cognitive load, I mean, [23:47] >> how do you even measure that anyway? And [23:48] I can guarantee that 99.9 recurring [23:52] companies are not measuring for that. [23:54] So I think the best thing you can do in [23:56] that situation is look at like a like a [23:58] proxy measure you know. So if you could [24:01] you could make a bet or you could make a [24:03] kind of hypothesis that if the cognitive [24:06] load increases [24:08] you know errors and bugs will also [24:11] increase and then you can measure that [24:14] to see if that's true and you can see [24:15] what's going on. So you're looking for [24:17] for proxy measures a lot of the time for [24:19] the cognitive load stuff. What would be [24:21] like an example of a proxy measure of [24:23] the [24:26] >> like you you could say customer reported [24:27] bugs. You could say the you could say [24:30] the amount of time it takes to review [24:34] PRs if if humans are still reviewing [24:36] them. Uh [24:38] right because that's a thing and and [24:40] things like that. So what are the what [24:42] are the what are the things where are [24:44] the moments where people have to reason [24:47] about stuff and is and and is that [24:48] number going up or is it going down? Is [24:50] it going in the wrong direction? [24:52] >> That's true. We found that uh the number [24:56] I I forget the exact number, but it was [24:58] 50% increase in some way of developers [25:03] spending more time finding bugs in AI [25:05] generated code because they didn't do it [25:08] and also somebody that doesn't [25:10] necessarily speak human didn't create it [25:13] either. So it's much harder to find. So [25:15] that increases it. What do we think [25:17] about the measurement of lines of code [25:20] as a signifier? [25:23] I'm saying I it doesn't mean it's a [25:25] positive signifier, but should we be [25:28] measuring lines of code at least because [25:30] lines of code is a signal of increased [25:32] complexity. [25:34] >> Oh, okay. Okay. Okay. I retract my uh [25:38] vomitus reaction there. My allergic [25:40] reaction. Oh, sorry. That that that [25:42] might that might work actually. that [25:43] might it be it'd be worth investigating, [25:45] right? Um it'd be worth experimenting [25:48] with. But yeah, my general reaction to [25:50] lines of code is allergic reaction and [25:53] basically vomit. [25:55] >> I'm with I'm with you and this is the [25:58] metrics like we can add to the prompt [26:00] and like make make the the lines of code [26:03] as as big as as possible and then you'll [26:06] you'll got it like you will you'll be so [26:08] successful with with with the with the [26:10] numbers. Uh Jen to to your question the [26:13] ba baseline there was um some funny [26:16] answer like like remove it like for for [26:19] for a couple of days remove it and he'll [26:21] get to the baseline like I I'm not a big [26:23] fan of that of that answer but um uh you [26:28] need to measure and it's better later [26:31] than than never. Um what you still can [26:34] do while you don't have the baseline [26:36] metrics is um doing cohort analysis like [26:40] for for example for for us um we were [26:43] trying to increase the the the time for [26:45] experimentation and we pick up uh people [26:48] um based on the AI level of adoption [26:52] um the the metrics that we are getting [26:54] like the frequency of the use the depth [26:56] of the use and put into into cohorts and [26:59] then uh apply this to Dora metrics and [27:02] um something like cycle times was the [27:05] metrics that we were trying to optimize [27:07] with guardrail metrics around the the [27:09] change failure rate and uh you could [27:11] clearly see like for people who were um [27:14] using AI like so so not not that deep [27:19] like shallow um cycle time was like flat [27:23] during the 12 months but for people who [27:26] uh who did um more of AI adoption there [27:30] was a drop two to three times So I think [27:33] cohort analysis could be could be a good [27:35] one for um for getting to the the the [27:40] impact. [27:41] >> Then how do you turn that actionable for [27:43] those that are the they have low daily [27:46] usage or or daily active usage is low? [27:52] How do you find out why they have low [27:55] usage? because something's not clicking [27:58] for them because they are using they've [28:01] they've signed up maybe they use it once [28:03] a week or something. How do you figure [28:05] out then why they aren't using it or [28:08] what's not working for them? [28:10] >> Yeah, matrix matrix is not obsolete. [28:12] This is not like you're just looking at [28:13] the matrix and like not talking to the [28:16] people like uh this is this is not the [28:19] right way to to to do this. So metrics [28:22] is the beginning of the conversation and [28:24] conversation could be with starting um a [28:27] survey um and and trying to get the [28:30] signals signals across all the people [28:33] because um like you may learn lots of [28:35] good things even from people who are [28:38] high adopters. So this could be one but [28:40] then also starting to talk to people uh [28:43] what what exactly it could be and it [28:45] could be that there is not enough [28:47] training there is um not confidence in [28:50] AI generated code and there will be like [28:52] different strategies depending on what [28:55] you learn through through this u not not [28:58] quant uh quantit like going to like uh [29:03] actually the the details [29:08] >> also like a uh [29:12] I just want to we have been also talking [29:14] a little bit about like the metrics and [29:16] then showing this um there things are [29:20] changing so fast and then sometimes the [29:22] information that we had one month ago [29:24] actually it's outdated so there's like a [29:27] lot of questions about okay how do I [29:29] tell the this story you know like how I [29:32] can communicate through this [29:34] organization or in this in the case here [29:37] to our executives is to actually [29:39] understand and trust that. So do you [29:41] have any tips about like a formats like [29:44] narratives, forums, the way that we [29:46] could approach this in such like a like [29:50] light speed changing environment that [29:52] we're going through now. [29:59] Please like whoever wants to jump in and [30:02] then talk about it. [30:06] Yeah, for me it's exa exactly the same [30:09] thing like uh you need to tie to what [30:11] what uh business cares um and and um [30:18] have the storytelling from that point of [30:20] view. Um, so it's it's similar approach [30:23] as uh uh Yeah. [30:27] >> Yeah. [30:28] >> I don't Yeah, that's what Yeah. I It [30:30] doesn't feel like it's different. [30:33] It feels like it's the same. And it's [30:36] it's kind of what Staryi said in terms [30:38] of [30:40] you got to you got to you got to frame [30:42] everything that's going on [30:45] in terms that your you know your exact [30:48] team care about. [30:51] So for our CFOs that is you know the [30:54] easy one is going to be cash [30:56] money. [30:57] >> So how can you how can you turn [31:00] developer experience [31:03] you know survey data and you know [31:05] metrics into and here's and here's what [31:09] this means for our bottom line [31:11] >> in a way that gets your CFO those [31:13] attention or gets them excited or [31:14] whatever. [31:15] >> And I don't think that's I don't think [31:16] that's different. Um, [31:19] >> but I mean maybe maybe I need to think [31:21] on it some more. [31:23] >> I guess the biggest cost in most tech [31:26] companies what and I'm including like [31:29] banks and stuff in there is the salary [31:32] of engineers and your cloud cost. So [31:36] those are deeply affected by AI. But [31:39] then it's just about tying it back to [31:41] business metrics. I don't the [31:45] I love the salary proxy when we're [31:48] talking about advocating for platform [31:50] engineering because it makes developers [31:52] lives better and you should invest it if [31:54] you're spending that much. But I hate it [31:57] for anything tied with productivity [31:59] because that usually is a euphemism for [32:01] layoffs. [32:03] >> So I'm very cautious about that. But I [32:07] think it's I actually have an example [32:09] from Spotify from a couple years ago [32:11] because I've been writing about and [32:14] engaging with on teams OKRs, objective [32:18] key results. [32:19] >> Yeah. [32:19] >> Different from KPIs, but KPIs you can [32:21] have too um for over a decade because [32:25] nobody like very few companies get them [32:27] right. I think it's very difficult. [32:31] So I'm going to give this example and [32:34] it's not grounded in AI but you can see [32:37] AI as some of the use cases and I have [32:39] reached the age I have to take off my [32:40] glasses to read it. [32:43] And so this idea of you need to measure [32:47] the leading and the lagging data and you [32:49] need to pull it all together to come [32:51] back to tying the [32:54] business [32:56] the work of engineers which can seem [32:59] very technical to business objectives. [33:02] Okay. So an engineering department could [33:06] have an OKR, I guess a KR on an [33:10] objective and then measurable key [33:12] results on the lagging metric of [33:14] meanantime and recovery. And then a [33:18] platform team supporting SRRES could [33:21] have a leading metric of log ingestion [33:24] speed. And these would both be in [33:27] support of company level OKR, which is [33:30] to increase customer satisfaction, [33:33] which could be [33:36] measured by things like net promoter [33:38] score, active users, churn rate, and [33:40] this is straight out of Spotify. So I [33:42] guess in Spotify, a lot of turn rate [33:44] would be canceled subscriptions because [33:47] things are too slow. So by focusing on [33:49] something on providing more [33:53] data and improving the S sur or [33:56] whichever operator developer experience [34:00] it then [34:02] improves the business experience. So it [34:05] involves like translate and I'm going to [34:07] say something crazy here and do not and [34:10] in no way do I ever say put IP [34:12] information into AI for this. Could AI, [34:16] because it's good at explaining things, [34:19] be a tool for technically [34:23] minded people [34:26] to help translate their technical [34:29] achievements and connect it to business [34:32] objectives. Could it help them phrase or [34:35] word it again with a special paid for [34:38] version of a of AI, but not putting like [34:42] extreme IP in there? But I don't know is [34:45] that something that you all could see as [34:47] useful in helping you have these [34:49] conversations with business or those [34:51] that are technical. [34:54] >> Yeah. And we could put this this kind of [34:56] layered of of um like what what is the [34:59] thing that we are trying to optimize? [35:00] What the system metrics? What's the [35:02] leading indicator? Um one of the [35:04] challenges is that you see let's say [35:07] spike in adoption of of AI. But most [35:11] likely what you will see from the very [35:13] beginning is a drop on your system [35:15] metrics because this is changes in your [35:17] organization. It will be always like [35:19] first slow down before you see the [35:21] results. Uh so so that's that's why [35:23] there will be always like this waves. So [35:25] you you have the improvement on on the [35:28] behavioral thing then there will be a [35:30] drop initially on the system hopefully [35:33] followed by by the by the growth and [35:36] then after some time you will start [35:39] seeing the impact on the on on the [35:41] business and the thing is that that [35:44] attribution again like what we were [35:46] discussed is is hard thing to do because [35:49] business impact they will be not [35:51] immediate and often like delayed by [35:53] quarter. So yeah that that's that is [35:57] like but having that that mental model [35:59] of starting from the end of what we're [36:02] trying to optimize and what are the [36:03] metrics that are needed and convert it [36:06] into the uh engineering or cars. I think [36:10] this is this is absolutely the right way [36:11] to to to go. [36:17] >> I um we have one on one qu we need to [36:21] wrap up. It's very exciting like there's [36:23] like a lot of uh good questions, good [36:25] ideas, insights uh [36:28] like uh but but but before that I just [36:31] want to say one thing like I think we [36:33] during this discussion like we talked [36:35] about it's very important I I forgot [36:38] sorry I forgot who said that but it's [36:39] important that also the executive and [36:41] CFO they also understand what you can do [36:43] with u uh AI. So it's very important [36:46] that you actually give them like a [36:48] computer with like all the the [36:50] environment properly set up and all the [36:53] tools there so that also they can [36:54] understand how is uh the process of [36:57] getting on boarded and then um see like [37:01] a little bit about productivity and I [37:03] think this is like a key point for [37:05] making them understand all of the things [37:07] that we have been discussing and then [37:09] it's so funny to see in uh in the media [37:13] today especially like stockish exchange [37:16] the markets. People like really having [37:19] crazy hallucination. They they are [37:21] hallucination hallucinating themselves [37:23] about what's going to be the future of [37:26] like the world without with AI. So some [37:30] weeks ago they were considering that [37:32] they would we wouldn't have like [37:34] software companies anymore because of [37:36] that right. So it there are like so many [37:38] things going on and then it's very [37:40] important that executive like leadership [37:42] they understand what actually you can do [37:44] and not what so what you cannot do with [37:46] like a tool such AI and then I I have [37:50] seen myself that it's much better to [37:52] pass through those metrics that we are [37:56] those numbers we are measuring the [37:58] baseline discussion uh all of that I [38:01] think that's like they can build up much [38:03] more knowledge because they have gone [38:06] through the the the tools and then they [38:08] could understand how like may I think [38:12] it's also we have this sort of like as [38:15] leads we also need to kind of have this [38:18] responsibility to also on board them [38:20] into AI if if they haven't been yet. So [38:23] I think also it's going to make our [38:24] lives much easier. Um but yeah um just [38:29] like bringing some of the questions from [38:32] the Q&A and then this also can spark al [38:34] also other discussions. We have one here [38:38] uh talking about I think it's more like [38:40] a comment but it's like a very recurrent [38:42] topic. There's a risk that with AI the [38:44] expectations to do more is rising. So [38:47] the risk of burnout is increasing [38:51] like [38:52] um and al another one about also mental [38:55] health um how can we take care of our [38:58] mental health in these new times. [39:02] Yeah, this is this is absolutely true [39:04] and this is not only um expectations [39:07] from the leadership like hearing all [39:09] this uh all this news um that hey [39:13] developers should be right now 10x and [39:15] then they're coming top top button uh [39:19] saying that why we are not having 10x [39:21] like what's going on so this is this is [39:23] one level of pressure but also another [39:26] pressure that that that I see is people [39:29] who are really going to the agentic um [39:32] uh coding. And usually this is this is [39:34] true product engineers who know what [39:37] what customers need and they they build [39:39] they get into into this doamine um kind [39:42] of excitement when they can't stop [39:45] working because they they're doing [39:48] something and they see the outcome right [39:50] away and they want to do more and they [39:52] want to do more. And we we already had [39:54] some of some of the people who were [39:56] burned out by by this excitement and [39:59] this power um that you you can you can [40:03] really change the world. So now this is [40:05] like what do we do with this as leaders [40:08] and um yeah so we need to take care of [40:11] our people. We need to bring awareness [40:13] that this is happening. We need to to [40:15] set up some of um of hours. Uh we we [40:18] need we need to um uh to to talk to to [40:22] people that like especially who are at [40:24] that scale like hey [40:28] you will be delivering so much more if [40:31] you're if you're in a good mental state [40:34] so let's do let's be in this together. [40:40] >> Thank you. [40:42] >> Yeah. Yeah. I'm just not sure how to [40:45] measure these things. [40:48] because you can there's like we haven't [40:50] talked a lot about qualitative but [40:51] regular qualitative developer surveys [40:53] that are actionable or very useful maybe [40:56] quarterly [40:58] but the only thing is warning you can't [41:02] measure this if you don't have [41:04] psychological safety we are not in a [41:06] psychologically safe time right now [41:09] people do not feel secure about their [41:11] jobs so [41:14] you are not going to get the right [41:16] response for a leading or lagging [41:19] indicators of burnout in these human [41:22] surveys if you've not made it safe that [41:25] is these surveys are anonymous [41:29] and that people can be candid and people [41:32] feel safe in their work. So, I'm not [41:34] sure the questions to ask and everything [41:38] until you can [41:42] like make sure your employees feel safe, [41:45] which I'm honestly in this world, who [41:47] knows? But it's a tough. [41:49] >> Yeah. And they they could be trusting uh [41:52] but but the thing is that they maybe are [41:55] not aware yet. So, it it's it's that [41:58] kind of thing. But we can look at the [42:00] telemetry at the usage and if you see [42:02] that like it's strike for like hours and [42:06] hours this could be a signal for us. [42:08] Sorry Vernon sorry [42:10] >> I was going to talk about signals. I was [42:12] going to talk about a couple of things. [42:13] One is from a leadership perspective [42:15] looking out for those kind of signals [42:17] you know are are commits being made at [42:20] weird hours of the day PR being [42:22] submitted at weird hours of the day by [42:24] humans of course that could be one [42:26] thing. Um, and then as as an as a person [42:30] in a team, as an individual contributor [42:31] in the team, forgive me, I cannot read [42:34] your name who submitted the question and [42:36] then the follow-up comment. But that [42:37] follow-up comment is the thing like you, [42:41] we are going to have to start building [42:43] in a level of discipline around making [42:46] sure that we take breaks and getting [42:49] away from our machines. It's a it's a [42:52] bit of a weird one and it well on the [42:54] surface level it might seem like a bit [42:56] of a weird thing to talk about when we [42:58] when we're discussing you know the ROI [42:59] of of AI but I think [43:04] uh star was right the exhilaration of [43:06] using these tools is itself the actual [43:09] danger and so you'll just find yourself [43:11] doing stuff at all hours day and night [43:14] so pulling yourself away [43:16] is what we need to do. It's concerning [43:19] because it's the opposite of what was [43:21] promised. [43:23] >> Indeed. [43:24] >> We were supposed to be working less. [43:27] >> Indeed. [43:28] >> Yeah. Yeah. [43:30] >> Seems good metrics for the seauite to [43:32] have to know their well-being of their [43:34] staff and to start measuring those and [43:36] not to measure at the individual level [43:38] because it's not about the staff at that [43:40] point. You're doing something wrong as a [43:41] manager too. [43:42] >> That could go sideways pretty quickly. [43:45] Yeah. Imagine what the number like the [43:48] cognitive uh switch is so much like with [43:51] the uh with team of agents and then then [43:54] you you switch context so quickly and to [43:58] to what you said Jennifer is like [43:59] actually you're doing much more and [44:01] switching context much faster than [44:03] before because now you have like a team [44:05] of agents and then looking into the [44:07] outcome of each of them. Um and then [44:11] like following up on all of that uh [44:14] reviewing the code and then a lot of [44:17] like PRs to review every day. So yeah, [44:20] it definitely is increasing more and [44:23] more by the day [44:27] but like I would like to add also if you [44:29] have any more comment to end also like [44:32] in a high note as well. [44:36] I know you wouldn't say it's a high [44:37] note, but metrics have power and the [44:41] board is seeing the metrics. So be [44:44] careful. Uh we we talked a lot about how [44:47] you need this baseline. You need to [44:48] measure a lot of stuff to improve. But [44:50] on the other hand, be careful that the [44:53] metrics you are measuring are not [44:55] harming the people because I still [44:58] believe businesses were created to hire [45:01] people and they have a duty to the [45:03] employees. [45:06] Yeah, matrix is just the beginning of [45:08] the conversation. It's not the full [45:10] story. But at the same time, um, Jen, as [45:13] you told, you can't improve what you're [45:15] not measuring. [45:19] >> Um, I think all I'd add is [45:22] don't overblow the metrics in the sense [45:24] of like, you know, hype them up to be [45:26] more precise and [45:28] giving you information and telling a [45:30] story that they're not telling. [45:33] And that was exactly like yeah high tone [45:37] wasn't the good word here Jennifer but [45:39] exactly like this like okay how can we [45:41] see that in a different way that's a [45:43] great uh way to to show to like uh the [45:47] audience that uh yeah like look at this [45:50] in a different way and then also like [45:53] you have like a lot of time as well to [45:56] think about the narrative think how you [45:58] communicate through the through the [46:00] organization and also [46:02] upwards to the leadership and CFO in [46:05] this case or not. But like also like [46:07] remember that also you have this time to [46:09] bounce back with other people in your [46:12] team in your organization to to best [46:15] communicate that to the world. So thank [46:17] you very much. We are out of time now [46:20] and we are heading to the Slack channel [46:23] uh for Q&A there as well. So please if [46:25] you're interested go there and then drop [46:27] your question. we all will be there also [46:30] answering and continue our discussions. [46:32] Thank you very much Stacia, Jenny [46:35] friend, Vernon, you all have a great [46:37] day, evening, morning like wherever you [46:39] are. Uh and uh yeah, great to talk to [46:43] you all. Thank you. [46:45] >> Thank you. [46:46] >> Thank you. [46:47] >> Yeah.  📄 **Full text:** ![[Full Document Contents/Articles/How to Measure the Business Impact of AI|How to Measure the Business Impact of AI, YouTube]] ## Highlights - uh and a huge thank you for joining us today for the discussion on how to measure the business impact of AI. This is a lead dev online panel discussion creating partnership with geek cranken. The discussion will last roughly around f 45 minutes after which both myself and the panelists will head over to the lead ([View Highlight](https://read.readwise.io/read/01km99d9899zg1zgvmvxxxgak0)) - Tags: 👻 ai highlighted - traditional metrics were not viewed to AI powered workflows and adoption doesn't equal value. Um so productivity gains might affect quality, risk, cost and developer experience in complex way. So the real question becomes what should we measure and how we should connect those signals to outcomes leadership actually really cares. Today we are going to explore those topics. How to move beyond vanity metrics, balance ([View Highlight](https://read.readwise.io/read/01km99d98r0wghgrgw0wv0setk)) - Tags: 👻 ai highlighted - productivity with quality and risk, and build a practical measurement framework that helps us clearly communicate AI impact to business and leadership. So we're going to go through some questions um and we're going to share a lot of like a good topics, insights, experience today. We're going to start with like uh the first question about moving beyond experimentation. Uh so organizations are moving beyond AI pilots but not all towards scale. Some ([View Highlight](https://read.readwise.io/read/01km99d98zydhvvfch18axepzp)) - Tags: 👻 ai highlighted - is the time when we are going from excitement piloting all of this positive stuff towards the accountability. So now we need to start showing the results and um return of investment I would I would categorize this in three layers. The first one is is culture. um when it when I'm saying like 80% of my developers are using um AI is it good enough then no it's it's like similar to uh saying that 80% of people are using ([View Highlight](https://read.readwise.io/read/01km99d99beeayehqbwfqxpwyv)) - Tags: 👻 ai highlighted - etc. we know all this stuff but now we're going to be forced to actually do it because otherwise we won't be able to able to answer this question. >> Definitely. Well, very good. Very good. Thank you very much. Uh I feel like we also mentioned about metrics things that we have been doing before and how to move towards this new reality and then it comes the question about metrics and the metric gap. uh we know many managers they lack clear AI impact metrics that helps them a lot to understand the goals ([View Highlight](https://read.readwise.io/read/01km99d9b2f486n2xefqkjw855)) - Tags: 👻 ai highlighted - understand what are the priorities for for a company right now and um it it could be that um improve customer satisfaction because stability is not there. It could be that we want to have more experimentation. We it could be that we want to deliver more more things. So these are the different signals that we should be capturing um that that executives do care and then we need to start translating to the uh language of of engineering metrics. Um ([View Highlight](https://read.readwise.io/read/01km99d9d87xqxebc85h0rpsc0)) - Tags: 👻 ai highlighted