Measuring team performance isn’t getting any easier 🔗 **Source:** [Measuring team performance isn’t getting any easier](https://www.youtube.com/watch?v=L0ZTdw5flsw) > [!note]+ Document Notes > tags: [engineering-metrics, team-performance, ai-in-software] > author: [MISSING] > source: [MISSING] > reference: [https://www.youtube.com/watch?v=L0ZTdw5flsw] > date created: > > # 🏷️ Title: Measuring team performance isn’t getting any easier > <!-- deep-read:sections start --> ## 🧰 Action Inbox - [H3] Draft a WordPress.org-friendly metrics rubric — blend DORA + SPACE. https://leaddev.com/reports/engineering-team-performance-report-2025 #articleseed - [H5] Pilot a “DORA + outcomes” dashboard — pick two business KPIs. https://cloud.google.com/devops/state-of-devops #xref - [H6] Add monthly qualitative health checks — rotate facilitators. https://www.youtube.com/watch?v=L0ZTdw5flsw&t=880 #addendum - [H10] Track post-release quality signals — incidents, reverts, churn. https://www.youtube.com/watch?v=L0ZTdw5flsw&t=1547 #articleseed - [H12] Propose an L&D OKR for AI skills — report quarterly. https://leaddev.com/reports/engineering-team-performance-report-2025 #share - [H14] Run curiosity clinics — pair seniors with AI-native juniors. https://www.youtube.com/watch?v=L0ZTdw5flsw&t=2225 #share - [H15] Document “AI done right” playbook — prompts, reviews, guardrails. https://www.youtube.com/watch?v=L0ZTdw5flsw&t=2445 #articleseed - [H11] Require context with metrics — anomaly notes by owners. https://www.youtube.com/watch?v=L0ZTdw5flsw&t=1779 #aphorisms - [H17] Start peer-led AI hours — weekly, agenda-less, show-and-tell. https://www.youtube.com/watch?v=L0ZTdw5flsw&t=2781 #articleseed ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - LeadDev x O’Reilly Engineering Team Performance Report 2025. https://leaddev.com/reports/engineering-team-performance-report-2025 - SPACE framework (Microsoft Research). https://www.microsoft.com/en-us/research/publication/the-space-of-developer-productivity - DORA metrics overview (Google Cloud). https://cloud.google.com/devops/state-of-devops ## 📝 Summary - The panel reviews the 2025 report with O’Reilly. It stresses persistent difficulty. [Report link](https://leaddev.com/reports/engineering-team-performance-report-2025). - Leaders value qualitative context more. They still track delivery health with DORA. Yet they add outcomes. - Prompt engineering interest rises. ICs show less enthusiasm than managers. AI adoption varies by tenure. - Risks include metric absolutism, throughput worship, and context loss. Tradeoffs appear around speed and quality. - Bottom line: pair lightweight DORA with context, outcomes, and learning programs. Avoid lines-of-code metrics. ## 📖 Main Content - The report shows teams still struggle to pick metrics. The panel affirms the pain. They blame context gaps and tool churn. [LeadDev 2025](https://leaddev.com/reports/engineering-team-performance-report-2025). - DORA remains useful for delivery health. It misses product outcomes and team well-being. Leaders add business KPIs and qualitative checks. [DORA overview](https://cloud.google.com/devops/state-of-devops). - SPACE complements DORA. It frames satisfaction, performance, activity, communication, and efficiency. Teams must choose local measures. [SPACE](https://www.microsoft.com/en-us/research/publication/the-space-of-developer-productivity). - Throughput alone misleads. Teams should watch incidents, reverts, and churn. They should narrate anomalies with short notes. - L&D jumps in priority. AI pressure and hiring constraints drive the shift. Upskilling beats replacement narratives. ### 🗒️ Notes #### ⏱️ [00:00:16](https://www.youtube.com/watch?v=L0ZTdw5flsw&t=16) Session framing - ### [H1] Team performance measurement remains difficult - Context: third annual report; complexity persists. #### ⏱️ [00:02:22](https://www.youtube.com/watch?v=L0ZTdw5flsw&t=142) Panelist backgrounds - ### [H2] Veteran leaders report varied team dynamics - Broad experience informs metric nuance. #### ⏱️ [00:06:23](https://www.youtube.com/watch?v=L0ZTdw5flsw&t=383) Metric selection - ### [H3] Many teams struggle to select metrics - The report highlights persistent selection pain. [LeadDev 2025](https://leaddev.com/reports/engineering-team-performance-report-2025). #### ⏱️ [00:08:25](https://www.youtube.com/watch?v=L0ZTdw5flsw&t=505) Frameworks and skills - ### [H4] Prompt-engineering interest outpaces IC enthusiasm - Managers rate AI skills higher than ICs. #### ⏱️ [00:12:52](https://www.youtube.com/watch?v=L0ZTdw5flsw&t=772) DORA sentiment - ### [H5] Perceived DORA effectiveness declines in 2025 - Maturity exposes gaps beyond delivery. #### ⏱️ [00:14:40](https://www.youtube.com/watch?v=L0ZTdw5flsw&t=880) Qualitative balance - ### [H6] Qualitative context complements delivery metrics - Context turns numbers into decisions. #### ⏱️ [00:18:38](https://www.youtube.com/watch?v=L0ZTdw5flsw&t=1118) Framework churn - ### [H7] Framework churn risks metric misuse - Leaders must protect team signals. #### ⏱️ [00:20:28](https://www.youtube.com/watch?v=L0ZTdw5flsw&t=1228) Blending methods - ### [H8] Leaders blend qualitative and quantitative inputs - Start with a problem; then pick measures. #### ⏱️ [00:24:11](https://www.youtube.com/watch?v=L0ZTdw5flsw&t=1451) Local fit - ### [H9] Teams need context-specific measurement approaches - No single framework fits every team. #### ⏱️ [00:25:47](https://www.youtube.com/watch?v=L0ZTdw5flsw&t=1547) Throughput critique - ### [H10] Throughput alone misses post-release quality - Track incidents, reverts, and churn. #### ⏱️ [00:29:39](https://www.youtube.com/watch?v=L0ZTdw5flsw&t=1779) Narratives - ### [H11] Leaders must add narrative to metrics - Require owner notes for anomalies. #### ⏱️ [00:31:32](https://www.youtube.com/watch?v=L0ZTdw5flsw&t=1892) L&D shift - ### [H12] Learning and development priority rises sharply - AI and hiring limits drive attention. [LeadDev 2025](https://leaddev.com/reports/engineering-team-performance-report-2025). #### ⏱️ [00:35:05](https://www.youtube.com/watch?v=L0ZTdw5flsw&t=2105) AI disruption - ### [H13] AI disruption forces rapid skill shifts - New skills appear; budgets tighten. #### ⏱️ [00:37:05](https://www.youtube.com/watch?v=L0ZTdw5flsw&t=2225) Hiring signal - ### [H14] Curiosity plus fundamentals beat tenure alone - AI-open seniors outperform skeptics. #### ⏱️ [00:40:45](https://www.youtube.com/watch?v=L0ZTdw5flsw&t=2445) AI usage - ### [H15] Engineers using AI gain efficiency, not replaced - Skills and reviews still matter. #### ⏱️ [00:42:40](https://www.youtube.com/watch?v=L0ZTdw5flsw&t=2560) Communication - ### [H16] Strong communication improves prompt outcomes - Clear prompts yield better code. #### ⏱️ [00:46:21](https://www.youtube.com/watch?v=L0ZTdw5flsw&t=2781) Peer adoption - ### [H17] Peer support reduces AI adoption fear - Bottom-up momentum beats mandates. ## 📚 References - LeadDev: Engineering Team Performance Report 2025. https://leaddev.com/reports/engineering-team-performance-report-2025 - Microsoft Research: The SPACE of Developer Productivity. https://www.microsoft.com/en-us/research/publication/the-space-of-developer-productivity - Google Cloud: DORA and the State of DevOps reports. https://cloud.google.com/devops/state-of-devops - Video: Measuring team performance isn’t getting any easier. https://www.youtube.com/watch?v=L0ZTdw5flsw ## 🗣️ Paraphrase - The report shows measuring engineering remains hard. DORA helps, but leaders add outcomes and context. Teams avoid lines-of-code metrics and prefer qualitative checks. - AI changes skill demands. Managers prioritize prompt skills more than ICs. The panel recommends L&D, peer support, and balanced dashboards. ## 📰 Newsletter - Team performance still resists easy gauges. Pair DORA with outcomes and SPACE-style health. Prioritize coaching, not counting lines. See the LeadDev x O’Reilly 2025 report for patterns and pitfalls: https://leaddev.com/reports/engineering-team-performance-report-2025. WordPress folks: apply this in Core triage, performance bounties, and Training curricula. The WPOCC can trial micro-grants tied to qualitative health and post-release quality. ## ✅ Action Items #### DevRel - Publish a short “metrics that matter” guide with examples. - Record a 10-minute demo: anomaly notes in dashboards. - Add workshop content: prompt patterns and review checklists. - Launch a quarterly survey for team context and blockers. - Capture funnel from demos to adoption with clear UTM plans. #### OSPO - Ship a lightweight contribution policy for metrics disclosure. - Define a license-compliance training loop with sign-off cadence. - Document an embargo and incident-response playbook with contacts. - Track SBOM freshness and ownership using an SLO per repo. - Offer AI consult hours; publish guardrails and approved tools. - Formalize upstream funding with MoUs and operator agreements. - Align OKRs with exec KPIs; report quarterly deltas. #### Community - Host roadmap clinics for maintainers; share templates. - Run “first issue” pairing rotations monthly. - Publish budget and milestone reports for sponsors. - Schedule cross-project incident drills twice a year. #### WordPress - Core: define release criteria tied to MTTR and incidents. - Core Performance: add profiling labs and perf bounties. - Core AI: set review rules, eval sets, and pilot tasks. - **Core Program Team:** implement weekly cross-team standups with notes. - Mobile: review telemetry; improve offline UX backlogs. - Playground: create browser labs for repro and lessons. - Community: update meetup kits; add speaker mentorship tracks. - Training: map learning paths; prototype certification pilots. - Marketing: ship a narrative on outcomes-over-outputs. - Support: tighten SLAs; document escalation flows. - Design: run quarterly UX audits; adopt tokens. - Documentation: fill DevHub gaps; add examples-first patterns. - Photos/TV: improve captions and moderation guidelines. - Polyglots: run translation sprints; align glossaries. - Accessibility: prioritize WCAG audits with heatmaps. - Meta: stabilize .org APIs; improve contributor tools. - CLI: expand command coverage and smoke tests. - Hosting: publish cross-host defaults and reliability checks. - Tide: refine scoring transparency and false-positive reviews. - Openverse: tune ingestion; clarify moderation policy. - Themes/Plugins: mentor reviewers; add provenance checks. #### The WPOCC - Launch fellowships on governance and release leadership. - Fund micro-grants for documentation and health sprints. - Publish a transparent ledger of awards and outcomes. ### FAIR - Add signature checks and provenance to local metadata. - Publish funding covenants with sunset clauses and anti-capture terms. - Ship reusable charters and scorecards as living artifacts. ## 🧪 Hypotheses - Blending DORA with two outcome KPIs improves predictability. - Monthly qualitative health checks reduce incident counts. - Peer-led AI hours improve prompt quality and review time. - Curiosity-focused hiring improves cycle time within two quarters. - Anomaly narratives reduce leadership misinterpretation incidents. ### 📊 Methodology - Run a six-month A/B across teams. Treat one group with “DORA + outcomes + narratives.” Compare cycle time, incidents, and revert rates. - Log peer-led AI hours. Sample prompt quality and review duration pre/post. - Track hiring cohorts. Compare changes in lead time to prod. ### ❓ Questions - Where does DORA fail without context? - Which qualitative inputs predict incidents best? - How do AI adoption curves differ by tenure? ### 🔮 Future Research - Build open benchmarks for narrative quality in dashboards. - Standardize a minimal SPACE questionnaire for engineering teams. - Study prompt clarity’s impact on defect density. ### 💡 Implications - Practice: present metrics with narratives and context owners. - Policy: avoid lines-of-code incentives. - Security: tie MTTR to incident rehearsal drills. - Education: fund AI literacy and prompt communication. ### 🧠 Learning Styles Assessment - Reading level estimate: Flesch-Kincaid Grade ≈ 11 (panel transcript). #### 👀 Visual learners - Present: slides and dashboards. - Improve: architecture and data-flow diagrams with captions. #### 🔊 Aural learners - Present: recorded panel audio. - Improve: a 3-minute audio summary. #### 🗣️ Verbal learners - Present: definitions of DORA and SPACE. - Improve: term comparisons and glossaries. #### 🧩 Kinesthetic learners - Present: none. - Improve: hands-on labs with fake incident drills. #### 🧠 Logical learners - Present: metric trees and KPIs. - Improve: causal maps and error budgets. #### 🧑‍🤝‍🧑 Social learners - Present: Q&A and Slack follow-ups. - Improve: cohort guides and peer reviews. #### 🧘‍♀️ Solitary learners - Present: report PDFs. - Improve: reflection prompts and self-checks. #### 🌿 Naturalistic learners - Present: limited alignment. - Improve: sustainability and lifecycle notes. ### 📊 Accessibility Assessment #### Screen reader semantics - Present: headings and captions. - Improve: consistent alt text and language attributes. #### Keyboard navigation - Present: basic focus order. - Improve: remove traps; add skip links. #### Contrast and readability - Present: adequate text contrast. - Improve: check large-text thresholds and line height. #### Text resizing and reflow - Present: responsive layouts. - Improve: support reduced motion; prevent clipping. #### Media and captions - Present: captions and transcript. - Improve: audio descriptions; downloadable transcript. #### Forms and interactions - Present: clear labels. - Improve: inline validation and forgiving inputs. #### Standards reference - Map gaps to WCAG 2.2 AA criteria. ### 🌍 DEIB Assessment #### Cultural breadth - Present: global applicability. - Improve: more non-US examples. #### Inclusive language - Present: neutral terms. - Improve: plain-language sidebars. #### Neurodiversity support - Present: consistent structure. - Improve: icon+text labels and predictable patterns. #### Representation - Present: varied roles on panel. - Improve: broader voices across regions. #### Psychological safety - Present: respectful framing. - Improve: clear escalation paths in communities. ### ✅ Logical Validity #### Strong reasoning - Causal link between context and decisions is solid [H6]. It aligns with SPACE. - Inductive claim that L&D rose due to AI pressure fits market signs [H12]. - Analogical critique of throughput vs quality is apt [H10]. #### Fallacies - Potential hasty generalization: “DORA effectiveness declines, so it fails” [H5]. Safer: “DORA helps, yet needs complements.” - Ambiguity risk: “AI replaces engineers” countered by clearer framing [H15]. Use: “Engineers using AI become faster.” #### Tone of voice - Pragmatic, candid, and evidence-seeking. Minimal hype; some caution on AI. ### 🛡️ DARVO Analysis - Deny: none apparent. - Attack: none apparent. - Reverse victim/offender: none observed. - The conversation stays constructive and professional. ## 🎯 Conclusion - Delivery health metrics still matter. Context and outcomes matter more. - Fund learning, pair AI-native and senior engineers, and narrate metrics. Then iterate. Share results with stakeholders. ## 🧱 Refactor Appendix (append after 🎯 Conclusion) ### 📚 Literature Split Pad - Purpose: Stage Konik CEE units that become per-source Literature notes. - Split level: **H3**. - Include verbatim quotes under 25 words and a [view highlight] link. ### [H1] Team performance measurement remains difficult > [!quote] [Measuring team performance isn’t getting any easier](https://www.youtube.com/watch?v=L0ZTdw5flsw) by [an unnamed author]] via [[YouTube]]. Published [MISSING]. Accessed . > “Measuring team performance never seems to get easier.” Two lines: use this to motivate a blended metrics approach. Cite LeadDev 2025 for context. [view highlight](https://www.youtube.com/watch?v=L0ZTdw5flsw&t=16) #xref #articleseed ID:rw_[MISSING] ### [H2] Many teams struggle to select metrics > [!quote] [Engineering Team Performance Report 2025](https://leaddev.com/reports/engineering-team-performance-report-2025) > “Selecting the right metrics remains a top pain point.” Use this to justify a minimal, outcomes-aware rubric. [view highlight](https://leaddev.com/reports/engineering-team-performance-report-2025) #xref ID:rw_[MISSING] --- ### 🔖 Atomic Split Pad (Zettel Candidates) - Purpose: Promote reusable claims into Atomic notes. - Split level: **H4**. - Each item: one idea, present tense, specific, testable. #### [Z1] Pair DORA with outcome KPIs - Works when teams align on two business goals. - Links: [[DORA]] [[SPACE]] #### [Z2] Require narratives for metric anomalies - Owners add short context notes per spike. - Links: [[Measurement]] [[Leadership]] #### [Z3] Track post-release quality, not just throughput - Use incidents, reverts, and churn as signals. - Links: [[Reliability]] [[Quality]] #### [Z4] Peer-led AI hours reduce adoption fear - Weekly sessions build skills and trust. - Links: [[AI-Practice]] [[L&D]] #### [Z5] Curiosity plus fundamentals outperforms tenure - Hiring signal for adaptive teams. - Links: [[Hiring]] [[Skill-Development]] --- ### ✏️ Fleeting Queue (Extract by Selection) - Purpose: Capture quick actions or sparks for Fleeting notes. - Extraction: **Select one line** → “Refactor: Extract selection to new note (content only).” - Keep each line short and self-contained. - [F1] Draft minimal “metrics + outcomes” rubric — /reports/engineering-team-performance-report-2025 #fleeting #inbox - [F2] Schedule peer-led AI hours — Mondays 11am #fleeting #inbox - [F3] Add anomaly-narrative requirement to dashboards #fleeting #inbox --- <!-- deep-read:sections end --> ## 📺 Video ![](https://www.youtube.com/watch?v=L0ZTdw5flsw) --- # 📄 Full Transcript *Auto-generated from YouTube captions · timestamps roughly every 30 seconds · [watch on YouTube](https://www.youtube.com/watch?v=L0ZTdw5flsw)* **[00:12]** Hello everyone and welcome to today's online panel discussion. We are going to be talking about how measuring team performance isn't getting any easier. And I am your host today, Scott Kerry. I am the editor-inchief here at Lead Dev and I'm joined by two brilliant panelists to talk about this topic and the report that we recently published with O'Reilly, the engineering team performance report 2025. It's actually our third edition of that report and I'm really excited to get into some of the subtleties and some of the details that we've seen kind of change or mostly not **[00:42]** change over those three years. Um, so Jezel and Sean, I'll get to introduce you both in a second. Uh, I can see some attendees still coming in. Um but while they are I'll do some housekeeping. Uh so um this panel discussion will last roughly 45 minutes after which both uh myself and the panelist will be heading over to the lead dev slack uh where we can answer any questions that are left over or any questions you might have after the call. Um that channel is the hash webinar Q&A channel. Um we also **[01:14]** should have time to answer questions live. Uh so there's a Q&A functionality in Zoom. uh type your questions in there and send them in and I will try and get to as many of them during the session as we can. Uh and so without further ado uh I'm going to introduce our panelists. So um Sean, let's start with you. How you doing today? >> Pretty good. How you doing? How's everyone else doing too? >> Yeah, tell us in the comments. Uh Jezel, how are you? How you doing? >> Doing good. Doing good. Bride and early **[01:46]** here in the West Coast. So >> nice. So that's um tell us a little bit about yourself, Jessel, and what you what you're up to at Kora AI. >> Yeah. Um so currently I'm the co-founder CTO of Kora. We're building uh agents for account management and uh post sales. So we build agents that manage customer enterprise customer workflows, launching, launch management, engagement, value reporting. Um and before uh Kora I was working in another **[02:17]** agentic uh AI startup that was in healthcare. Uh but overall been in the industry for 15 uh plus years in all sorts of capacity. I see engineering leadership managed very tiny teams and also very large ones. Uh and I've kind of kind of seen the the pain of uh kind of mentoring coaching engineers and also kind of upskilling myself as an engineer. So pretty excited to dive in into this report. There's some very interesting findings that I'm sure we'll get into. **[02:49]** >> Nice. And Sean, tell us a little bit about what you do at O'Reilly. >> Yeah. Um, so I'm an engineering director at O'Reilly and I manage a few teams. Um, and uh, some of the teams are the mobile team. We have O'Reilly that uh, develops a mobile app for our learning platform. Um, and I also manage some data teams and um, our AI team. Um, which has been fun. Um, as far as like my background, um, I've been in the industry for, you know, greater than 15 **[03:19]** years, almost 20 years or so. Uh, I just checked, uh, it'll be 20 years here in a year, which is kind of crazy to me. Um, and I, you know, similar to Jessel, I've done everything from, uh, you know, IC roles, um, working at large multi-billion dollar corporations, um, to working at very small teams with, uh, you know, one other dev. Um, so I've kind of seen, uh, the landscape of what teams are like and what, uh, kind of it means to be a leader in engineering. And **[03:52]** it's been something that's been a fun journey for me. Yeah, nice. I um I just came back from New York for Lead Dev New York and it was interesting because I always think that the average tenure in tech is short, but I was just shocked by how many conversations I had with folks who had worked at places 10, 15, 20 years. And so maybe we're self- selecting. Maybe there are people that come to lead dev that just like stay in good places. Uh maybe. But yeah, it's it's it's wild to me that um you see that kind of tenure still today. uh it's not the kind **[04:23]** of narrative that you hear. Um and so um to set a little bit of context before we get into the to the discussion here um this is our second year in a row publishing the engineering team performance report in association with O'Reilly. It's our third year doing the report as a whole. Um and what we really try and do with this report and have done from the start is trying to understand like how and why you measure engineering team performance. And this was the first ever report we did at Lead Dev. And the reason we focused on this **[04:54]** topic is because it just relentlessly came up when we were talking to our audience and to our members of our community in terms of it being something that is just hard. It is like really difficult and no one has like a tried and like true way of doing it repeat repeatedly like there are uh picking metrics is hard like working out what KPIs and what like what targets you're going to set. Uh how do you get people to rally behind those? How do you upskill efficiently to like keep that performance humming? Like there's just **[05:24]** so much to think about and um and I think that uh a lot of engineering managers spend a lot of time thinking about this topic. So we wanted to put some data behind that. Um I think the headline for this session today, measuring team performance never seems to get any easier basically reflects my like top takeaway when I was writing this report. Um it just like three years in now it seems to me like a lot of these things like don't move like it just if anything um there's a few uh kind of uh industry standard practices **[05:55]** that we started to see emerge and then they kind of dropped again this year. And so um in terms of and I know we're going to get into this because I think Sean and Jez have interesting takes on kind of frameworks and and kind of popular ways of measuring productivity and performance. Um, but yeah, I I my takeaway was that it doesn't seem to be getting any easier even though we spend so much time and effort thinking about this, which I think is always interesting. Um, in terms of like the kind of top takeaways, so just to to kind of give you some headline figures before we get **[06:27]** into the conversation. So 53% of uh respondents still can't choose the right metrics to measure team performance. This is stubbornly the like biggest pain point when it comes to measuring team performance is just choosing the metrics that you're going to that you're going to measure. Um 65% avoid measuring lines of code. This was like a big talking point a couple of years ago, especially when uh Elon Musk took over Twitter and made it one of their key KPIs. Um but it's still just like a an unpopular and probably deservingly so unpopular metric **[06:58]** for measuring team performance. And we can get into that a bit as well. The one that stood out to to me uh was 41% believed Dora is an effective measurement framework which was down from 56% in 2024. We're definitely going to talk about that. And then learning and development shot up in uh in terms of priorities from 70% last year to 83%. Um and so there's an interesting kind of uh focus on learning and development which maybe will take a few people by surprise. Um and so the other thing uh **[07:32]** that I've done today is included this report in the resources section. So for anyone on this call um you can go click on resources and there's some documents there and you can actually download the the full report and and check it out after this call. Um if you want to do that um we normally ask you for a bunch of details but there you go. You can just download it. So that's um reward for giving us your time. Cool. I've talked too much and really I want to hear from you Sean and Jezel about like what was uh what kind of jumped out at **[08:03]** you about this report. So, um, that's my first question. Sean, um, when you when you first sat down to read this report this year, what kind of jumped out? >> Um, to me, like the the decline at Dora, um, was something that that stood out, uh, to me. And I think there's a few reasons why it stood out. Um, but personally, it's something that, you know, I know that I've paid attention to and, um, you know, cared a lot about like in terms of measuring. I think that it was, um, I don't know if I want to say it was revolutionary. that might be **[08:34]** really a little bit too much. But um it was a really helpful framework um for thinking about um tracking teams and tracking performance and um to see a decline in that to me um was interesting. And I think it also speaks a little bit to kind of what I've uh encountered and what I've felt um you know when I've talked to other engineering leaders, when I talked to you know even just people at you know my company that I work at O'Reilly. Um I I think that it's definitely something that you know I see this um I I guess uh **[09:04]** realization that um you know maybe Dora doesn't solve everything for everyone. Uh so yeah that was the things for me that really stood out. >> Nice. What about you Jess? >> Yeah. Uh I think yeah Dora was definitely one. The other thing that stood out was how the prompt engineering as a skill uh and people and teams wanting to upskill in that area just shot up to number one straight up like [laughter] and that's just like I've never seen anything like that because all the other ones are like slightly have been and **[09:35]** flowing through the years. Um but but yeah, also addition to that, I think Scott, you you'd pointed this out was like the disparity between uh managers versus IC in wanting to scale in that area. And that is so fascinating. And I think that also like points to a little bit of a perception gap between IC's is kind of thinking hey like what value do I bring in this organization and that value is like tech hard technical skills um versus kind of management has shifted **[10:08]** in perception thinking that oh the the value may be uh like the broader context of the organization the customer context and thus like us needing to get better at like prompt engineering and that kind of stuff. So, yeah, very interesting to kind of uh unpack that and what that means. >> Yeah, I thought that was fascinating in the report. And just to kind of like um unpick it a little bit, we we basically asked folks like what area are you most interested in upskilling around? Um and **[10:38]** prompt engineering and managing agents shot to the top of the pile. Um and it was like clearly the most popular area. Um but when we just split out the engineers or the individual contributors um it dropped by about 20 percentage points and so they're like this is something that I see in the market a lot which is this gap between the kind of pragmatic realism of engineers who are like yes okay you know it's but it's another tool in the toolbox whereas um the further up the the ladder you seem to go the the more that kind of fervent **[11:11]** kind of excitement seems to build and and uh I I think a lot of engineering managers, especially middle managers, have found themselves stuck between that that rock and a hard place. Um Sean, was there anything you wanted to add on that before I jumped jumped ahead and got back to Laura? But yeah. >> Yeah. No, honestly, like I think it's related. Um I think as you see skills that even five years ago weren't in the lexicon of um anyone uh becoming **[11:42]** what I'll call uh like standard kit for engineers. Um, I think you see less like this is why I think Dora is maybe interested in Dors claiming is because it feels like you see more of an interest in like business outcomes and more an interest in seeing like what can be delivered versus like how it's being delivered because I think there's a lot of unknowns like it's I still think we're figuring out as you know a larger group how to really deliver effectively **[12:15]** with um AI tooling in a way that is um I'll call it uh maybe trackable. I don't know. Um there's I think a lot of different um skill sets that are required now that weren't required before. And to me, like I think that there's there's a relation between a decline in Dora and a an uptick in the use of AI and the the requirement of it to be something that is um every **[12:45]** engineer's using. So I I see like it being very related. >> Yeah. And I'll just add real quick um to that is like Dora essentially is tracking the artifact of delivery, right? like it's cracking code and code itself when you think about it is now getting more and more commoditized and AI is really getting good at generating code. So that's why I think there's that drop in like excitement for DORA and and kind of shifting that to measuring something else that's more qualitative **[13:17]** uh and measuring the engineering like context and communication and the broader landscape. >> Yeah, I think that that door narrative is interesting. It really caught my eye in the report because I mean I was I was hosting a dinner with engineering leaders in New York last week and the topic came up and it it kind of evokes quite strong feelings because like some people have really bought in and like and they've got it working for them and like there's like and that's why it's become popular is because I think it's quite easy to get started with Dora once you get it kind of humming like you're **[13:48]** going to you're going to be able to kind of track a lot of really important things. Um, but then I I get the impression that you hit a bit of a ceiling with it and then that that's maybe why this sentiment is coming down. Maybe as maturity increases, sentiment comes down. I I don't know. Sean, have you got any view on that hypothesis I've thrown at you? >> I don't know. So, yeah, I think Dora is really great like I don't know. I think in terms of like analogies, a lot of times metaphors >> and to me Dora is measuring like a car like it's measuring the performance of **[14:19]** the engine. Like how is that engine performing? like is it performing really well? Like is it like you know what's what's going on with it, right? Is it spewing out a lot of smoke like some cars that I probably owned in my teenage years or is it a little more of a high performance vehicle, right? Um and I think that it's great for tracking that, but if you are in a car and you're not going anywhere, it doesn't matter how great the engine performs. And I think that or if you're in a car and there's no seats for the people that are driving it, right? Um, and I think that what we're seeing is more, which I'm really **[14:51]** happy about actually, is more a trend towards the qualitative aspects, right? And not just the performance aspects in terms of like how how we're delivering. Um I think that there's a larger picture that Dora misses out on um that you it's really important as um an engineering leader uh to understand about your team and I think that having the we keep using the word context but having that **[15:22]** context I think is very important in order to understand like how the team performs and how the teams delivers beyond you know what is our meanantime to recovery or you know what's our you lead time or something like that for changes. Um, I think those are important metrics, right? I think those are very important to track and I think that it would be foolhardy to walk into a situation where you're not tracking something like that to understand, you know, at the end of the day what's being delivered. But it is only a slice of a larger picture. And I think maybe what we're maybe we're starting to see people realize that or come to that conclusion **[15:52]** more. >> Yeah. And we haven't mentioned space yet. And um you know that was a that was a real effort by the original authors of Dora to to to try and round out that picture to try and like you know bring in the quality as well as the kind of quantifiable metrics. Um Jez what's your view on because like in the report we've never really seen space kind of catch up to door in terms of popularity. you know, we ask a bunch of questions around kind of like is this something you use **[16:22]** and it's just never really kind of got out of that minority kind of position and and what do you think about why that might be, Jess? >> Yeah, I think the beauty of Dora is it's so clear and it's like you know exactly what you need to measure, how you need to measure it. space on the other end kind of expands and broadens the aperture and says, "Okay, uh we're not going to dictate exactly what you need to measure, but we'll just tell you like the dimensions that are important and then we'll give you the agency to decide **[16:55]** like how you measure those dimensions and how you actually track them." So I think that inherently creates a little bit of cognitive load and makes it so that it's not a standard framework across the organization. So it leaves a lot on the on the organization like how do we measure this? How do we come up uh and and like figure out that there's engineering team uh cohesion and happiness? What do what do we want to me measure underneath the hood of each of those dimensions? Um and I think that it **[17:27]** creates that uphill friction and adoption and even if you adopt it open question on like did you measure the right thing uh and then is it moving the needle uh with your actions year-over-year uh it's not so clear as as Dora is. >> Yeah. Sean, any anything to add on that? >> Yeah, it's super new too. Um I mean like I the fact that it's represented on here at all like I think I'm looking at the **[17:58]** report right now and it's represented as uh you know pretty substantial I think for being so new um you know I think what was it uh I forget when it was it was like 2021 or something like that. It was it's fairly >> ballpark about five years that checks that checks. >> Yeah, something like that. And I think that and I I mean I didn't hear about it until the last couple years I think. Um, and so I think that it's it's really interesting how quickly it has um come on the heels of Dora. Um, and you Dora **[18:29]** Dora itself isn't that old either. Like and I think going back to the you know the sub what was the subtitle of this measuring team performance isn't getting any easier. I think that that's a trend that like there's always a framework. There's always something that um you know will replace the next one and you'll see you know the thing that was um really important uh one day suddenly is not important the next um or it's you know abhorent you shouldn't use it um but I think **[18:59]** >> a lot of this the a lot of the tracking of metrics I think really we can't lose sight of like the true purpose of this which is to understand and to paint a picture of the landscape of the team and how the team's performing Right. And part of that like I think I don't know maybe this is controversial take. I think sometimes managers um and leaders and uh you know CTO's and whoever within engineering we tend to lean on metrics as a crutch and something that um we can just point to because it's it's quantitative and it's it's we're looking **[19:30]** for some some part hard data. We leave out like the people aspect of it, right? And we leave out I think sometimes our like job as managers which is to understand the context of what these metrics mean and to understand um you know what's going on behind the uh change in lead time right u maybe there's a good story there that's actually really important um and I think that that's what I think like not having um something and not measuring in some way is flying blind and you definitely don't want to do that but you also have **[20:02]** to really understand and bring um like the understanding that um you you you have as a human to these situations because numbers are just you know lines on a spreadsheet without that and um I think that that's we we have to make sure that we don't lose sight of that. >> Yeah, that kind of brings me like neatly on to my next question but then Kristen Belchure friend of the show has actually kind of asked my question for me so I'm going to I'm going to ask her a question instead. Um which is how do you balance **[20:32]** uh how do you view the balance of these metrics? So, how do you balance quantitative and qualitative within the context of your teams? Jess, I'll start with you. >> Yeah, that's it's a it's a tough uh question and I think as engineering leaders, you kind of have to have the pulse on how your team is doing and qualitatively you're you're just checking in with those teams like having one-on- ones uh kind of writing down like feedback. Um, and once you kind of **[21:02]** build a broader contextual understanding of how this team is humming, then you know like what's working, what's not working, what are the themes that are working and not working. And then you reverse engineer from there and say, okay, now I think we have some hard metrics that we should measure like we're maybe there's a lot of like customer fires. Then we're like, okay, is our do we not have the right alerting? Do we not have the our meanantime to recovery is not right? Then you kind of attach those metrics and that Sean to your point. It's like **[21:33]** just measuring metrics blindly is not going to tell you anything. It's really you kind of have to come in with like what problem are we trying to solve and then kind of backing into that. >> Yeah. What do you think, Sean? >> I got lost looking at all the Q&A questions. Which question? Which question was this one? >> This was Oh, so I've already move I've already moved it on, but it was Kristen's and it was Yeah. How do you **[22:04]** balance qualitative and quantitive in the context of your teams? >> Yes. Okay. Um, man, that's the question, right? Um, I think that there's you have to have some level of quantitative information to understand what's happening, right? Um, and I think I've seen a lot of different approaches to quantifying qualitative aspects of um of teams and uh how teams perform um and you know with varying degrees of success **[22:36]** um I know one of the things that um we've used at O'Reilly on some teams is uh the pedals framework which I don't know if you familiar with that um essentially you're you're you're essentially asking the engineers the IC's in the team you know how things are going and you're using that to track um kind of metrics over time uh which I really like because it tells a it tells a story and shows a picture of like how the team what the team feels um versus just like how the team is doing and I think um those oftentimes really go hand **[23:08]** inand um so I think you know definitely try to apply quantitative measurements to you know those qualitative aspects um I also think that uh and this might just be you know maybe a little too old school I don't know or too simple just talk you you have to communicate you have to communicate with your engineers is you have to communicate with people. Um I I have a standing policy of um I will make it awkward in our one-on- ones based on a medium article I read many many years ago by I think um one of the someone who worked at Facebook and I **[23:39]** can't remember who exactly now. Um but I try to make it awkward in my one-on- ones and try to ask questions that really give me an understanding of you know how the team's doing because these are all inputs. These are all things that you know allow us to understand as leaders and as managers um you know how things are actually going u because numbers can tell a story that aren't true and you have to rely to a certain degree on your instincts and your you know what you're hearing from people who are you know actually working in at I'll **[24:09]** say the code level to understand like what's actually happening and how to maybe move things forward. I think that's a really hard question to answer to them though. Um because you've also got very every team is also really different too and so I don't want I don't want to take that out. I think that's an out right but um I do think that that's a that is like the question >> also I'll just say that like if the authors of space are here they would say that hey like that is the way you do it. It's the qualitative and and quantitative like the nice balance of **[24:40]** each. Uh but then it's again of course up to every Oregon leader to say like okay what what do I measure in that in that bucket but but at least that that's like a overarching foundation is space I believe >> I yeah I like I think I think you're spot on Sean like like there just isn't a shortcut like you've got to talk to people like you're with the with the dorometrics you're able to maybe get a bit of a diagnosis but you're not able to actually like cure anything if you don't talk to your engineers uh and and **[25:10]** kind like try and try and get to that. And so I think like in this era of AI, like a lot of people are looking for shortcuts all the time. And I've read a few posts of people being like, I don't need to do oneto ones anymore because I can just like get, you know, I can just do it all through like transcripts of meetings. I'm not sat in. And I just think that that is >> I'm not going to swear. Uh but it's uh I just don't think there are shortcuts. Yeah. Yeah. It's crazy. >> And if you're enjoying this, Lee Dev has a ton more to offer. [music] >> I'm quite fond of the articles. I feel like lead dev blogs are my favorite and **[25:44]** it's been really really useful to kind of create these more meaningful connections as well. >> If there's something you're struggling with as an engineering leader, there's a pretty good chance that lead has published something that addresses it. >> So make sure you check out leadv.com. Um I want to move us on to um talk a little bit about some of the stuff that we found in there about learning and development because I thought it was really interesting. But while while we are on metrics there's a couple of um kind of comments and questions from the **[26:15]** audience here. Uh just talking about the idea of throughput and the importance of throughput. Um and so there's one from uh Anna here which is like how how do you measure engineering teams throughput and then there's another comment here which is basically like Dora measures health but throughput is what the business really cares about. I don't know do either of you have a kind of strong view on throughput as a metric. I know it's not something we've discussed before. >> Mhm. Yeah. I mean I think one uh uh thought that comes to mind is now I've **[26:47]** seen a lot of engineering teams that experiment with uh cloud uh agents right like they you can just have a agent sitting in linear and the ticket comes in the agent just starts and creates a PR um so in that world I just kind of question if throughput is really going to be kind of measuring what's important I think more about what happens after the throughput. It's like what what what happens after it goes to production? Are we shipping really good quality code? Uh **[27:19]** maybe in meantime to recover potentially not quite but like something like that is like okay are we maybe churn might be a metric like are we are you shipping something and then kind of reverting it. Those are the things that I think become more important rather than purely looking at throughput. Yeah, I think number of incidents is a really popular metric as well. >> Yeah, that's >> yeah, for us um the lead time uh to change is actually being **[27:51]** in production is like is our metric that we're using to measure throughput, right? uh because it doesn't matter if um you've got you know a large amount of deployments uh that's happening very frequently or some of the other like that's the one that measures in a lot of ways um how much capacity I think that you can handle and that's a metric from door right how much capacity you can handle in some ways because you can take that and we there's some calculations that we feel **[28:23]** like work pretty well with regard to the number of engineers and previous velocity And um we use that for historical information to essentially get a percentage of confidence that we'll be able to ship something on time. Um and and that that helps understand when we have projects that we're um estimating out um you know if we actually what level of confidence we think that we'll be able to deliver these in the time frame that we say which I think goes directly to throughput like do we need to add more humans onto this um you know is that going to actually uh impact this in a **[28:55]** way that's material or is it not um that's I think for me uh as someone who's managed team it's been really effective to understand what capacity my team has um >> yeah [clears throat] and I want to I want to move us on from metrics but I can't because there's a really good question here so um sometimes you have the context and the full picture but then your chain of command doesn't and takes numbers out of context how do you manage that >> Jezu any top this this has happened all **[29:26]** the time classic [laughter] question uh uh yeah I think I one of my things was like if we had a report I would never send it out to my leadership without context without me kind of adding an overarching narrative of like why are you seeing a drop in cycle time? Why why is it drop there's a drop in X? Uh I think it needs to you kind of have to tell the story behind the metrics to for it to make sense. And in one of my **[29:57]** previous companies we had a ritual that anytime there was anomaly it has to come with commentary. uh otherwise there's just no way to interpret that. Um so and and then I think all the second thing that's more important is I think management and your leadership chain has to be aligned with that thinking that metrics don't alone tell the story. There's there's a more uh behind that. So they all also have to be like seeking that like what is the more contextual narrative behind these metrics. if they are if they don't believe that then I **[30:28]** think there's a problem then then it's like okay cycle time dropped there's something terribly wrong with the team we need to fix it go fix it so >> I think uh to your point which you just suggested that's something that I think is really important uh if there's something that is uh an outlier controversial or like whatever make sure it has context don't let the numbers speak for themselves um we have a standing kind of you know unofficial policy I'll say I'll I'll say of um **[31:01]** presenting um over uh you know sending uh messages or send sending sending content or sending numbers or something like that. >> And I've probably made more slides um slideshows in my time at O'Reilly uh than any other time. And it's not necessarily me sitting in front of a room presenting um but it's really just using that format of a slideshow to provide more context um to provide more information than just sending a spreadsheet would provide. The reason being is everyone's going to interpret numbers you know differently. If I send **[31:32]** you if I say my team is 25% effective deliver new features you might hear that you might but that's horrible right? But what if they were just last you know two sprints less three sprints ago 5% effective right? like that's actually an incredible story and so I think you need to provide whatever that information is you have to you have to manage that context yourself in some in some ways you have to provide that commentary and have to provide some level of storytelling around it like we as engineering leaders are storytellers and um you know ideally positive **[32:03]** storytellers for a team but sometimes you know ne the story is not a good story sometimes um there has to be some kind of story there versus just uh you know delivering numbers and walking Perfect. All right, I'm going to switch up and start to uh focus on the the kind of learning and development section because I think there's some really interesting stuff in there. And so the one the number that just really jumped out at me, we we asked folks like how much of a priority is learning and development and like frustratingly like it's it's been low for a while like it's **[32:35]** been kind of in the 50s or 60s and then this year it jumped up to 83%. Um, and I thought that was that was really fascinating because I think that when things get chaotic or busy, learning and development tends to drop down the priority list. But there's something happening right now which has put it to the top. And I've got a couple of theories like I think there is as with everything right now an AI element of this. There's a new thing to learn and develop. And so I think there's a focus there on on trying to to kind of catch **[33:06]** up there. Like I had an interesting conversation with someone from OpenAI a few months back where they were saying like even their like biggest super user customers think they're behind. So like everyone thinks they're behind and so that might be driving some of this kind of new focus on learning development. But also like the hiring market is rough like people aren't able to really hire in talent like they were and kind of um fill those capability gaps. So maybe you have to focus on the development of your team. Those are just my theories, but Sean, this is very much your area, so **[33:37]** I'm interested in your take. >> Yeah. You know, like I I'll give my little pitch here, Riley. You know, we have a a you we're known for our our our content, our books and videos, and but we have a learning platform that um really focuses on upskilling uh you know, engineers, upskilling um business professionals with um a huge variety of uh you know, content and skills and topics. It's not just like learning Pearl or learning Python or something like that. It's all kinds of different things. Um I think what like in my **[34:09]** opinion the job market is this is not opinion. The job market is terrible. It's it's horrible if you're looking for a job right now. Um and budgets are tighter um pretty much all around. I would say from everyone I've talked to I haven't heard someone say, you know, it's great time to be uh managing a budget. uh it's a really tough time and um what that means is um I think there's more investments in terms of individual engineers and there's more investments in terms of **[34:40]** employees in in two ways. Um, one is you can't hire a new skill set uh nowadays. Like you can't hire it away and just go on a hiring spree. And so you have to invest more in the individual employees that you've got underneath you in order to get the skills you need to actually deliver things. And those things you're delivering though, like there's a there's a really huge um innovation disruption cycle that's happening right now. I I feel like um with AI and there is um a huge amount of new skills that **[35:12]** you know I said before five years ago these weren't in Alexa and now like they're required and this is this really weird environment I think where you've got um this investment in new employees and there's new skill sets and you're not really doing a lot of hiring and those skills are really new too and so I think that that's really why what's driving that's my opinion at least what's driving this uptick in learning and development um there's just you know not as big a budget as before and you still have to get the job done um with these new **[35:42]** frameworks, new tools um and I think that that is certainly driving a lot of this >> and Jessel you are hiring so it kind of provides different challenges for you. >> Yeah. No, Sean, you nailed it. Uh I think uh the the talent market right now is very difficult. And the other thing I'll add is uh I've been hiring the whole year and been interviewing engineers and it's it's been very interesting to see that um younger **[36:14]** engineers, shorter tenure engineers, they are very AI native uh and uh they they they they do really well in kind of coding interviews where you kind of have them bring in all your IDE AI ids and everything. But uh on the other side of the spectrum like senior more tenure engineers tend to struggle with with the new tools and they tend to kind of uh have their head buried in the sand a little bit versus like okay I think I I I know I'm a very tenured engineer. I **[36:45]** know how to ship code but they're not necessarily leveraging all the latest and greatest tech. So that makes the them inherently really so slow at shipping code code generation, right? So that that's I think kind of like what's driving the pressure is like when you have so many engineers that are tenure experience but also kind of have a perspective on like how they want to build software and how they want to shift software but now you're having like a whole paradigm shift in like how we actually build and think about **[37:16]** delivering software. Um, so that's I think like making the engineering leadership like this needs to be the biggest area we need to focus on. How do we upskill our existing team? Um, and then I'll just I'll just add one real quick thing is like when it comes to hiring, like one thing that I've kind of noticed is engineers who are not necessarily AI native, but they're open and willing to learn um in in the new tech stack, they have tend to do really **[37:46]** well. uh they've been like the best performers even though they're not as tenured um because they're like hey I want to I want to embrace the new tech uh but also have good engineering fundamentals and those two combinations m makes for a really great high performing individual in a team so I think that's leaders are seeing that and saying hey like can we do that can we ignite that like passion for learning in our teams >> yeah that feels like the kind of profile that I would be looking at regardless of what the hot technology is, you know, **[38:16]** you're looking for that kind of curiosity, that openness, but also, yeah, they've got the fundamentals. That's the kind of sweet spot. >> Um, Jessel, you wrote a piece for us a while back about kind of the this the most important skills in the AI era. Um, in our survey, we kind of picked out prompt engineering and um, managing agents. Um but the as with this anything in this space it moves so fast and so like even when we were doing our prep call for this I was like I wish we had included building MCP servers in there **[38:49]** because that is you know a really key skill now in in this era but there's a there's a thousand things that we didn't include. Um I'm just interested in because you've been doing this hiring you're obviously thinking about this. What are the skills outside of the kind of like general profile that you just mentioned that you are looking out for? Yeah. Uh I I think of the the way I think about it is uh backing into like what is AI really good at versus what is AI really bad at and AI is good at code generation but it's really bad at **[39:20]** context and and so that already kind of gives you like where is where's the gap like who do we need on the team and the kind of people we need on the team are really good at sucking in the context of the organization bringing in customer empathy and then translating that into a plan of action where so there a lot of the investment goes up up upf funnel where it's like okay uh communication skills collaboration skills world model building skills context gathering skills **[39:52]** and then you translate all of that into your favorite uh AI tooling and the execution is somewhat the the last mile if your thinking prior thinking is not clean and crisp the execution is not going to be great. So, it's like garbage in garbage out kind of situation. So, that's what that's what we prioritize is like, yeah, if you're lagging in some of the hard skills of like how to build MCP servers, that's not a problem at all. Like, I I we just need you to be curious, leaned in, great communication, **[40:24]** and then I think that rest everything will will take care of itself. >> Yeah. Nice. Sean, you get such unique insight here in terms of what people are looking at, what people are trying to learn about, like where where do you see the kind of key skills here? >> I think that there's um I think Jezel covered a lot of what I would cover, right? I I think that um you know, learning and curiosity still remain to me like a primary thing to look for whenever you're looking at a new hire. Um the landscape is moving really fast **[40:55]** and that seems really cliche to say, but it does feel like things have picked up very much, right? And I think that's because, like I said before, I think we're in one of those disruption cycles. Um, and that's why I think focusing very hard on a specific aspect of AI, you know, driven development is is kind of tricky because um that will it has the potential to become not, you know, as important in just a year, less than a year sometimes. Um, so I'm always **[41:28]** looking for people that are curious and that want to learn. And I also think that, um, there's a big difference between people that, you know, have I will say embraced AI in the sense that it's part of their workflow. Um, I think that there's a real uh, fear um, in the community and in the industry around AI at an individual contributor level. And I think that that speaks to what we mentioned at the beginning of this call, which was that discrepancy between manager and developer in some ways. Um I **[41:59]** do think that um I don't think AI is going to replace engineers. Um I think that engineers who use AI are going to be faster and more proficient um engineers as well as long as they know how to use it well, right? And that goes back to that learning and curiosity. Um, I'll also say that I think um, it's interesting working at a company like O'Reilly that has such a focus on content and like we have a whole editorial department, right? And so I've got access to people that, you know, are great writers and that can produce **[42:30]** content, super smart, super technical, very knowledgeable people. And the people who are really good at communicating, who are really good at writing, who are really good at speaking can translate that really well into prompt engineering. Um, versus it feels like people who aren't as good at communicating have a harder time with that. And to Jesse's point about, you know, garbage in, garbage out, like I really think communication is one of those skills that super matters for um any kind of prompt driven development because you have to be able to **[43:01]** communicate effectively in order to um get the results you want. >> Yeah. Nice. And then um my last question is like how do you how do you like that's obviously what you're hiring for right now, but how do you turn around maybe slightly more stubborn developers who are already in your team who maybe aren't so much along for the ride? Sean, obviously you said that there there is some fear I think underlying it for some folks like that it's coming for their jobs or that they don't really want to pick up these tools. They like, you know, they've used **[43:31]** them a couple of times and it spits out garbage and then they have to clean up the garbage and it's like I don't really want to do this. But it feels like a runaway train. So Jezu, is this something you've come across? Is it something you've been able to kind of manage through? How do you kind of overcome that maybe a little bit more stubborn skepticism? >> Yeah. Yeah. And I think uh engineers tend to be leftrain. So I we we need to lead with the evidence like is the tool that you're using is it going to really help you? And I think that's kind of **[44:02]** where your hires that are very leaned in of that very curious, want to learn, get better, you kind of have to leverage those team members, the advocates, uh, to say show them that, hey, like yes, this is AI is like a a Ferrari and a racetrack. If you kind of are not an experienced Formula 1 driver, you're you're not you're going to crash right away in that car. Um, so you kind of have to have built some like techniques um of how to steer the AI correctly and then you kind of have to demonstrate **[44:33]** that back to the team that hey uh if you just go in and type a simple prompt it's probably not going to yield the right outcome but if you prompt it in a certain way and here's some of the techniques to do it uh then it's really going to accelerate your velocity. Um, so yeah, this just comes down to like having some like brown bag sessions, workshops, showing the evidence to bring the skepticism around. >> What about you, Sean? >> Yeah, I think you're right on the money there. Like I what you're describing um **[45:06]** is, you know, in a lot of ways bottom up versus top down, right? Um I think having something that like AI forcing that top down um is not as effective as creating um momentum um amongst fellow engineers and you know making it be something that's more of a grassroots kind of operation. Um not that's not to say that I think it is important for sometimes leaders to step in and say like set the tone and set the **[45:36]** standard u that this is something that's important for you to learn and um I'll also say create the space um and bring that human aspect to it to it because I we um we had a really great uh moment with our teams um where we an engineer shared that they were very uncomfortable with AI openly in Slack channel and um that led to um this individual finding people to work with and finding people to actually talk and do this with. Um **[46:06]** and it was to me it was a really powerful moment because you see like it's always great whenever people can be transparent and open um in you know a space where you know it's not natural to be vulnerable in. And um so that was really encouraging, but then also to see this individual really like start to learn and start to kind of um understand and not fear and more be curious about AI um was really interesting to see that too because I saw how effective that bottomup kind of approach can be. It wasn't um this person's manager coming to them and saying you have to learn AI, **[46:36]** you have to use AI. Um even though like I said I think sometimes that's needed. Um it was people around them like the peers um being able to walk through and we you know we started doing EIT times um and having places to share and creating that space I think has been it's been really effective and really helpful. >> Perfect. Well, sadly that's all we got time for. Um but thank you so much Sean. Thank you so much Jess. Thank you so much to everyone who tuned in live. Um, **[47:09]** there's still loads going on in the Q&A, but I'm trying to pass what is more of a comment and what is more of a question. Um, but as I said, we're going to go over to the um, lead dev Slack and there's a webinar Q&A channel there. If there's anything you want to [music] ask us that we didn't get to, go and post it over there and we'll we'll try and get to it. Um, before you get out of here, there is the whole report available to you in the resources section. Grab that before you go. Um, but I just want to thank O'Reilly for putting this session on, for supporting this report, and **[47:39]** thanks again to to Sean and Jezel, and see you both soon. >> Thank you. This was super fun.