Make AI productivity gains stick  🔗 **Source:** [Make AI productivity gains stick](https://www.youtube.com/watch?v=-6XbGcLEm7U) > [!note]+ Document Notes > tags: [ai-productivity, developer-workflows, change-management] > author: [MISSING] > source: Lead Dev > reference: https://www.youtube.com/watch?v=-6XbGcLEm7U > date created: {{date:YYYY-MM-DD}} {{time:HH:mm}} > > # 🏷️ Title: Make AI productivity gains stick > <!-- deep-read:sections start --> ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ## 📝 Summary - **What happened**: Lead Dev panel featuring AB (CTO at Metro Fox), Anastasia (MVP Product Engineering at GitKraken), and Maxim Najim (Distinguished Engineer at Target) discussing how to sustain AI productivity gains in development teams. - **Why it matters**: AI tools provide initial productivity boosts, but making those gains permanent requires deliberate change management and systematic measurement approaches. - **Key stakeholders**: Engineering teams, CTOs, dev leaders, individual developers at all levels. - **Risks and tradeoffs**: Over-reliance on AI can cause skill atrophy; quality may suffer if speed is emphasized without proper verification; costs will increase as AI subsidies end. - **Bottom line**: Sustainable AI gains require treating adoption as major organizational change management, not just tool rollout. ## 📖 Main Content The panel explored three key types of AI gains: behavioral change (adoption patterns), system improvements (DORA metrics), and customer impact (business outcomes). Success requires moving from "vibe coding" to "agentic engineering" with disciplined workflows. **Core gains identified:** - **Test coverage improvements**: AI translates specs into comprehensive test cases - **Delivery speed**: Features that took 3-4 days now complete in under one day - **Context switching efficiency**: Staff+ engineers can manage 6-7 parallel agent sessions - **Ramp-up acceleration**: AI agents help explore unfamiliar systems and domains **Distribution patterns**: Gains are strongest among senior managers and junior developers. Mid-career engineers often resist AI, feeling it threatens their coding-based identity. Staff+ engineers benefit most because they already know how to delegate and set clear goals. **Measurement framework**: Three-pillar approach covering adoption metrics, system changes (velocity, quality), and business impact. Teams must follow work end-to-end rather than measuring stages in isolation to avoid shifting bottlenecks elsewhere. ### 🗒️ Notes #### ⏱️ [00:03:12](https://www.youtube.com/watch?v=-6XbGcLEm7U&t=192) AB on productivity gains - Increased test coverage: developers creating more comprehensive test cases with AI assistance - Higher delivery speed: architectural design phase remains human-driven, implementation accelerated by AI - One developer can work on three parallel workstreams simultaneously #### ⏱️ [00:06:45](https://www.youtube.com/watch?v=-6XbGcLEm7U&t=405) Anastasia's three-pillar measurement approach - Behavioral change metrics: adoption, engagement, frequency of use, autonomous agent operation - System change metrics: DORA and SPACE frameworks, developer satisfaction tracking - Customer impact metrics: experiment velocity, shipping frequency, customer satisfaction #### ⏱️ [00:09:30](https://www.youtube.com/watch?v=-6XbGcLEm7U&t=570) Maxim on staff+ engineer role transformation - AI raises ceiling on engineering capabilities but fundamentals become more important - Staff+ role shifts toward shaping intent, designing verification loops, creating alignment - Six to seven coding agent sessions running simultaneously for different projects - Context preservation enables involvement in broader project surface area #### ⏱️ [00:15:20](https://www.youtube.com/watch?v=-6XbGcLEm7U&t=920) Uneven distribution of gains - Biggest gains: senior managers and junior developers - Mid-career engineers resist due to identity threat around coding skills - Solution: train mid-career engineers in delegation and management skills - Hiring focus shifting to "product engineers" who understand business context #### ⏱️ [00:18:45](https://www.youtube.com/watch?v=-6XbGcLEm7U&t=1125) Measurement challenges and solutions - Follow work end-to-end, not just stages - Track incident rates alongside delivery speed - Use cohort analysis to compare AI-adopters vs. non-adopters - Layered measurement prevents optimization of wrong metrics #### ⏱️ [00:22:10](https://www.youtube.com/watch?v=-6XbGcLEm7U&t=1330) Pitfalls and fragility points - Skill atrophy: developers stop exercising engineering judgment - Over-reliance: copying stack traces to AI without basic debugging - Context limitations: AI struggles with large codebases requiring domain knowledge - Cost sustainability: current token subsidies will end, requiring efficiency optimization #### ⏱️ [00:28:15](https://www.youtube.com/watch?v=-6XbGcLEm7U&t=1695) Making gains stick - systematic approach - Treat as change management, not tool adoption - Standardize agent instructions across teams through shared markdown files - Create learning feedback loops from mistakes back to agents and teams - Monitor engineering organization daily/weekly for quality degradation signals #### ⏱️ [00:32:40](https://www.youtube.com/watch?v=-6XbGcLEm7U&t=1960) Skills for AI-augmented engineering - Understanding over outsourced thinking - First principles reasoning and architectural thinking - AI literacy and supervision skills - Domain knowledge and system thinking remain critical #### ⏱️ [00:38:25](https://www.youtube.com/watch?v=-6XbGcLEm7U&t=2305) Preventing skill atrophy - Form opinions before consulting AI - Practice deliberate comparison between human and AI solutions - Focus on problem-solving rather than code generation - Maintain curiosity and continuous learning mindset ## 📚 References - [Lead Dev Slack Community](https://theleaddeveloper.slack.com) - GitKraken Developer Tools - DORA Metrics Framework - SPACE Metrics Framework ## 🗣️ Paraphrase Three engineering leaders shared their experiences making AI productivity gains sustainable in development organizations. The key insight is that initial AI adoption often focuses on speed improvements, but lasting benefits require systematic change management treating this as cultural transformation rather than tool rollout. The panelists identified uneven distribution of AI benefits, with senior engineers and juniors gaining most while mid-career developers often resist due to identity concerns around their coding expertise. The solution involves retraining these engineers in delegation and product thinking skills. Measurement requires a three-pillar approach covering behavioral adoption, system improvements, and customer impact. Teams must track work end-to-end rather than optimizing individual stages to avoid simply shifting bottlenecks elsewhere. ## 📰 Newsletter **AI Productivity Gains: Making Them Stick for Your WordPress Team** Great discussion from Lead Dev on sustaining AI productivity improvements! Key takeaways for WordPress developers: 🎯 **The Reality Check**: Initial AI speed boosts often fade without systematic change management. Treat AI adoption as cultural transformation, not just new tooling. ⚖️ **Uneven Benefits**: Senior devs and juniors see biggest gains. Mid-career developers often struggle with identity concerns as coding becomes commoditized. Solution: upskill them in product thinking and delegation. 📊 **Measurement Matters**: Track adoption → system improvements → customer impact. Don't just count commits—follow features end-to-end to avoid shifting bottlenecks to code review or debugging. 🛠️ **WordPress-Specific Applications**: - AI-assisted test case generation for plugins - Accelerated theme development with human architectural oversight - Documentation and RFC drafting for Core contributions - Faster onboarding to complex WordPress codebases 💡 **Pro Tips**: Standardize AI instructions across your team, surface AI errors publicly for learning, and maintain engineering fundamentals even as AI capabilities grow. The future belongs to "product engineers" who combine AI fluency with deep domain knowledge. Start building those muscles now! ## ✅ Action Items #### DevRel - Developer experience: Create standardized AI instruction templates and context engineering guides for WordPress development workflows - Education: Design workshops on "agentic engineering" practices for WordPress teams transitioning from "vibe coding" - Community management: Establish forums for sharing AI productivity patterns and failure stories - Developer marketing: Create content showcasing sustainable AI integration in WordPress development #### WordPress - Core: Develop architectural decision record (ADR) templates optimized for AI agent consumption - Documentation: Create AI-friendly documentation patterns that improve both human and agent understanding - Training: Build curriculum on AI-augmented WordPress development focusing on maintaining engineering fundamentals - Community: Facilitate discussions on ethical AI use and skill development in WordPress ecosystem #### The WPCC - Programs: Launch fellowship tracks focused on AI literacy and sustainable productivity practices - Projects: Fund development of open-source tools for measuring and optimizing AI productivity in WordPress environments ## 🧪 Hypotheses 1. **Distribution Hypothesis**: AI productivity gains will remain unevenly distributed, with senior engineers and juniors benefiting most while mid-career developers struggle with identity adaptation. 2. **Measurement Hypothesis**: Teams tracking end-to-end workflow metrics will achieve more sustainable AI gains than those optimizing individual development stages. 3. **Cost Sustainability Hypothesis**: Current AI productivity models will face significant challenges when token subsidies end, requiring efficiency optimization strategies. 4. **Skill Evolution Hypothesis**: Engineering roles will increasingly emphasize product thinking, system understanding, and AI supervision over direct code generation. 5. **Quality Trade-off Hypothesis**: Teams emphasizing speed metrics without corresponding quality measures will experience technical debt accumulation that undermines long-term productivity gains. 6. **Cultural Change Hypothesis**: Organizations treating AI adoption as change management rather than tool rollout will achieve more durable productivity improvements. ### 📊 Methodology - **Longitudinal cohort studies**: Compare AI-adopting vs. non-adopting developer groups over 12-month periods - **End-to-end workflow analysis**: Measure feature delivery from conception through post-deployment monitoring - **Skills assessment surveys**: Track changes in engineering capability self-reporting and peer evaluation - **Economic impact analysis**: Calculate total cost of ownership including AI tooling costs, training, and productivity gains - **Ethnographic studies**: Observe daily workflows and cultural adaptation patterns in development teams ### ❓ Questions - How can mid-career engineers be most effectively retrained for AI-augmented roles without losing core technical expertise? - What specific metrics best predict long-term sustainability of AI productivity gains? - How will AI cost structures evolve and what optimization strategies will prove most effective? - What governance models work best for standardizing AI agent behavior across large development organizations? ### 🔮 Future Research - Development of AI productivity sustainability frameworks for different organizational sizes and contexts - Investigation of cognitive load patterns in AI-augmented development workflows - Analysis of skill evolution trajectories for developers at different career stages - Economic modeling of AI productivity ROI across various cost scenarios ### 💡 Implications **Practice**: Development teams need structured change management approaches rather than ad-hoc AI tool adoption. Engineering education must emphasize product thinking alongside technical skills. **Policy**: Organizations require governance frameworks for AI agent standardization and quality assurance in code generation workflows. **Security**: Increased reliance on AI-generated code necessitates enhanced verification and testing protocols to prevent subtle vulnerabilities. **Education**: Computer science curricula should integrate AI supervision and prompt engineering alongside traditional programming concepts. **Funding**: Investment in AI tooling must account for total cost of ownership including training, governance, and eventual cost increases as subsidies end. ### 🧠 Learning Styles Assessment **Reading level estimate**: Flesch-Kincaid Grade Level ~12 (technical content with specialized terminology) #### 👀 Visual learners - **Present**: Transcript format with timestamps, structured hierarchical information - **Improve**: Add workflow diagrams showing AI integration points, measurement framework visualizations, before/after productivity charts #### 🔊 Aural learners - **Present**: Original video discussion format with multiple speaker perspectives - **Improve**: Provide chapter markers for key topics, create audio summaries of main takeaways #### 🗣️ Verbal learners - **Present**: Rich discussion format with definitions and examples, analogies like "muscle cars" for AI capabilities - **Improve**: Add glossary of technical terms, provide more real-world metaphors for abstract concepts #### 🧩 Kinesthetic learners - **Present**: Actionable measurement frameworks and implementation steps - **Improve**: Create hands-on workshops for implementing measurement systems, provide checklists for AI adoption phases #### 🧠 Logical learners - **Present**: Systematic three-pillar measurement approach, clear cause-and-effect relationships - **Improve**: Add decision trees for AI adoption strategies, provide more quantitative data on productivity gains #### 🧑‍🤝‍🧑 Social learners - **Present**: Panel discussion format with peer experiences, community slack integration - **Improve**: Facilitate peer learning groups, create mentorship programs for AI adoption #### 🧘‍♀️ Solitary learners - **Present**: Comprehensive written notes for individual study - **Improve**: Add reflection questions for personal AI adoption assessment, self-paced learning modules #### 🌿 Naturalistic learners - **Present**: Ecosystem thinking about AI integration and sustainable practices - **Improve**: Emphasize sustainability concerns about AI costs, connect to broader technology adoption lifecycle patterns ### 📊 Accessibility Assessment #### Screen reader semantics - **Present**: Well-structured headings hierarchy, descriptive section labels - **Improve**: Add alt text for any embedded media, ensure timestamp links have descriptive text #### Keyboard navigation - **Present**: Standard text navigation, clear section structure - **Improve**: Ensure all interactive elements are keyboard accessible, add skip links for long sections #### Contrast and readability - **Present**: Standard text formatting with clear hierarchy - **Improve**: Test against WCAG 2.2 AA standards, ensure sufficient contrast for all text elements #### Text resizing and reflow - **Present**: Text-based content that scales well - **Improve**: Test at 200% zoom, ensure no content gets cut off #### Media and captions - **Present**: Full transcript provided for video content - **Improve**: Sync transcript with video timestamps, provide downloadable transcript option #### Forms and interactions - **Present**: Minimal interactive elements - **Improve**: Ensure any future interactive elements follow accessibility guidelines #### Standards reference - Content generally aligns with WCAG 2.2 AA for text-based materials ### 🌍 DEIB Assessment #### Cultural breadth - **Present**: Global technology industry perspectives, enterprise-scale insights - **Improve**: Include perspectives from smaller organizations, non-Western development contexts #### Inclusive language - **Present**: Professional technical terminology, gender-neutral role references - **Improve**: Define more technical terms, avoid assumptions about organizational structure #### Neurodiversity support - **Present**: Clear structure with consistent formatting, logical information hierarchy - **Improve**: Provide content warnings for discussion of job displacement concerns, offer multiple information processing formats #### Representation - **Present**: Mixed gender panel, different organizational roles and perspectives - **Improve**: Include perspectives from different geographic regions, various company sizes beyond enterprise #### Psychological safety - **Present**: Acknowledges real concerns about job displacement and skill obsolescence - **Improve**: Provide more specific guidance for managing career anxiety, clearer pathways for skill development ### ✅ Logical Validity #### Strong reasoning - **Causal analysis (00:18:45)**: Clear connection between measurement approach and sustainable gains through end-to-end workflow tracking - **Inductive reasoning (00:15:20)**: Pattern recognition about uneven AI benefit distribution supported by multiple organizational examples - **Systems thinking (00:28:15)**: Understanding AI adoption as change management rather than tool deployment shows strong architectural reasoning #### Fallacies - **Hasty generalization (00:06:45)**: "2.3x productivity improvement" claim lacks sufficient context about measurement methodology and confounding variables - **Appeal to authority (throughout)**: Panel format relies heavily on personal experience without citing broader research or data validation - **Post hoc reasoning (00:22:10)**: Some productivity claims may conflate correlation with causation regarding AI tool adoption #### Tone of voice - **Professional but accessible**: Technical discussion maintained conversational tone appropriate for practitioner audience - **Collaborative**: Panelists building on each other's points rather than competing perspectives - **Realistic**: Balanced discussion of benefits and challenges rather than purely promotional ### 🛡️ DARVO Analysis No DARVO patterns detected. The discussion maintains a balanced perspective on both benefits and challenges of AI adoption without defensive rhetoric or victim/offender reversals. ## 🎯 Conclusion Sustainable AI productivity gains require treating adoption as organizational change management rather than simple tool rollout. Success depends on systematic measurement across adoption, system improvements, and business impact, combined with deliberate investment in human skills like understanding, judgment, and AI supervision. The teams that will thrive are those that strengthen engineering fundamentals while embracing AI augmentation, creating feedback loops that prevent quality degradation, and preparing for the economic realities of AI cost optimization as subsidies end. ## 🔄 Refactor Appendix — Make AI productivity gains stick ### 📚 Literature Atoms #### AI raises coding ceiling but fundamentals become more important > [!quote] ID:rw_1 > AI raises the ceiling on what engineers are able to do. But for those gains to last, we have to kind of raise the importance of the fundamentals even more. **Context & use**: - Essential framework for evaluating AI adoption strategies and avoiding technical debt accumulation **Tags**: #ai-productivity #software-fundamentals #xref #### Mid-career engineers feel threatened while seniors and juniors benefit most > [!quote] ID:rw_2 > The biggest gains that we are getting is on seniors managers and followed by tadam by juniors. Mid-career engineers feel that AI is threatening to their identity. **Context & use**: - Critical insight for team management and training strategies during AI adoption **Tags**: #ai-adoption #career-development #xref #### Behavioral measurement requires layered approach across three pillars > [!quote] ID:rw_3 > What to measure I introduced this concept with three pillars. One is being this behavioral change, than system change and customer impact and all of them are having certain measurements. **Context & use**: - Framework for establishing comprehensive AI productivity measurement systems **Tags**: #ai-metrics #change-management #xref #### AI productivity requires moving from vibe coding to agentic engineering > [!quote] ID:rw_4 > Vibe coding could be useful but if we want AI productivity gains to last, teams have to move towards more like a disciplined agentic engineering. **Context & use**: - Key distinction for scaling AI tools beyond individual experimentation to team workflows **Tags**: #ai-engineering #engineering-practices #xref #### Understanding cannot be outsourced despite AI's capabilities > [!quote] ID:rw_5 > You can outsource some thinking to AI, but you can't outsource your understanding. AI can generate options, but engineers still own the understanding. **Context & use**: - Fundamental principle for maintaining engineering judgment and avoiding skill atrophy **Tags**: #ai-limitations #engineering-judgment #xref #### Current AI tokens are subsidized creating unsustainable usage patterns > [!quote] ID:rw_6 > Lots of tokens today they're subsidized like we are not paying the full price once the real prices will come all the gains that we have will be just losing those. **Context & use**: - Strategic consideration for long-term AI adoption planning and cost optimization **Tags**: #ai-economics #sustainability #xref <!-- deep-read:sections end --> ## 📺 Video  ## Highlights - Attributing all productivity gains to AI without controlling for other variables (process changes, tool improvements). Fix: Use controlled cohort studies. - Note: 🔍 Fallacy: Post Hoc #fallacy/post-hoc --- # 📝 Full Notes (Deep-Read Analysis) # 🏷️ Title: Make AI productivity gains stick ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ## 📝 Summary - **What happened**: Lead Dev panel on sustaining AI productivity gains with three engineering leaders sharing practical experiences. - **Why it matters**: AI adoption is accelerating but teams struggle to maintain benefits as tools become permanent parts of workflows. - **Key stakeholders**: Engineering teams, CTOs, staff engineers, mid-career developers facing identity shifts. - **Risks and tradeoffs**: Skill atrophy, over-reliance on AI, quality degradation, cost sustainability when subsidies end. - **Bottom line**: Sustainable AI gains require disciplined engineering practices, strong feedback loops, and treating adoption as change management rather than tool rollout. ## 📖 Main Content The panel features three perspectives on AI productivity gains: AB (CTO at Metro Fox), Anastasia (MVP Product Engineering at GitKraken), and Maxim (Distinguished Engineer at Target). **Core gains identified**: - **Increased test coverage**: AI translates specs into comprehensive test cases, improving developer confidence - **Delivery speed acceleration**: Features that took 3-4 days now complete in under a day with proper architectural design - **Faster ramp-up time**: AI agents help engineers understand unfamiliar systems and domains - **Context switching efficiency**: Multiple concurrent AI sessions enable broader project involvement - **Documentation productivity**: AI reduces "blank page" friction for design docs and communication **Measurement frameworks**: Panelists emphasize three-pillar measurement: behavioral change (adoption metrics), system change (DORA metrics), and customer impact (business outcomes). GitKraken reported 2.3x velocity improvements with 98% AI-assisted code changes. **Distribution challenges**: Gains aren't evenly distributed. Senior staff and junior developers see biggest benefits, while mid-career engineers struggle with identity threats as coding becomes commoditized. **Sustainability concerns**: Current token costs are subsidized; real pricing will force optimization. Teams need disciplined "agentic engineering" versus casual "vibe coding" to maintain benefits. ## 🗒️ Notes #### ⏱️ [00:03:12](https://www.youtube.com/watch?v=-6XbGcLEm7U&t=192) Panel introductions and context - AB: CTO using AI to function like much larger team - Anastasia: Product engineering leader studying AI impact on developer workflows - Maxim: Staff engineer focused on AI's effect on understanding unfamiliar systems #### ⏱️ [00:07:45](https://www.youtube.com/watch?v=-6XbGcLEm7U&t=465) Productivity gains experienced - AB: Test coverage dramatically improved, delivery speed increased - Anastasia: 2.3x velocity gains, 98% AI-assisted changes after 3-month transformation - Maxim: Faster ramp-up, better documentation drafting, expanded project involvement #### ⏱️ [00:15:20](https://www.youtube.com/watch?v=-6XbGcLEm7U&t=920) Measurement approaches - Three-pillar framework: behavioral change, system change, customer impact - DORA metrics remain relevant but need developer satisfaction tracking - Follow work end-to-end rather than isolated metrics to avoid shifting bottlenecks #### ⏱️ [00:22:30](https://www.youtube.com/watch?v=-6XbGcLEm7U&t=1350) Challenges and pitfalls - Skill atrophy from over-reliance on AI tools - Mid-career developer identity crisis as coding becomes less differentiating - Context requirements for large codebases still challenging for AI - Need to maintain experimentation mindset as AI evolves rapidly #### ⏱️ [00:28:15](https://www.youtube.com/watch?v=-6XbGcLEm7U&t=1695) Code quality and standards - Standardized markdown files for agent instructions across teams - Architectural decision records (ADRs) and reference implementations crucial - Better verification loops needed for agent output - Version control for agent instructions and standards #### ⏱️ [00:35:40](https://www.youtube.com/watch?v=-6XbGcLEm7U&t=2140) Making gains stick - Treat as change management, not tool adoption - Disciplined "agentic engineering" vs casual "vibe coding" - Strong feedback loops and engineering practices essential - Surface AI errors to teams to reinforce human oversight #### ⏱️ [00:40:20](https://www.youtube.com/watch?v=-6XbGcLEm7U&t=2420) Future-relevant skills - Understanding remains core: "you can outsource thinking but not understanding" - First principles thinking and architectural skills increasingly valuable - AI literacy and supervision capabilities essential - Form opinions before consulting AI to maintain skill development ## 📚 References - [Lead Dev Slack community](https://theledeveloper.slack.com) - GitKraken Desktop and VS Code tools - DORA metrics framework - SPACE metrics for developer productivity ## 🗣️ Paraphrase Three engineering leaders discussed how to sustain AI productivity benefits beyond initial adoption. While AI dramatically improves test coverage, delivery speed, and developer ramp-up time, maintaining these gains requires treating implementation as organizational change management rather than simple tool deployment. Key insights include uneven benefit distribution (favoring senior staff and juniors over mid-career developers), the importance of end-to-end measurement rather than isolated metrics, and the critical need for standardized agent instructions and strong verification loops. As AI token costs transition from subsidized to market pricing, teams must evolve from casual "vibe coding" to disciplined "agentic engineering" with clear intent, context, and constraints. The panelists emphasized that sustainable gains depend on preserving human understanding and judgment while using AI to augment rather than replace core engineering skills. Teams that succeed long-term will prioritize quality signals alongside velocity metrics and maintain strong feedback loops for continuous improvement. ## 📰 Newsletter **AI Productivity: Beyond the Honeymoon Phase** 🤖⚡ Ready to move from AI experimentation to sustainable productivity gains? This Lead Dev panel reveals the real challenges teams face when AI tools become permanent workflow fixtures. The standout insight: gains aren't evenly distributed. Senior engineers and juniors thrive, while mid-career developers struggle with identity shifts as coding becomes commoditized. GitKraken's 2.3x velocity improvements came with 98% AI-assisted code changes—but only after treating adoption as change management, not tool rollout. **Key takeaways for WordPress teams:** - Standardize agent instructions in version-controlled markdown files - Measure end-to-end workflows, not isolated metrics, to catch shifted bottlenecks - Prepare for token cost reality when subsidies end - Evolve from "vibe coding" to disciplined "agentic engineering" The panel's wisdom: "You can outsource thinking to AI, but you can't outsource understanding." Perfect advice as we integrate AI into WordPress development workflows while preserving the craft that makes great software. [Watch the full discussion](https://www.youtube.com/watch?v=-6XbGcLEm7U) | [Join Lead Dev Slack](https://theledeveloper.slack.com) ## ✅ Action Items #### DevRel - Create standardized AI agent instruction templates for common WordPress development patterns - Develop workshops on "agentic engineering" vs casual AI use for developer community - Document case studies of sustainable AI adoption across different team sizes and skill levels - Build measurement frameworks for WordPress-specific AI productivity gains #### OSPO - Establish guidelines for AI-generated code review and security scanning processes - Create policies for managing AI agent access to sensitive codebases and documentation - Develop training programs on AI literacy and supervision skills for maintainers - Design feedback loops for capturing and sharing AI productivity learnings across projects #### Community - Survey WordPress developers on AI adoption challenges and success patterns - Create mentorship programs pairing AI-experienced developers with those struggling with identity shifts - Establish forums for sharing standardized agent instructions and architectural decision records - Develop contributor onboarding paths that integrate AI tools while preserving core skills #### WordPress - **Core**: Integrate AI-friendly documentation standards and architectural decision records into development processes - **Core Performance**: Use AI agents for automated performance testing while maintaining human oversight for optimization decisions - **Core AI**: Establish responsible AI guidelines and verification loops for AI-assisted development - **Training**: Create curriculum modules on sustainable AI adoption and skill preservation for developers - **Documentation**: Develop AI-readable format standards for DevHub content and code examples - **CLI**: Implement AI agent scaffolding commands while maintaining educational value for developers ## 🧪 Hypotheses 1. **Skills distribution hypothesis**: AI productivity gains will continue to disproportionately favor senior engineers and entry-level developers, creating a "squeezed middle" of mid-career engineers who must rapidly upskill in AI supervision and product thinking. 2. **Cost sustainability threshold**: When AI token subsidies end and true pricing emerges, only teams with disciplined measurement and optimization practices will maintain productivity gains, forcing a second wave of AI adoption evolution. 3. **Quality debt accumulation**: Teams prioritizing AI-generated code velocity without equivalent investment in verification systems will accumulate technical debt that manifests as increased maintenance costs 6-12 months later. 4. **Context engineering becomes bottleneck**: As AI capabilities improve, the primary limiting factor for productivity gains will shift from model quality to teams' ability to provide structured context, documentation, and architectural guidance. 5. **Cultural adaptation timeline**: Organizations treating AI adoption as tool deployment rather than change management will see productivity gains plateau or reverse within 6 months, while those emphasizing cultural transformation will see sustained improvement. 6. **Measurement sophistication correlation**: Teams implementing multi-layered measurement (behavioral, system, business impact) will identify and resolve AI-related bottlenecks faster than those relying on single metrics like code generation speed. ### 📊 Methodology **Longitudinal cohort study**: Track teams over 18 months comparing AI adoption approaches (tool-focused vs change management), measuring velocity, quality, developer satisfaction, and business outcomes at 3-month intervals. **Skills assessment framework**: Develop pre/post evaluation instruments measuring architectural thinking, first-principles understanding, and AI supervision capabilities across different seniority levels. **Cost-benefit analysis**: Monitor token usage patterns and productivity correlation as pricing transitions from subsidized to market rates, identifying efficiency optimization strategies. **Context engineering audit**: Analyze correlation between documentation quality (ADRs, agent instructions, API docs) and sustained AI productivity gains across different codebase sizes and complexity levels. ### ❓ Questions - How do we measure the hidden costs of AI-generated technical debt that won't surface for months? - What specific training interventions help mid-career developers transition from coding-focused to orchestration-focused roles? - How can organizations prepare for the economic reality of unsubsidized AI token costs while maintaining productivity gains? - What architectural patterns make codebases most amenable to productive AI assistance? - How do we maintain engineering skill development when AI handles increasingly sophisticated tasks? - What early warning indicators predict when AI productivity gains are becoming unsustainable? ### 🔮 Future Research **AI supervision skill taxonomy**: Define and validate the specific competencies needed for effective AI agent management, creating assessment and training frameworks. **Context engineering best practices**: Systematic study of documentation structures, architectural decision formats, and instruction standardization that maximize AI productivity while minimizing errors. **Economic sustainability models**: Analysis of token cost optimization strategies and their impact on sustained productivity gains as AI pricing matures. **Team composition optimization**: Research optimal team structures and skill distributions for sustainable AI-augmented development workflows. **Quality prediction mechanisms**: Development of early-warning systems for identifying when AI-assisted development is accumulating unsustainable technical debt. ### 💡 Implications **For practice**: Engineering teams must evolve hiring criteria to emphasize architectural thinking, product understanding, and AI supervision capabilities rather than pure coding skills. **For policy**: Organizations need formal change management approaches for AI adoption, with standardized measurement frameworks and sustainability planning for cost transitions. **For security**: AI-generated code review processes require new verification paradigms and security scanning approaches to catch issues that human review might miss. **For education**: Developer training programs must integrate AI literacy while preserving fundamental engineering skills through deliberate practice and first-principles thinking. **For funding models**: Open source projects need sustainable strategies for AI-assisted development that account for token costs and skill development requirements for maintainers. ### 🧠 Learning Styles Assessment **Reading level**: Flesch-Kincaid Grade Level ~12-13. Technical content with management concepts requires college-level reading comprehension. #### 👀 Visual learners - **Present**: Panel video format with speaker identification - **Improve**: Process flow diagrams for three-pillar measurement framework, skill distribution charts, before/after productivity visualizations #### 🔊 Aural learners - **Present**: Full video discussion with multiple speaker perspectives - **Improve**: Audio-only podcast version, chapter markers for key topics, transcript with speaker attribution #### 🗣️ Verbal learners - **Present**: Clear definitions of "agentic engineering" vs "vibe coding," technical terminology explained - **Improve**: Glossary of AI productivity terms, analogy comparisons (muscle cars example), story-driven case studies #### 🧩 Kinesthetic learners - **Present**: Practical examples from real teams, actionable measurement frameworks - **Improve**: Step-by-step AI adoption checklists, hands-on workshops for agent instruction creation, measurement dashboard templates #### 🧠 Logical learners - **Present**: Three-pillar measurement framework, cause-effect relationships for skill atrophy - **Improve**: Decision trees for AI tool selection, ROI calculation templates, systematic troubleshooting guides #### 🧑‍🤝‍🧑 Social learners - **Present**: Panel discussion format, community Slack continuation - **Improve**: Team retrospective guides, peer mentoring frameworks, cross-company experience sharing forums #### 🧘‍♀️ Solitary learners - **Present**: Self-assessment questions embedded in content - **Improve**: Personal AI adoption journal prompts, individual skill development tracking sheets, reflection exercises #### 🌿 Naturalistic learners - **Present**: Ecosystem thinking about AI impact on engineering culture - **Improve**: Analogies to natural adaptation processes, sustainability frameworks, balance considerations ### 📊 Accessibility Assessment #### Screen reader semantics - **Present**: Video has transcript, structured heading hierarchy - **Improve**: Speaker identification labels, timestamp navigation, alt text for any visual elements #### Keyboard navigation - **Present**: Video player supports keyboard controls - **Improve**: Skip links to key topics, keyboard shortcuts for transcript sections #### Contrast and readability - **Present**: Text transcript provides high-contrast alternative - **Improve**: Increase font size options, check video caption contrast ratios #### Text resizing and reflow - **Present**: Responsive transcript layout - **Improve**: Ensure 200% zoom compatibility, test with screen magnification #### Media and captions - **Present**: Full transcript available - **Improve**: Synchronized captions on video, audio descriptions for visual content #### Forms and interactions - **Present**: Basic video controls accessible - **Improve**: Enhanced player accessibility, interactive transcript with search ### 🌍 DEIB Assessment #### Cultural breadth - **Present**: Global perspective from different company contexts - **Improve**: Include perspectives from smaller organizations, non-Western development practices #### Inclusive language - **Present**: Gender-neutral technical terminology - **Improve**: Define all acronyms and technical terms, avoid industry jargon assumptions #### Neurodiversity support - **Present**: Multiple information formats (video, transcript, structured notes) - **Improve**: Clear section breaks, predictable information architecture, summary bullets #### Representation - **Present**: Mix of leadership levels and company sizes - **Improve**: Include underrepresented voices in tech, diverse geographic perspectives #### Psychological safety - **Present**: Acknowledges challenges and skill concerns honestly - **Improve**: Validate mid-career developer concerns, provide reassurance about adaptation paths ### ✅ Logical Validity #### Strong reasoning - **Empirical evidence**: GitKraken's 2.3x productivity measurement with specific methodology (cohort comparison, before/after milestones) provides concrete validation. - **Causal analysis**: Clear connection between subsidized token costs and current AI adoption patterns, with logical prediction of behavior change when pricing shifts. - **Systems thinking**: Three-pillar measurement framework addresses multi-level organizational impact rather than isolated metrics. #### Fallacies - **Survivorship bias**: Panel focuses on successful AI adopters; teams struggling or abandoning AI tools aren't represented. Fix: Include case studies of failed adoptions. - **Post hoc reasoning**: Attributing all productivity gains to AI without controlling for other variables (process changes, tool improvements). Fix: Use controlled cohort studies. - **Appeal to novelty**: Assumption that AI-first approaches are inherently better than traditional methods. Fix: Compare outcomes rather than approaches. #### Tone of voice - **Professional optimism**: Speakers maintain positive outlook while acknowledging challenges ("excited to speak about gains as well as pains"). - **Practical realism**: Honest about difficulties and sustainability concerns without dismissing AI value. - **Collaborative expertise**: Panelists build on each other's points rather than competing, creating comprehensive perspective. ### 🛡️ DARVO Analysis No DARVO patterns detected. The discussion maintains professional tone focused on practical challenges and solutions rather than defensive rhetoric. Panelists acknowledge AI limitations and skill concerns directly rather than deflecting criticism. ## 🎯 Conclusion AI productivity gains can be sustained through disciplined engineering practices and treating adoption as organizational change management rather than simple tool deployment. Success requires standardized agent instructions, comprehensive measurement frameworks, and deliberate skill preservation as teams transition from "vibe coding" to "agentic engineering." The most critical insight: understanding remains the core human competency that cannot be outsourced to AI. Teams that maintain strong verification loops, measure quality alongside velocity, and prepare for economic sustainability when token subsidies end will see lasting benefits. Organizations must actively support mid-career developers through the identity transition as coding becomes commoditized and orchestration skills become paramount. --- # 📄 Full Transcript *Auto-generated from YouTube captions · timestamps roughly every 30 seconds · [watch on YouTube](https://www.youtube.com/watch?v=-6XbGcLEm7U)* **[00:12]** Hello everybody. We're just going to get started very very shortly. But thank you to those people who I know are already here waiting for us to to get on. Uh really appreciate you joining us today. Okay, I think we're going to get cracking then. Um, so yes, thank you so much for joining us for what promises to be a fantastic discussion on making AI **[00:43]** productivity gains stick. And this is a lead dev online panel discussion created in partnership with Git Kraken. The discussion today is going to last roughly 45 minutes after which both myself and the panelists are going to head over to the lead dev Slack to answer some of your questions in the panel Q&A channel. And to get there, uh, if you're not familiar already, you can go to theledeveloper.slack.com and either sign in or join up. You don't have to hold on to your questions till **[01:13]** afterwards though. Please feel free to submit any that might come up during the course of the conversation through the Q&A feature on Zoom. If I can bring them in, hopefully I will try to do my best to do so. Otherwise, we'll uh hopefully get to a couple at the end if we've got time. And if not, we'll port everything over to Slack for you. But before we get started, I will do a very quick introduction. My name is Sally Lates. I will be your host today. My pronouns are she and her. I work as a fractional technology leader and adviser and I'm **[01:44]** based in the east of England in the UK. But as I've said, the discussion today is all about making AI productivity gains stick. So I imagine everybody on the panel or everybody watching here today has probably seen some different benefits in the course of their work and their life. But making sure that these gains really hold up as AI becomes more of a permanent part of how teams work can be another matter. So what we're going to be talking about today is discussing the different kind of gains **[02:14]** that we can get that our panelists have experienced in the course of their work. How we can measure these, how we can um most importantly get them to stick as part of our everyday work. So here to share their experiences, their stories. I'm joined today by um some incredible panelists and I'm going to let them all introduce themselves very shortly in alphabetical order because I love a bit of structure. So AB will be starting with you. Um could you tell us a bit about you please, the context of your work with AI and why you're interested **[02:46]** in this topic? Hi all, my name is AB. I'm currently the CTO at Metro Fox and as a small team, we function and ship like a much bigger one now. And that's because AI has become an integral part of our workflow. And so in so many ways, I get to see the gains as well as the pains. and I'm excited to speak about them all today. Thank you. >> Very excited to have you here. Thank you. And Anastasia, over to you, please. >> Thanks, S. Uh, great here to be with **[03:18]** everyone. Um, Anastasia, MVP product engineering, the Git Kraken. Um, many of you may know us, um, uh, through tools such as Git Kraken Desktop, uh, or GitLab, a plug-in in VS Code. But over the last 18 months, we've been deeply investigating and trying to understand how AI is impacting developer workflows and how we can help developers and dev leaders through this. So into this discussion um I'm bringing my personal **[03:49]** experience as I'm running engineering team and we are going through the same transformation as well as um inputs from lots of um dev leaders that we were talking to um that again like going through the same and wrestling through the same pain points as you are happy to be here. >> Thank you. I'm really excited to hear some of those those data points some of those anecdotes that I'm sure you can bring and Maxim over to you as well please. Yeah, it's great to be here. Uh, thanks Ali. Um, I'm Maxim Najim. I'm a distinguished engineer at Target and **[04:20]** I've spent about two decades working in the tech industry across large tech organizations. Like many of you, I've been using AI in my day-to-day engineering work. Um, in my recent article I wrote for lead dev, I wrote about how AI is changing my work as a staff plus engineer, especially how it can reduce the time it takes to understand unfamiliar systems. So to me, it's clear that AI can help us move faster. But I think that the deeper question is, can AI help organizations **[04:52]** make better, safer changes faster? And what I found is that answer depends on a few things which I'm excited to uh to to discuss today. But what I will say now is like AI raises the ceiling on what engineers are able to do. But for those gains to last, we have to kind of raise the importance of the fundamentals even more. So like good architecture, good guard rails, keeping the craft of software development at the center while uh using these tools what they you know to to do what they truly help us with. **[05:23]** Well, you're all dropping some really nice sort of clickbaity little tidbits here already. So, that's uh very exciting. And thank you all so much for giving your time today um to to chat to me and to chat to um everybody else out there as well watching. Um but we're going to start by taking this idea of gains. So, again, I'd love to kind of go around the virtual room and get everybody's perspective to get us started because I know that everybody's kind of coming at this from a slightly different angle. Um, so AB, if you could **[05:54]** kick us off again, what kind of gains have you personally experienced AI bringing to your work or or to the environment that you've been in? >> Yeah, that's a great question. I can speak to two gains in particular and the first one being increased test coverage. Testing used to be this thing that although very important required a lot of time and investment and did feel like manual labor. So you would find that **[06:24]** developers in most cases did the barest minimum. However, now with AI being able to translate specs into test cases, we're seeing higher test coverage. We're seeing PRs with more test cases, more scenarios being tested and that's giving us more stability and more confidence when we ship because we know that if something were to go wrong, we would know ahead of time. The second one is increased delivery speed. Right now we are seeing that what we as engineers **[06:56]** need to lock in most times is the architectural design and those decisions. And once we have that down AI can then go and do the code implementation. Before now a feature could take about 3 to four days after you have locked in the um the architectural design. But now once you do that AI can get that done in less than a day and that same developer can then work on three other work streams at the same time thereby having more gains **[07:27]** in terms of speed. And is that kind of echoing what you've been seeing as well, Anastasia, or do you have any other kind of like different viewpoints to add into the mix? >> Absolutely. Absolutely. Similar kind of insights. Um I'm looks I'm looking um also at uh the gains from maybe this three pillars. The first one is around adoption. Um and are we doing the cultural change and initially it it could be like very very like AI assisted **[07:58]** like uh very minor adoption and then go into AI first uh kind of flow. Then if you have this leading indicator of um people adopting AI, do we see that our system is changing such as we have better test coverage such as if you have better velocity um if you release more frequently. So this is second pillar but what we are all marching towards is what's the impact to the business? Are we able to deliver value to the customers uh more? So I see the the **[08:30]** gains in all of those areas and they're also super interdependent in in in our case. Uh if you talk about impact um then when we really leaned into EI first uh type of model we see um just in 3 months um 2.3 times perform uh improvements in in velocity and um around 98% of of our code changes are AI assisted but it **[09:01]** wasn't a a journey that um just we did overnight there was a a journey with initially developers resisting then going through experimentation adoption and then scaling and optimizing and of course something that helped is changing the cultures uh um releases in the models um with OPUS like I think the whole world for everyone has changed uh with GPD 5.1 um so we really saw um ma maj major major changes but yeah um so **[09:33]** see similar kind of observations as Abby And Maxim, I know that you really care about the cultural side that Anastasia mentioned. You really care about the adoption side. Um, but thinking about the gains specifically, how uh how has that kind of affected your work, what what sort of things have you been seeing? >> Yeah, I have about like three of them that I think are major gains that I've gotten. So, one is kind of just the ramp up time. For example, if I use AI agents to kind of explore a domain or system I **[10:03]** don't know well. So I have an AI agent read the documentation, compare that to the code and kind of help me understand how a system is structured. Uh that kind of like onboarding work usually takes a lot of time and so AI could help me kind of compress that time and kind of reach a first useful understanding. From there, you know, I take that initial understanding and validate it with engineers who actually own those systems. And I'm I'm a staff plus engineer. So I have to create a lot of documentation, a lot of lead a lot of conversations. So **[10:35]** I've also been seen a real gain in the I call the blank page part of my work. So things like drafting a design doc, turning messy notes into like clear arguments, summarizing trade-offs, um adapting communication for different audiences. So for me, AI kind of lowers that activation energy. So instead of spending an hour just getting to a workable starting point, I can get there much faster. Then I can spend more time kind of applying my judgment, refining the argument, um getting alignment. **[11:06]** And another kind of big shift has been around just context switching. So on my desktop right now, I have six or seven coding agents sessions running at a you know, they're open and running. And so each each um tie to kind of a different project I'm involved in. And they help me kind of research a topics, synthesize notes, review the ideas, uh create artifacts for collaboration. And so over the last four or five months, that has changed the shape of my work. I've been able to kind of stay involved in more **[11:36]** projects than I normally could because AI helps me preserve the context, kind of re-enter projects faster, and then kind of turn my scattered thinking into something I can share with others. So for me, it has expanded the surface area of the projects I'm involved in where I can provide more useful input, feedback, and direction without relying kind of on my own memory or finding long stretches of uninterrupted focus time. >> I've now got this picture of you with **[12:07]** your five or six AI agents kind of like doing all the work in the background whilst you're here chatting to us. So, thank you. And I hope they're not too distracting with everything that's going on. Um but just to kind of like pull on that thread a little bit more um your role I think we've all got some quite different roles here but in particular what's your experience of um how those gains are perhaps distributed around the people that you've been working with or how have you seen them affecting different roles? Are they pretty evenly **[12:37]** distributed? Are they concentrated at different levels of seniority or or types of uh work? What's what's your experience Maxine there? Yes, I I think the AI gains are strongest when people kind of bring the right context, the right judgment and verification to these AI tools. So for for SAP plus engineers, that's that changes kind of the center of gravity of the role. If if code becomes cheaper to produce with AI, then the scarce thing becomes knowing what code should exist. You know, what **[13:07]** constraints matter and how does it fit into the wider ecosystem. And so that means AI is is most valuable in the kind of work where staff plus engineers are already spent spend a lot of their time. And so I think AI makes the staff plus role more important and shifts those that role towards kind of shaping intent um designing verification loops creating alignment across teams. So those are already like staff plus responsibilities but AI makes that role even more important. Um and so I I also think step **[13:40]** plus role is making sure the AI adoption does not stop at like people are just using the tools. It has to show up in the team's operating model how people plan, shift, verify and learn and uh that is when you see kind of the true gains of AI and that's where that productivity gains can and can can start to stick. I was actually reading a lot on on this and um I would like to compare with experience that that we were having in in our teams. Um and maybe we're having **[14:11]** a little bit different experience than what is the industry term definitely what I see that that it's not evenly distributed at all. Um and the biggest gains that we are getting um is on seniors managers and followed by tadam by juniors. Uh so and this is both true in in in develop in across devs and and QA midcareer engineers that worked so hard **[14:43]** to become those people who gained like to that that experience that they're writing great code um they are a little bit stepping back they feel that AI is threatening to their identity um and um I really love growing people I I see that this is a mission for for uh for dev leaders and uh for me this is how can I help them to uh to get this new identity where they'll be not feeling threatened. Um so in this case **[15:14]** how we can train them to become managers because basically like why it's so easier for staff plus for for Maxim um to to orchestrate six like 10 agents in parallel because they know how to delegate. They know how to set clear goals. they know how to review the code not micromanage uh how to do the retrospectives and provide the feedback back to to this. So these are the skills that that we are teaching for the mid-career engineers for them to get the new the new identity. Um and definitely **[15:47]** when we are going more towards um hiring um then in this in this case uh we are looking for true product engineers um coding is not enough um they need to be orchestrators uh and also understand the business and be able to solve the problems not just uh writing writing the code. I've definitely been hearing that a lot. I think the the balance of the the product thinking is going to be incredibly important um in terms of people's development and and things like that. But we're going to come on a **[16:17]** little bit more to some of the human stuff. Um I'm really glad you called that out. I think it's very very important. But one of the things that I know can really help people is to understand um the measurement the the sort of the structure around these kind of topics to make them a little bit less ambiguous. So when you think about measuring the gains um and measuring the impact on different ways of working and and different facets of the business, how do you think about that Anastasia? **[16:48]** >> Um something that that um I'd like to start with is um now measurement are becoming even more important than at any other other time. Um we are going through the major change management. So, so that's why be able to measure understand what to measure is is super critical and talking to the dev leaders who had that experience before or who are obtaining that that experience urgently now. I think these are the teams that are showing the biggest improvements. Uh for me what to measure **[17:21]** um I introduced this uh this this concept with three pillars. Uh one is being this behavioral change. um than system change and customer impact and all of them are having certain measurements that uh that um um you you would you you would be adopting depending on the phase of your AI adoption for example for behavioral change it could be initially is there engagement that is happening is there adoption is there frequency of the use but when uh you are in more um maxims type type of experience um uh are your **[17:54]** agents able to run autonomously for a long time without much of the of the supervision without making too much of mistakes. So there will be different type of measures uh on the behavioral change for the system. Um I would I would rely on on the DORA type of metrics uh on on uh space type of metrics. Um here super super critical with I in addition to the adoption we need to as dev leaders track developer satisfaction **[18:25]** because in this world when you're running tens of agents it's so easy to get to burnout and like super easy not because people are are really like trying to do as much as possible just because that enderfin that you're getting from this short from from the quick wins It's just like keep going, keep going and and now it's like oh it's already like 18 hours without non-stop like easily. And yeah, the last is is customer impact. Um are we running more **[18:57]** experiments? Are we shipping faster? All of those things. Do we see customer satisfaction improvements? These are the the metrics that I would be looking at and they're all important. We can't ignore um just adoption. uh we uh we we this is leading indicator that your your system is changing and the end goal is is customer of course and just as you started speaking we had a question come in um on the zoom Q&A um which again just a reminder to everybody you're very **[19:27]** very welcome to submit things there as well um I think you've perhaps answered quite a bit of it so the question was um could you please name some specific metrics and value that are specific for AI and overall improvement and efficiency with using AI. But then there's a follow-up of how are you able to report this up to the board and executive leadership to show how AI is able to improve products collaboration and ultimately success. So I wonder if there's anything else you just wanted to kind of build on with your with your **[19:57]** previous answer or any other points that could be useful to this uh this person. >> Yeah, on the on the metrics um again it depends on where you are in adoption. Um so so that's why initially it could be maybe line like lines of code or acceptance rate that kind of things on on the far spectrum it it could be are your agents working efficiently um and taking into account the cost that will be coming up very very soon as as a problem your **[20:27]** uh your investment having the the right level of ROI um in Dora it's it's all clear space also like just just take those frameworks and and apply um for customer impact what I'll suggest is um especially talking to uh to the board and and uh exposing the the business metrics look for the signals what are important for your company. Sometimes it could be um uh is your go to market faster then are you shipping faster? Do **[20:59]** you have more experiments? Sometimes it could be um is your is your organization cost efficient? So get get those questions that are important for the board that are important for exact level um uh and and try to translate the metrics the system metrics that you have towards that. Um for us uh we we were sharing the productivity gains with with with a board and uh something that I shared just January versus April it's uh 2.3 across all the team. Of course it's **[21:30]** not equally distributed but it's across the teams. So you can use this this comparison before certain milestone and after certain milestone. In this case there could be also some other things that are changing in parallel your process changes um or your uh your tools introduction. So there could be a little bit of variance. Um other way how you could measure is building on cohorts and comparing. um you will have people who are um in **[22:01]** learning AI adoption and you will have um really those people who are going going aggressive they're going through the same or organizational changes. So um there if there's significant um uh number of people in in your team and you can you can derive from that then potentially you can do the cohort uh analysis and compare uh one cohort versus another and represent this as um as the data for your board. >> Thank you. I think you've touched on some really interesting nuances there. **[22:33]** One of the things that I'd just love to to stick on for for a minute longer is just how do you separate those real genuine productivity gains um from gains that maybe just shift the work elsewhere? So into review or debugging or or rework. How how do you kind of actually get down to the nitty-gritty of measuring the the real gains? >> Um I can I can take this question as well. I'm very passionate about it. about about about the the data. Um, an **[23:04]** analogy that I like to to uh to use is now we are getting all these muscle cars that are 10 times faster, 100 times faster, are we getting home sooner? And I think in most of question in most of areas it's like no. And then the question raised why why is why is not it's not that. And here um a little bit of bummer. There is no one sizefits-all. There is no one recommendation. Oh, it's **[23:35]** because of this and this. But my recommendation is is um is use this layered measurement and this is why why it measures without this uh without the endto-end visibility. Um you can say that my coding has improved to that my code my my test coverage have improved by that. But you'll be not tying this. Are you seeing the business outcomes? So having the the layered measurement is um is super important. Um for for us some **[24:06]** of the bottlenecks that we we gain after this uh this transformation is is like we have uh PR review bottleneck, we have requirement bottleneck and this gives opportunity as as for for organization to tune for the next things. So bottlenecks are not are not bad things like if you know about them because this gives you an opportunity to really tune and get even more from uh from from um what what you have and yeah **[24:37]** any anyone else would like to add >> I want to add something to that I feel that it's highly important to follow the work through as opposed to following stages. So some people say oh we had two times PRs and then PRs have have increased by four times or five times. However incident rate has also increased. So we need to be able to follow the work through from inception to delivery as well as what happens after. So if we're able to say we're **[25:08]** shipping 10 times faster, however the incident rate has stayed the same or even reduced now we're sure that we haven't moved that work elsewhere. So if we follow the work through completely, we're able to tell if we're really having AI gains or the work just moved elsewhere. >> Well, let's stay on that slight downer then. Uh Amy, I'd love you to take us through um maybe some of the pitfalls that you've seen um or maybe some of the gains that look great but then ended up being quite fragile um and you know **[25:40]** disappeared or something like that. Could you tell us a few stories about uh some things in the in the less good realms then? >> Yeah. Okay. So I can speak to this in two contexts. In the context of the user and in the context of the usage. In the first, I'm seeing more and more developers become overly reliant on AI such that they begin or they stop working their engineering muscles. For example, there's an incident and the **[26:11]** first thing the dev does is to take a copy of the stack trace, place it on um on cloud code and then return to say plot said the issue could be a b c d and you're wondering with the amount of context you have if you just took some more time and looked more closely you would see that the issue couldn't be any of these things and it's totally something else. So with more people outsourcing their thinking to AI, we're seeing skill atrophy and that isn't good **[26:42]** for any team at all. And that's why I agree with something Stasia said around mid devs feeling like their job is gone because most of what they knew was just to code. And now that AI is doing more of that, we're seeing that it does seem that they don't have much to actually bring to the table anymore. Now, in terms of usage, I would say that I've seen AI gains look fragile when we **[27:15]** apply them to um large code bases, especially where you need a lot of context to be able to generate good working code. And most of the time the context lives either in an RFC somewhere or in someone's brain somewhere else. And it's difficult and very challenging to pull all of that together and give that to AI. So in those situations you will find that in isolation the code **[27:47]** looks good but then as part of a whole it doesn't help anyone at all and there's actually higher churn rate. So in areas where you need a lot of context, I'm seeing that we can't really find stable AI gains there unless we do the work of investing in providing that context. >> Yeah, thank you. And um I've had a question come in that Oh, sorry, Anastasia. Did you want to chip in? **[28:18]** >> Yes. Yes. uh if you if if you talk about the context something that that is is important for us to keep gaining this uh this this values is how we can learn from the past mistakes. Um so agents will be not repeating this over over and over again. Um this this is something that will be making us less fragile in in in the future. Um and another pitfall that that I saw is um people were getting to AI adoption uh and then they **[28:51]** stopped experimenting that we already um have AI adopted uh we we're good let's focus on our road map or whatever thing is that AI is changing so fast and this is not just step function that you conquered and that's it. So what we need to do is keep having this build measure um learn loop otherwise we'll be losing our our gains. And something that I personally um is is is is afraid of of um is that lots of tokens today they're **[29:24]** subsidized like we are not paying the full price once the real prices will come and um I'm looking right now at at yesterday's announcements from cloud code that starting from June 15 paid cloud plans include monthly cloud agent SDK credits. So this is beginning towards that. So that means that if we go in in that route and it we will be going there 100%. um all the gains that we have with AI with uh be able to **[29:56]** work with parallel 10 of agents in at the same time we'll be just losing those uh and for now for for us I think what we need urgently do while we still on that subsidize we we need to um uh to see how we can use those uh tokens efficiently but you can do this only after you scaled so first we need to scale AI adoption But after that optimize and we have an amazing phase right now to to get the biggest ROI from **[30:27]** each token that we're getting. >> Super important point I think on a number of different fronts. But as you say the the sustainability angle is going to be I think a real discussion point uh once this initial kind of revolution has happened. There's going to be I think another wave of uh evolution after that. Thank you. Um I wanted to go to a question from um Vishal, thank you for the the question which was what have you done for your code bases to generate reliable code **[30:58]** which follows the standards and has context of other code bases and it feels like this is coming from a place of Michelle's possibly had a bit of a painoint. So Maxim, I'd love to throw that one over to you if possible. Um I wondered if you had any thoughts on this one. Yeah, I think I think you have to have in your repo the right instructions and markdown files about the what your repo is about. There's also um you know agents MD and skills MD things that you can markdown files that can help uh **[31:29]** define how to interact with the codebase. And you definitely want to expose your architectural intent more explicitly like your ADRs, um, reference implementations, things um, and give those as MCP tools to the agent so that it's able to discover those as it needs needs to do that. Um, and so I think that's really the context engineering aspects of of doing that correctly. Um so you need you need to have those standards those examples easy **[32:00]** to retrieve for for the agent. Uh and then you also need better verification right so whatever the agent produces you also need to uh have the right uh quality checks and verification in place so that it's not you know whatever it produces you're blindly accepting. >> I'd like to add something to that. I've encountered situations where we have different developers, they have their own agents and all the agents have their own MD file and they're doing things **[32:31]** their own way and it's like what's the standard here exactly? So I think it might also help as a team to come together and say what are our standards put that in an MD file what are our architectural decisions if you have a file like that where you add the links based on the domains or the models and this should be git like you should be able to actually commit this to git so that as it changes everybody gets those changes and then all the agents are working with one instruction as it **[33:02]** regards the standards and where they should pull docs from. I think that has actually helped my team because we were at a point where everybody's agent is just doing what they want and then you have to constantly be reviewing the code to say oh no this is not the standard but once we were able to standardize that in one file that everybody then shares things became much easier. And talking about measurements, sorry that this is that this is that person uh something that will be very good um **[33:33]** across all your repositories have that kind of assessment as a score um like um uh how we are trending. Are we are we improving on on our um uh on our instructions? Are they standardized? Um and this can be building a road map towards what do you need to invest next. So just having that visibility. >> Well, we could probably stay here all day talking about challenges, difficulties, hearing stories from you **[34:04]** folks, hearing stories from the the folks watching. Um, but I did want to return us to the topic uh or the title that we were going with for this session, which is how do we get the gains to stick? So, we've heard about a few ways um that we can actually improve adoption. we can improve some of our practices, but what are the really deliberate things that we can do to mitigate those potential issues um or to make sure that we are actually actively working towards getting gains to to **[34:35]** stick. So, Anastasia, I know you really care about this. Is there anything else that you'd like to add into the mix here? >> Um yeah, yes, absolutely. um in order for games to stick um we we need to treat this not as as a tool adoption um or a tool roll out. This is our like biggest change management that we went through for a long time. Uh this is our workflow and c cultural change. So that's why setting the goals what um setting the standards are super important. Adoption is just like minor **[35:07]** minor step here. Um and as a part of this transformation, growing people, growing developers, um uh QA folks to become the the the the product engineers. Then um uh I think I mentioned this in one of one of the one of the questions is how we can create that learning feedback loop um that um if something is not working or we are got getting not the efficient results from uh from from the agentic workflows how we can bring this back to **[35:41]** you as a developer like have that then how we can bring this to the team through for example this g versioning inside repository that Abby was was was talking about and then actually to the agents themselves. So they will be reusing that that information uh uh and that learnings together. Um and yeah for for for me again measurement person here uh love the data so much. Um we we need to monitor our **[36:11]** engineering organization at daily weekly basis and see what are the what are the trends uh because something maybe was working great but then some small portion have changed and you have this butterfly effect where the quality have degraded um and you think that everything is great everything is is good my velocity is is is amazing and you are having this side effect so measuring velocity speed quality and and please please measure developer experience. Burnout is is real thing **[36:43]** that we need to take care of. Um and when you're looking through this through this data then identify what are the new raising questions new raising issues and this is what will be helping this the the changes the the improvements that you have already today stick and you will multiply you will improve and improve and improve. Let me add something to that. This might sound really funny, but there's something I do on my team which is surfacing AI errors to the whole team. **[37:14]** We kind of joke about it, but it helps to reinforce that that learning that AI is not the driver. You are the driver and AI is the tool and you shouldn't let AI be the driver because AI can actually make mistakes. So at the times where we see this, we just kind of surface it to the team and be like AI just did this and this wrong. That just lets everyone know that yes, **[37:45]** even though AI is giving us these gains, we should actually still have that human in the loop such that we are overseeing the work and we don't get to the point where it's like everything hits the roof. Yeah. And I like the distinction between vibe coding and kind of agentic engineering because I think it gets to something important. So, you know, vibe coding could be useful. It helps people explore, prototype, get something out **[38:17]** quickly. But if we want AI productivity gains to last, teams have to be move towards more like a disciplined agentic engineering. To me, that means the agent is like operating within a real engineering workflow. There's clear intent. There's curated context. There's explicit constraints. There's tests, boundaries, and reviews. And that requires more more than just better prompts. It means teams need to to have better specs, uh, you know, better instructions, better internal documentation, a lot a lot of stuff **[38:49]** we've been talking about. Uh, we Yeah. And we also need to make the systems, you know, more legible for agents. So they might you means better API documentation, MCP servers uh you know for the agents to have to inspect our environment better and yeah context engineering you know matters matters a lot here um but has to be maintained like any other uh parts of our engineering system engineering systems and the failure mode for me is not always that you know AI generated code that fails immediately or is bad code **[39:21]** often the code works today but quietly it kind of raises the cost of every future change. So the team that makes that the gain stick will not necessarily be the team with the most, you know, tokens or the newest models. It's those that uh they will be the teams that have the strongest feedback loops and the most disciplined engineering practices around these AI assisted work. >> Well, let's stick a little bit there on the on the more human side of things then. What kind of skills, Maxine, would **[39:52]** you say are the ones that are going to become most valuable as AI improves, helping us to really kind of maximize those gains? >> Yeah, I think as as AI improves, I think the one of the most valuable human skills is is understanding. So, there's a there's a line I I like a lot, which is you can outsource some thinking to AI, but you can't outsource your understanding. And so, AI can help draft the spec. It can explore the code base. It can generate options. It can write the test. It can propose a solution. But **[40:22]** as the engineer, I'm still I still need to understand the problem deeply enough to know whether we are solving the right thing. And I need to understand the solution deeply enough to kind of explain why it might work, where it might fail, uh what the trade-offs, you know, might be and whether we can maintain it, right? And that that becomes even more important as AI output gets more polished. And you see kind of the the weaker AI answers are easy to reject, you know, but a plausible polished answer is more dangerous **[40:53]** because it can look right while hiding kind of a bad assumption or a security issue or design choice that does not fit the system. So skills like judgment, taste, domain understanding become more important and all those skill depend on on us understanding you know and what what AI is proposing. And if um and if I if I don't understand the problem, I can't give the agent good intent. If I don't understand a system, I can't provide good context. And if I don't understand the solution, I can't, you **[41:24]** know, reasonably approve it. So I would summarize it like AI can generate options, but engineers still own the understanding. And the more capable AI becomes, the more important it is that humans kind of stay accountable for what the system is actually doing and and why. Ay, anything to add on that front? >> Yeah, I'd say two things are very important now and that's understanding things from first principle and **[41:55]** architectural thinking because if you don't have the deep understanding, you can't actually critique AI's work or push back where necessary. So right now it has become extremely important for developers, engineers everywhere to begin to understand things from first principle so that you can actually push back and critique work of AI. >> I agree with uh with all points from Maxim and Abby. Um maybe one more thing that that is um that that I'll add here **[42:26]** is being uh like understanding AI and AI literacy and um know how to supervise AI efficiently. Um this is the skill that we're obtaining at very rapid pace right now. Um that this will be this will be super useful combined with your human judgment with ethics with with deep do domain knowledge with um uh with with this um system thinking. Um yeah, we've actually had we've had a ton of **[42:57]** questions. So thank you so much um for everybody who's really been engaging with this. We're not going to get to them all. Um but I did just want to pull out one last one which I think is related to this of what ways have you found to avoid engineering skills atrifizing or atrophing? Um and I think my interpretation of that is maybe um some of the more um dare I say air quotes traditional engineering skills. We've talked about the over reliance on coding agents and things like that. But **[43:28]** I'm guessing it also ties into um getting left behind, not being up to date. So, do any of you have any thoughts on on how to keep skills relevant, how to keep them up to date, and how to do some deliberate practice around some of the things that we might lose along the way? Um for me maybe it will be um keeping curiosity for for for um deaf leaders for as developers as as as SQA and keep **[43:59]** learning keep learning. This is this is one second um is uh how you we can understand better customers and um their problems what we are trying to solve what's the best way to solve this. So again code writing code is is cheap now solving problems is still very relevant. Um this will be number two and number three um we still need to understand how system is working. We still need to understand u what their architectural **[44:30]** principles. So this these are the things that um for me will be top three. There'll be definitely more more other things toward that. >> I can add something to that which is the fact that you should try to form an opinion before you ask AI anything. So before you say how can we architect a system that does A B and C try to architect it yourself first and then you can compare find gaps and that way you are actively growing as **[45:02]** opposed to just asking AI to do everything for you and then you just review. So this way you still be flexing the muscles and growing them. >> I feel I was going to say you're absolutely right and then ran the risk of sounding exactly like AI. So, thank you for that. Um, so I'm just going to kind of bring us to time in a sec. Uh, I think this time has absolutely flown by. So, thank you for sharing everything that you have. But what I'd like to do is just go around the room really quickly, give you a chance to just um **[45:34]** offer any closing advice that you'd like anybody listening today to to take away from this conversation. So, uh, Maxine, let's start with you. >> I would echo what Anastasia was saying. I think you know you have to measure quality not just velocity and it's easy to just count the commits and PRs and feel good about you know AI but the harder and more kind of important question is will the code base be healthier in 6 months from now and so building a visibility into the quality signals looking at code duplication look **[46:06]** at code health look at incidents maintainability uh look at you know whether teams are actually able to change the system safely over time. I think the team that wins with AI in the long run will be the teams that make the quality signals just as visible as the speed signals. >> Fantastic. And Amy, how about you? Any closing wisdom? >> I would say as a developer, do everything that you can to make sure that your skills do not atrophy. **[46:38]** >> Lovely. Short and sweet. And Anastasia, >> this is this is super wise. Abby um superwise um for me um we we're going through the major transformation that we we never went through previously. Um and as as this is major change management be able to build system tune this measure and learn and keep iterating and having this loop is super important. It was **[47:08]** never that important as today. So that that will be my my recommendations um to have really gains that will stick that will multiply that will grow and not have the major side effects on your company um and and your customers. >> Well, thank you all so much for sharing. It's been great to hear some different perspectives. It's been great to hear um how you've practically been uh attacking some of these topics and and making them **[47:40]** work for you. And thank you as always for sharing all of your all of your thoughts, all of your wisdom with the audience today. Um what we're going to be doing is heading over now to the lead dev slack and we're going to be in the panel hyphen Q-N a channel. Um so for all the folks um there's a few outstanding questions, some really really good ones, some meaty ones. I'd love to have had time to discuss the mid-level devs one a bit more. I think that's a really fantastic conversation. Um, if you want to **[48:10]** continue the conversation, if you want to get the answers to your questions, please head over to Slack with us now to continue the discussion. And just a reminder, if you're not in there already, you can get to it through the lead developer.slack.com. But thank you all so much for joining panelists, everybody watching, and we hope to see you next time. Thank you.