What to do when there’s too much code to review  🔗 **Source:** [What to do when there’s too much code to review](https://www.youtube.com/watch?v=0dbMKGS7uWk) > [!note]+ Document Notes > tags: [code-review, ai-assisted-development, engineering-leadership, developer-productivity, software-quality] > author: [MISSING] > source: Lead Dev > reference: https://www.youtube.com/watch?v=0dbMKGS7uWk > date created: {{date:YYYY-MM-DD}} {{time:HH:mm}} > > # 🏷️ Title: What to do when there's too much code to review > <!-- deep-read:sections start --> ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ## 📝 Summary - **What happened**: AI-powered development is dramatically increasing code volume, creating review bottlenecks for senior engineers - **Why it matters**: Teams need new processes to maintain code quality while leveraging AI productivity gains - **Key stakeholders**: Staff engineers, engineering managers, and development teams adopting AI tools - **Risks and tradeoffs**: Quality vs velocity; human oversight vs automation; building wrong things faster - **Bottom line**: Success requires treating code review as a systems problem, shifting feedback left, and implementing intelligent guardrails ## 📖 Main Content The panel addresses a critical challenge facing engineering teams: how to manage exponentially increasing code volumes from AI-assisted development while maintaining quality standards. ### Core Problems Identified **Volume Overload**: AI enables developers to produce significantly more code, but review capacity remains constant. Senior engineers become bottlenecks as they context-switch between numerous PRs. **Quality vs Speed Tension**: Teams risk building "the wrong thing" faster because conversations that previously happened during slower development cycles are being skipped. **Systems vs Tooling**: Many teams focus on "model shopping" rather than addressing the systemic workflow changes needed for AI-augmented development. ### Key Solutions Framework **Shift Feedback Left**: Move architectural and design discussions to the planning phase rather than post-implementation code review. This provides higher leverage for senior engineer feedback. **Guardrails Strategy**: Implement layered quality controls including deterministic tools (linters, static analysis) combined with AI review, rather than relying solely on human review. **Context Engineering**: Maintain repository-specific knowledge bases and policies that carry forward between reviews to improve AI review quality. **Continuous Improvement Culture**: Build team muscle for systematically identifying review pain points and pushing them into automated checks. ## 🗒️ Notes #### ⏱️ [00:00:00](https://www.youtube.com/watch?v=0dbMKGS7uWk&t=0) Panel Introduction - Delia Havens (SVP Engineering at LUSK) moderating - James (Staff Engineer at Tilt) - fintech serving underserved lending market - Hasset (Enterprise Engineering Lead at Code Rabbit) - AI code review tool - Pete Hodgson (Head of Technology at Tribe AI) - AI native transformation consultant #### ⏱️ [00:05:12](https://www.youtube.com/watch?v=0dbMKGS7uWk&t=312) Core Problems with Code Review - **James**: Keeping up with volume while maintaining focus at higher velocities - **Pete**: Too much feedback happening post-coding vs during planning phase - **Hasset**: Senior engineers becoming bottlenecks due to context switching across multiple PRs #### ⏱️ [00:07:30](https://www.youtube.com/watch?v=0dbMKGS7uWk&t=450) Bottleneck Shift to Verification - **Pete**: Bad pattern is "agentic meat puppet" - junior dev takes AI output and passes to senior for review - Good pattern: Upfront collaboration with senior engineers before writing code - Focus on getting feedback on design decisions before implementation - Architecture discussions and RFCs remain critical for describing intent #### ⏱️ [00:12:15](https://www.youtube.com/watch?v=0dbMKGS7uWk&t=735) Successful Team Patterns - **Hasset**: Winning teams treat this as systems problem, not just model selection - Collect right context, do proper configuration, plan before coding - **Pete**: Teams with continuous improvement culture systematically push review issues into automated checks #### ⏱️ [00:15:45](https://www.youtube.com/watch?v=0dbMKGS7uWk&t=945) Building Wrong Things Faster - **James**: With 10x development speed, cross-cutting conversations aren't happening - Results in siloed code that doesn't align with broader initiatives - Need to maintain architectural discussions despite faster implementation #### ⏱️ [00:18:20](https://www.youtube.com/watch?v=0dbMKGS7uWk&t=1100) Guardrails Implementation - **James**: Evolved from custom CLI tool to commercial solutions - Focus on refined code owners files for proper reviewer assignment - QA notes checker to ensure testing documentation - System awareness beyond repository (test coverage, integration impacts) #### ⏱️ [00:24:30](https://www.youtube.com/watch?v=0dbMKGS7uWk&t=1470) Feedback Loop Optimization - **Pete**: Push traditional outer loop feedback into inner loops - Teach agents to run linters and tests autonomously - ESLint rules often better than LLM feedback - deterministic, fast, cheap - Agents good at writing custom ESLint rules for specific patterns #### ⏱️ [00:27:45](https://www.youtube.com/watch?v=0dbMKGS7uWk&t=1665) Centralized Guardrails Challenge - **Hasset**: Difficulty centralizing guardrails when they run on developer machines - Need centralized enforcement at PR level, not just local pre-commit hooks - **James**: Seen issues with overly aggressive linting affecting all open files #### ⏱️ [00:31:10](https://www.youtube.com/watch?v=0dbMKGS7uWk&t=1870) Avoiding Nitpick Reviews - **Hasset**: Look for sufficient evidence for automatic review with safety guardrails - Mix deterministic tools with AI-based review - Continuous verification with repo-specific knowledge base - **James**: Regular cleanup of unused/ignored review rules based on developer response patterns #### ⏱️ [00:35:50](https://www.youtube.com/watch?v=0dbMKGS7uWk&t=2150) Productivity Metrics - **Hasset**: Code Rabbit tracks 64-65% acceptance rate (up from low 20s initially) - Avoid time-to-merge as primary metric - developers should control merge timing - **James**: Researching agentic PR metrics - PRs merged with no extra commits, agent vs human commit ratios #### ⏱️ [00:40:15](https://www.youtube.com/watch?v=0dbMKGS7uWk&t=2415) Feature Maturity and Review Standards - **Hasset**: Code changes should always be taken seriously regardless of feature flags - **James**: Devil's advocate - willing to ship less tested code for feature-flagged experiments at 1% rollout - **Pete**: Distinguish "building to learn" vs "building to scale" - different quality standards appropriate ## 📚 References - [Code Rabbit](https://coderabbit.ai/) - AI code review tool - [Lead Dev](https://leaddev.com/) - Engineering leadership community - [Tilt](https://www.tilt.com/) - Fintech lending platform - [Tribe AI](https://tribe.ai/) - AI transformation consulting ## 🗣️ Paraphrase Engineering teams are experiencing a productivity paradox with AI-assisted development: while individual developers can produce significantly more code, the traditional code review process is becoming a major bottleneck. Senior engineers find themselves overwhelmed by the volume of pull requests requiring review, leading to context switching fatigue and potential quality compromises. The solution isn't simply to review faster or add more reviewers, but to fundamentally restructure the development workflow. Successful teams are shifting critical feedback conversations earlier in the development cycle, implementing intelligent automation for routine checks, and building systems that capture and apply institutional knowledge consistently. The key insight is treating this as a systems engineering problem rather than a tooling problem. Teams need to identify which types of feedback can be automated, which require human judgment, and how to create feedback loops that provide value without creating friction. This includes everything from advanced linting rules to AI-powered contextual reviews to better architectural planning processes. ## 📰 Newsletter 🚀 **AI Code Review Crisis**: Engineering teams are drowning in pull requests as AI boosts developer productivity 10x, but review capacity stays flat. Three engineering leaders share battle-tested strategies: shift architectural feedback left to planning phase, layer smart guardrails (ESLint + AI + human oversight), and treat this as a systems problem, not a tooling one. Key insight: distinguish "building to learn" vs "building to scale" - different quality standards for each. Teams seeing 65% AI review acceptance rates (up from 20%) by focusing on context engineering and continuous improvement culture. 🛠️ **Quick Wins**: Set up bi-weekly "continuous improvement hours" to systematically push review pain points into automation. Your coding agents can write custom ESLint rules better than you can! [Read the full panel discussion](https://www.youtube.com/watch?v=0dbMKGS7uWk) ## ✅ Action Items #### DevRel - Document AI-assisted development best practices for community education - Create workshops on "feedback loop optimization" for engineering teams - Develop templates for architectural planning discussions pre-implementation #### Community - Share measurement frameworks for AI developer productivity - Create peer learning groups around code review process evolution - Document case studies of successful AI integration workflows #### WordPress - **Core Performance**: Establish AI-assisted performance review guidelines for Core contributions - **Community**: Update contribution guidelines for AI-augmented development workflows - **Training**: Develop curriculum on modern code review practices with AI tools ## 🧪 Hypotheses - Teams treating AI code review as a systems problem will achieve higher quality outcomes than those focused solely on tool selection - Shifting architectural feedback to pre-implementation phases will reduce review cycle time by 40% - Combining deterministic tools (linters) with AI review will achieve higher acceptance rates than AI-only approaches - Teams with regular continuous improvement processes will adapt to AI-augmented workflows faster than ad-hoc teams - Feature-flagged experimental code can maintain lower quality standards without impacting production systems ### 📊 Methodology - Survey engineering teams on pre/post AI adoption review metrics (cycle time, acceptance rates, defect rates) - A/B test feedback timing: architectural discussions pre-implementation vs post-implementation - Measure code quality metrics across different guardrail configurations - Track team satisfaction and productivity across different AI integration approaches ### ❓ Questions - How do you maintain code quality culture when AI can generate syntactically correct but architecturally poor solutions? - What's the optimal ratio of human to AI review for different types of code changes? - How should engineering career development change when junior developers can produce senior-level code volume? - What governance models work best for teams mixing human and AI code generation? ### 🔮 Future Research - Long-term impact of AI-augmented development on technical debt accumulation - Effectiveness of different AI models for domain-specific code review (fintech, healthcare, etc.) - Evolution of engineering roles and responsibilities in AI-augmented teams - Cost-benefit analysis of custom vs commercial AI code review tools ### 💡 Implications - **Practice**: Engineering teams must redesign their entire SDLC, not just add AI tools to existing processes - **Policy**: Organizations need governance frameworks distinguishing experimental vs production code quality standards - **Education**: Computer science curricula should include systems thinking about development workflows, not just coding skills - **Funding**: Investment in process automation and workflow optimization may provide higher ROI than additional engineering headcount ### 🧠 Learning Styles Assessment Reading level estimate: Flesch-Kincaid Grade 12 (appropriate for engineering professionals) #### 👀 Visual learners - Present: Workflow diagrams showing feedback loop optimization, before/after process comparisons - Improve: Add architectural decision trees, code review process flowcharts, metrics dashboards #### 🔊 Aural learners - Present: Panel discussion format with multiple perspectives and conversational learning - Improve: Add audio summary of key takeaways, follow-up podcast episodes #### 🗣️ Verbal learners - Present: Clear definitions of terms like "agentic meat puppet," detailed explanations of concepts - Improve: Add glossary of AI development terminology, more analogies for complex workflow concepts #### 🧩 Kinesthetic learners - Present: Practical implementation steps, specific tool configurations - Improve: Add hands-on workshop materials, step-by-step setup guides, interactive exercises #### 🧠 Logical learners - Present: Systematic framework for guardrails, metrics-based decision making - Improve: Add decision matrices for tool selection, ROI calculation templates #### 🧑‍🤝‍🧑 Social learners - Present: Panel discussion format, community learning emphasis - Improve: Add discussion prompts for team retrospectives, peer learning group guides #### 🧘‍♀️ Solitary learners - Present: Individual reflection on team practices, self-assessment opportunities - Improve: Add personal audit checklists, individual skill development paths ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Clear heading structure, descriptive link text - Improve: Add ARIA labels for video timestamps, transcript formatting improvements #### Keyboard navigation - Present: Standard web navigation patterns - Improve: Add skip-to-content links, keyboard shortcuts for video sections #### Contrast and readability - Present: Standard text contrast ratios maintained - Improve: Ensure code examples maintain sufficient contrast, test with high contrast modes #### Media and captions - Present: Video transcript provided - Improve: Add closed captions to video, audio descriptions for visual elements ### 🌍 DEIB Assessment #### Cultural breadth - Present: Global team examples (US, Australia, Brazil, New Zealand) - Improve: Include perspectives from different regional development practices #### Inclusive language - Present: Professional terminology, defined technical jargon - Improve: Expand glossary, avoid assumptions about team structure/resources #### Representation - Present: Multiple speakers with different backgrounds and company sizes - Improve: Include more diverse perspectives on team leadership and decision-making ### ✅ Logical Validity #### Strong reasoning - **Systems thinking approach**: Panelists consistently frame the problem as workflow redesign rather than tool adoption, supported by concrete examples - **Evidence-based metrics**: Discussion of measurable outcomes (65% acceptance rates, commit ratios) provides concrete validation - **Feedback loop theory**: Well-supported argument for shifting feedback earlier in development cycle #### Fallacies - **Appeal to novelty**: Some assumption that AI tools are automatically better than traditional approaches without sufficient comparison - **Survivorship bias**: Examples focus on successful implementations without discussing common failure modes in detail #### Tone of voice Professional and collaborative, with practical problem-solving focus. Examples from real experience provide credibility. ### 🛡️ DARVO Analysis No DARVO patterns detected. Discussion maintains focus on constructive problem-solving rather than defensive responses to criticism. ## 🎯 Conclusion The AI-powered development revolution requires a fundamental rethinking of code review processes, not just adding AI tools to existing workflows. Successful teams treat this as a systems engineering challenge, implementing layered guardrails, shifting critical feedback earlier in the development cycle, and building cultures of continuous improvement. The key practical next steps for readers: establish bi-weekly continuous improvement meetings, implement intelligent automation for routine checks, and clearly distinguish between experimental and production code quality standards. Teams that master these workflow adaptations will gain sustainable competitive advantages in the AI-augmented development era. ## 🔄 Refactor Appendix — What to do when there's too much code to review ### 📚 Literature Atoms #### AI increases code volume but reduces implementation conversation > [!quote] ID:rw_1 > With AI we're noticing not so much that the code is wrong but we're actually building the wrong thing because the conversations we would have had when it took 10 times as long to build the code. **Context & use**: - Understanding how AI acceleration affects product planning and cross-team coordination **Tags**: #ai-development #product-planning #xref #### Code review bottlenecks shift from volume to context switching > [!quote] ID:rw_2 > The staff engineers the lead engineers become the bottleneck after some point because the amount of volume that they see they cannot context switch between all of these amongst all these different PRs. **Context & use**: - Identifying scaling challenges in code review processes for senior engineers **Tags**: #code-review #engineering-management #xref #### Feedback should move from post-implementation to pre-coding phase > [!quote] ID:rw_3 > Rather than that maybe less senior engineer kind of not looking for feedback at all or just taking whatever the AI is producing and giving it to the senior person. What I think the good practices is a lot more upfront collaboration. **Context & use**: - Restructuring development workflows to prevent "slop cannon" from AI-generated code **Tags**: #ai-development #code-review #xref #### Successful AI adoption requires systems thinking over model shopping > [!quote] ID:rw_4 > The teams that win are the ones looking at it as a systems problem rather than just a model shopping problem. Like I see a lot of teams just talking about, oh, we have the best model. **Context & use**: - Evaluating AI tool adoption strategies beyond just choosing the best model **Tags**: #ai-adoption #systems-thinking #xref #### Traditional tools often outperform AI for deterministic code checks > [!quote] ID:rw_5 > An ESLint rule is just so much better than an LM can ever do. It's like cheaper, faster, deterministic, all of the things. **Context & use**: - Choosing appropriate tools for different types of code quality enforcement **Tags**: #code-quality #tooling #xref #### AI code review acceptance rates indicate system effectiveness > [!quote] ID:rw_6 > We're close to on an average I would say last time I checked was 64 65% acceptance ratios across the board across all companies which is amazing. **Context & use**: - Using acceptance rates as a metric for AI code review tool performance and configuration **Tags**: #metrics #ai-tools #xref <!-- deep-read:sections end --> ## 📺 Video  ## Highlights - Occasional framing of choices as either fully automated or fully manual rather than exploring nuanced middle grounds - Note: 🔍 Fallacy: False Dilemma #fallacy/false-dilemma --- # 📝 Full Notes (Deep-Read Analysis) # 🏷️ Title: What to do when there's too much code to review ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ## 📝 Summary - AI-assisted development is creating unprecedented code review volume, overwhelming staff engineers and senior reviewers - The bottleneck is shifting from writing code to verifying and reviewing AI-generated output - Teams need systematic approaches combining upfront design collaboration, automated guardrails, and strategic feedback loops - Success requires treating code review transformation as a systems problem, not just model optimization - Key stakeholders: engineering leaders, staff engineers, product teams, and QA - Bottom line: Move critical feedback earlier in the development cycle while building robust automated quality gates ## 📖 Main Content This panel discussion addresses the growing challenge of code review volume in AI-augmented development environments. Three industry experts share practical strategies for managing increased throughput while maintaining quality. ### Core Problems Identified **Volume Overwhelm**: AI enables developers to produce significantly more code, but review capacity remains constant. Staff engineers become bottlenecks as they context-switch between numerous PRs without adequate time for meaningful review. **Quality vs. Speed Tension**: Teams face pressure to merge faster due to increased output, risking the "rubber stamp" anti-pattern where reviews lose their critical evaluation function. **Wrong Thing Built Right**: Faster implementation means conversations that previously happened during slower development cycles are being skipped, leading to well-executed but misaligned features. ### Strategic Solutions **Shift Left Feedback**: Move design and architectural discussions to the planning phase before any code is written. This provides higher leverage than post-implementation reviews and prevents the "agentic meat puppet" pattern where developers blindly implement AI suggestions. **Systems Thinking Approach**: Treat code review transformation as a comprehensive systems problem rather than simply adopting better models. This includes context collection, configuration management, and systematic bottleneck identification. **Layered Quality Gates**: Combine deterministic tools (linters, static analysis) with AI-powered reviews and human oversight. Push automated checks into inner feedback loops while reserving human review for higher-level design decisions. ### Practical Implementation **Continuous Improvement Culture**: Establish regular retrospectives to identify recurring issues and systematically eliminate them through automation rather than repeated human intervention. **Guardrail Standardization**: Create consistent, centralized quality checks that run automatically rather than relying on developer-local tooling that can be bypassed. **Metric-Driven Optimization**: Track acceptance rates for AI reviews and code review churn to identify when systems are working effectively versus creating friction. ## 🗒️ Notes #### ⏱️ [00:03:12](https://www.youtube.com/watch?v=0dbMKGS7uWk?t=192) Problem Definition - Code Review Volume - James (Tilt): "Keeping up with the code review volume while maintaining the same focus that we did at lower velocity levels" - Hassan (Code Rabbit): Staff engineers become bottlenecks due to excessive context switching between PRs - Traditional review capacity doesn't scale with AI-generated output volume #### ⏱️ [00:05:45](https://www.youtube.com/watch?v=0dbMKGS7uWk?t=345) Bottleneck Shift - Verification Over Creation - Pete (Tribe AI): Bottleneck moving from code reviews to verifying AI output - "Agentic meat puppet" anti-pattern: developers taking AI output and passing it directly to senior reviewers - Need to shift feedback earlier in the development cycle, not later #### ⏱️ [00:08:30](https://www.youtube.com/watch?v=0dbMKGS7uWk?t=510) Systems Thinking Approach - Hassan: Successful teams treat this as systems problem, not just model shopping - Planning, context collection, and configuration matter more than having "the best model" - Teams that struggle focus only on model selection without systematic approach #### ⏱️ [00:11:20](https://www.youtube.com/watch?v=0dbMKGS7uWk?t=680) Building Wrong Things Fast - James: "We're noticing not so much that the code is wrong but we're building the wrong thing because the conversations we would have had when it took 10 times as long to build the code... they're just not happening" - Siloed code reproduction instead of cross-cutting initiatives #### ⏱️ [00:14:15](https://www.youtube.com/watch?v=0dbMKGS7uWk?t=855) Practical Guardrails Implementation - James: Refined code owners files, AI as one reviewer (not all), QA notes enforcement - Building systems with awareness outside repository (test coverage, integration tests) - Abandoned in-house tools in favor of commercial solutions for core review functionality #### ⏱️ [00:18:45](https://www.youtube.com/watch?v=0dbMKGS7uWk?t=1125) Feedback Loop Optimization - Pete: Push guardrails into inner loops - developer commits, not PR creation - Teach agents to run linters and tests before human sees code - ESLint rules > LLM guidance for deterministic checks #### ⏱️ [00:22:10](https://www.youtube.com/watch?v=0dbMKGS7uWk?t=1330) Centralized vs Local Guardrails - Hassan: Battle shifts to PR level when local guardrails can be bypassed - Need centralized enforcement, not just developer machine policies - James: Investment in guardrails justified by AI velocity gains #### ⏱️ [00:26:30](https://www.youtube.com/watch?v=0dbMKGS7uWk?t=1590) Avoiding Nitpick Anti-Pattern - Focus on machine-checkable policies and continuous verification - Repo-specific knowledge and context carryover between reviews - Mix of deterministic and AI-based feedback #### ⏱️ [00:29:45](https://www.youtube.com/watch?v=0dbMKGS7uWk?t=1785) Cleaning Up Codebase Patterns - James: Bi-weekly review of AI feedback - remove unused rules, identify recurring issues - Use AI agents to clean up inconsistent patterns across codebase - Address root causes (mixed patterns) rather than symptoms (recurring feedback) #### ⏱️ [00:32:20](https://www.youtube.com/watch?v=0dbMKGS7uWk?t=1940) Metrics and Measurement - Hassan: 64-65% acceptance rate for AI reviews (up from low 20s) - Avoid time-to-merge as primary metric - can incentivize premature merging - James: Measuring PRs merged with no extra commits vs human/agent iterations #### ⏱️ [00:37:15](https://www.youtube.com/watch?v=0dbMKGS7uWk?t=2235) Closing Feedback Loops - Pete: Bi-weekly continuous improvement meetings to review patterns - James: Partner with product peers to build business cases for technical debt - Hassan: Convert recurring patterns to linting rules or static analysis #### ⏱️ [00:40:30](https://www.youtube.com/watch?v=0dbMKGS7uWk?t=2430) Feature Maturity and Quality Standards - Hassan: Every code change should be treated with utmost importance - James: Different tolerance for feature-flagged experiments vs production features - Pete: Build-to-learn vs build-to-scale distinction with explicit quality standards ## 📚 References - [Steven Tobe article on 10 months of using AI with .NET runtime](reference needed) - Code Rabbit AI code review tool - Tilt fintech platform - Tribe AI consulting - Lead Dev community and survey data ## 🗣️ Paraphrase AI development tools have created a productivity paradox: developers can write code faster than ever, but the human capacity for meaningful code review hasn't scaled proportionally. This panel reveals that the solution isn't just better AI reviewers, but a systematic transformation of how teams approach quality assurance. The most successful teams are shifting critical conversations earlier in the development process, using AI to handle routine checks, and building cultures of continuous improvement. They treat code review bottlenecks as system design problems, not just resource constraints, and invest in automation that prevents issues rather than just catching them. The key insight is recognizing when you're building to learn versus building to scale, and adjusting quality standards accordingly while maintaining rigorous oversight for production-bound code. ## 📰 Newsletter **AI Code Reviews: From Bottleneck to Breakthrough** The WordPress community has always valued thorough code review, but AI-assisted development is changing the game entirely. If your team is shipping code faster but drowning in review backlogs, this Lead Dev panel has actionable solutions. Key takeaways for WordPress teams: - **Shift left**: Have architecture discussions before coding, not during PR review - **Automate the automatable**: Use PHPCS, PHPStan, and custom rules before human eyes see the code - **Measure what matters**: Track review acceptance rates and churn, not just merge speed - **Clean as you go**: Use AI to standardize patterns across your codebase Perfect timing as WordPress explores AI integration in core. The principles here apply whether you're reviewing Gutenberg contributions or custom plugin development. Focus on systems thinking, not just better tools. [Link to full panel discussion](https://www.youtube.com/watch?v=0dbMKGS7uWk) ## ✅ Action Items #### DevRel - Create templates for AI-assisted code review workflows and share with developer community - Develop workshop content on "Building AI-Safe Development Processes" - Document case studies from teams successfully scaling code review with AI tools - Host office hours focused on code review optimization for AI-augmented teams #### OSPO - Establish guidelines for AI code review in open source contribution workflows - Create scanning tools to identify when AI-generated code needs additional human oversight - Develop policies for acceptable AI assistance levels in different project contexts - Set up training programs for maintainers on managing increased contribution volume #### Community - Implement mentorship programs pairing senior reviewers with AI-assisted junior developers - Create "code review clinic" sessions for maintainers overwhelmed by volume - Develop templates for constructive AI code review feedback - Establish cross-project incident response for when AI-generated code introduces systemic issues #### WordPress - **Core**: Integrate AI-aware review checklist into contributor handbook, establish guardrails for AI-assisted patches - **Core Performance**: Set up automated performance regression testing before human review - **Core AI**: Create specialized review processes for AI-related contributions with appropriate expertise requirements - **Documentation**: Update code review guidelines to address AI-assisted development patterns - **CLI**: Implement pre-commit hooks that run full test suites before review requests - **Plugins/Themes**: Establish tiered review processes based on AI assistance level and complexity ### FAIR - Develop governance frameworks for AI code review transparency in funding decisions - Create standardized metrics for measuring AI-assisted development productivity - Establish policies ensuring human oversight remains available for critical infrastructure code ## 🧪 Hypotheses - Teams that implement systematic feedback loop optimization will see 40% reduction in code review cycle time within 90 days - Organizations treating AI code review as a systems problem rather than a tooling problem will achieve higher developer satisfaction scores - Moving architectural discussions to pre-coding phases will reduce post-implementation rework by 60% - Automated guardrail implementation will correlate with increased AI code acceptance rates - Teams measuring acceptance rates rather than merge velocity will maintain higher code quality standards - Feature-flagged experiments with relaxed review standards will achieve faster time-to-market without compromising production quality ## 📊 Methodology **Hypothesis Testing Approaches:** - A/B testing of pre-coding collaboration vs traditional post-coding review workflows - Longitudinal studies tracking code review metrics before and after AI tool adoption - Case study analysis comparing teams using systematic vs ad-hoc AI review approaches - Survey research on developer satisfaction across different review automation levels - Performance analysis of codebases with different AI assistance and review patterns - Cost-benefit analysis of commercial vs in-house AI review tool investments ## ❓ Questions - How do we maintain code ownership and knowledge transfer when AI handles routine implementation? - What's the optimal ratio of automated to human review for different types of code changes? - How should review standards differ between experimental features and production-critical code? - What metrics best predict when AI-assisted development is creating technical debt? - How can distributed teams maintain consistent review quality with varying AI tool access? ## 🔮 Future Research - Long-term impact of AI-assisted development on codebase maintainability - Optimal training approaches for reviewers working with AI-generated code - Evolution of code quality metrics in AI-augmented development environments - Cross-industry analysis of successful AI code review implementation patterns - Impact of AI code review on junior developer skill development and career progression ## 💡 Implications **Practice**: Development teams need to restructure workflows around AI capabilities, moving from review-heavy to prevention-heavy quality assurance. This requires significant process redesign and cultural change. **Policy**: Organizations must establish clear guidelines for AI assistance levels, review requirements, and quality standards that account for both experimental and production code paths. **Security**: AI-generated code at scale requires new approaches to security review, including automated scanning for common AI-introduced vulnerabilities and patterns. **Education**: Developer education programs need updating to include AI collaboration skills, systematic thinking about code quality, and advanced automation setup techniques. **Funding**: Investment priorities should shift toward systematic improvements and process optimization rather than just additional tooling, with ROI measured through productivity and quality metrics rather than simple output volume. ## 🧠 Learning Styles Assessment Reading level estimate: Flesch-Kincaid Grade 12.8 - appropriate for technical professional audience #### 👀 Visual learners - Present: Panelist roles and company contexts clearly labeled - Improve: Add workflow diagrams showing feedback loop optimization, visual metrics dashboards, before/after process comparisons #### 🔊 Aural learners - Present: Full transcript with timestamped sections - Improve: Audio summary highlighting key takeaways, podcast-style discussion segments #### 🗣️ Verbal learners - Present: Clear definitions of "agentic meat puppet," "shift left," "systems thinking" - Improve: Glossary of AI development terms, analogies comparing traditional vs AI-assisted workflows #### 🧩 Kinesthetic learners - Present: Specific implementation steps, actionable takeaways - Improve: Step-by-step setup guides for guardrails, hands-on workshops for process implementation #### 🧠 Logical learners - Present: Clear problem-solution structure, metrics and measurement approaches - Improve: Decision trees for review approaches, causal diagrams linking practices to outcomes, ROI calculations #### 🧑‍🤝‍🧑 Social learners - Present: Panel discussion format, community Slack integration - Improve: Peer review rubrics, collaborative improvement templates, cross-team sharing frameworks #### 🧘‍♀️ Solitary learners - Present: Individual reflection prompts in action items - Improve: Self-assessment checklists for current practices, personal improvement tracking sheets #### 🌿 Naturalistic learners - Present: Evolutionary perspective on development practice changes - Improve: Ecosystem thinking about development toolchain, sustainable development velocity concepts ## 📊 Accessibility Assessment #### Screen reader semantics - Present: Clear heading hierarchy, timestamped navigation - Improve: Alt text for any future diagrams, ARIA labels for complex section relationships #### Keyboard navigation - Present: Standard document structure, logical reading order - Improve: Skip links for long transcript sections, keyboard shortcuts for timestamp navigation #### Contrast and readability - Present: Standard text contrast meets WCAG AA - Improve: Ensure code examples maintain contrast, test with high contrast mode #### Text resizing and reflow - Present: Responsive text structure - Improve: Test at 200% zoom, ensure timestamp links remain functional #### Media and captions - Present: Full transcript provided - Improve: Video captions accuracy review, downloadable transcript format #### Forms and interactions - Present: Simple link interactions - Improve: Clear link purposes, accessible video player controls ## 🌍 DEIB Assessment #### Cultural breadth - Present: Global perspectives from US, New Zealand, Australia, Brazil contexts - Improve: Include perspectives from different cultural approaches to code review, translation considerations #### Inclusive language - Present: Professional technical discourse, defined jargon - Improve: Avoid assumptions about team structures, expand acronym definitions #### Neurodiversity support - Present: Structured format with clear sections - Improve: Consistent formatting patterns, predictable navigation, summary boxes #### Representation - Present: Technical leadership perspectives - Improve: Include various seniority levels, different role perspectives beyond engineering #### Psychological safety - Present: Constructive discussion of common challenges - Improve: Frame struggles as normal learning process, provide multiple solution approaches ## ✅ Logical Validity #### Strong reasoning - **Systems thinking approach**: Deductive reasoning from first principles about bottlenecks and capacity constraints is sound - **Feedback loop optimization**: Causal logic connecting earlier feedback to better outcomes is well-supported - **Metric-driven improvement**: Inductive approach from acceptance rates to quality outcomes shows good empirical grounding #### Fallacies - **Appeal to novelty**: Some assumption that AI tools are necessarily better without considering potential downsides or context dependencies - **False dilemma**: Occasional framing of choices as either fully automated or fully manual rather than exploring nuanced middle grounds - **Survivorship bias**: Focus on successful AI adoption without adequate discussion of failure cases or teams struggling with transition #### Tone of voice - **Collaborative and practical**: Panelists share struggles openly and offer concrete solutions - **Evidence-based**: Focus on measurable outcomes and systematic approaches rather than opinion - **Cautiously optimistic**: Acknowledging both benefits and challenges of AI-assisted development ## 🛡️ DARVO Analysis No DARVO patterns detected. The discussion maintains a collaborative, problem-solving focus without defensiveness, attacks, or victim-reversal dynamics. Panelists acknowledge challenges openly and take responsibility for finding solutions. ## 🎯 Conclusion The fundamental shift in AI-assisted development isn't just about handling more code—it's about reimagining how teams collaborate, when they seek feedback, and what humans should focus on versus what machines can handle. The most successful teams are those that treat this as a comprehensive systems redesign opportunity rather than simply a tooling upgrade. **Practical next steps for readers:** 1. Audit your current code review bottlenecks and map them to the solutions discussed 2. Implement one systematic improvement this month (bi-weekly review meetings, automated guardrails, or upfront design discussions) 3. Establish metrics that focus on quality outcomes rather than speed metrics 4. Begin conversations about build-to-learn versus build-to-scale standards for your team The goal isn't to eliminate human judgment from code review, but to ensure that human attention is focused on the highest-value decisions while automation handles routine quality checks. --- # 📄 Full Transcript *Auto-generated from YouTube captions · timestamps roughly every 30 seconds · [watch on YouTube](https://www.youtube.com/watch?v=0dbMKGS7uWk)* **[00:12]** All right, everyone. Thank you so much for joining us today. Uh, I'm very very excited about our topic and chatting with our panelists here. So, thank you as well for joining us. This is a lead dev panel discussion created in partnership with Code Rabbit. uh we will have about 45 minutes after it we will head over to the Slack channel that you will get a link to. It's a great way to continue the conversation. It's an awesome community of edge leaders. So I really encourage you all to join um and **[00:45]** sort of continue with questions or discussions that we probably will not have time to fit into this 45 minute slot. Um, I will keep an eye on the Q&A in Zoom, but again, if we happen to miss your question or not have time for it, don't hesitate to drop it in Slack and we will do our best to uh make sure to, you know, one of the panelists or myself uh get back with an answer or a link or something hopefully useful uh to you uh and what you're asking about. So, let's **[01:16]** get started with some introductions. Uh my name is Delia Havens. I'm the SVP of engineering at LUSK and I'm super excited to moderate and and sort of help shine a light on all of these wonderful panelists who have a ton to share with us. Uh we are very lucky that they bring in all the different perspective and I'll let them introduce themselves and give you some insight into their work. Uh James, we'll start with you. >> Sure. Thanks for having me. My name is James. I'm a staff engineer at Tilt. **[01:48]** We're a fintech company with employees ranging anywhere from the US, New Zealand, Australia, Brazil. Uh so we cover most of the globe and we do have a current uh roles open. Um what we do here is we essentially focus on those that are underserved by traditional lending models and so we use different AI algorithms instead of credit scores to lend to people who generally don't have access to credit. >> Very cool. Hit would you like to go next? >> Yeah. Um Hass nice to meet you all and thank you for having me on the panel. Um **[02:20]** I'm with Code Rabbit. I lead the enterprise engineering team here. Um code rabbit is a AI code review tool. U gathers a lot of context around your code and gives you very relevant code reviews. Uh it's a it's a tool that um has been changing a lot of enterprise workflows and happy to talk about it today. We also have a job description somewhere in in the links list. Please check out we have jobs for many **[02:50]** different roles. >> All right, Pete, take us home. >> Hi, I'm Pete Hodgson. Um, I work at a consulting firm called Tribe AI. I'm the head of technology for our AI native transformation practice. Um, so what that means is we're focused on kind of helping large enterprises rewire their workflows for Aentic AI. Um, and my particular focus is engineering organizations, helping large organizations kind of reimagine how we **[03:21]** build software, uh, rewire the SDLC in this kind of wild new world of AI. Um, and um, we're also hiring. Um, we are hiring a lot. So would love to would love to hear from folks. There will be a link. Um and I'm I'm particularly excited about this panel because like you know one of the first things that um engineering leaders talk to me about uh as you know the step one is figuring out how to use these get started using these these tools and step two is what do we do with all the code we're now uh we're **[03:51]** now needing to review. So super excited to dig into the topic. >> Very cool. All right let's set the stage. Uh if you're leading an engineering team right now, I'm pretty sure this topic is very very relevant to you. It's amazing how much code we're able to ship. Uh and that's probably still leaves us with the same number of reviewers, the same engineers on your team, uh trying to keep up with it all. So like at least for for me and the enterprise products and code bases that I work in, we're talking about **[04:22]** brownfield here. So like hold your objections green field. You can yolo your code if you want to, but really there is intentionality in the care not only of the implementation that you're serving your customers with, but the importance of quality and security and so on. And really what I want to do with this conversation is make it as practical as possible. um trying to plug in as many takeaways for each of you attending here to bring back to your team and hopefully continue to accelerate and find momentum and again **[04:54]** quality um with the work that you're doing with your team. So, we're going to cover specifically code reviews and how that's changing um how where to put the guard rails and and how to handle the amount of code that's coming to your team. So, let's go ahead and dig in. So I to to kick us off um maybe for the panelists here, I'd love to hear from each of you one s sentence or something, you know, quick on the single biggest thing that's broken about code reviews **[05:24]** right now. All right, James, we'll start with you. >> I think the biggest thing for us is keeping up with the code review volume while also maintaining the same v focus that we did at at lower velocity levels. >> Pete, do you want to go next? I think the thing that's broken right now is people are doing way too much stuff post coding that they could be doing when they're planning. So giving feedback after the whole thing is built versus iterating on the plan. **[05:55]** >> Let's put a pin in that. I have a question for you after Hasset. What about you? >> So yeah, I I'd like to echo uh what James said and the volume of code is increasing. Not just that the number of PRs is going up. Uh it's especially the the the staff engineers the the lead engineers become the bottleneck after some point because the amount of volume that they see they cannot effectively context switch between all of these amongst all these different PRs to to **[06:28]** actually give a meaningful review. Right. Uh so I think that is where we see the the biggest pain. >> Yeah. And it's not a rubber stamp like I don't I think we all agree that's not the value of the review which is why it's still a very critical point. All right Pete we'll come to you and I want to start with this question. Uh you've written about the bottleneck shifting from code reviews to verifying it. Could you walk us through what that actually looks like day-to-day for developers right now? And what's a new skill that **[06:59]** engineers need um that they didn't need maybe a year ago? I will say. >> Yeah. Um, so I think like I think that that there's there's kind of like these these kind of good and bad uh versions of of code review or these good and bad versions of like how people are using aentic coding tools. So like in the in the the bad version, you have an engineer who's kind of what I would call an agentic meat puppet or a reverse **[07:30]** centaur where they're they're telling they're taking the the the ticket, the requirements, giving it to the AI. The AI is doing the work. They're maybe doing some minimal kind of um answering some questions and then they're taking the output and they're just giving it to someone else, probably a more senior person to review. And that's just kind of creating what one of my colleagues calls a slot cannon. um and you know the staff engineer is like uh is is is suffering at the end of it. So I think that's like the the the bad version of the world. I think what the the good version of the world is is that shift in **[08:03]** where you're looking for feedback from those staff engineers and who's giving that feedback. So rather than that maybe less senior engineer kind of not looking for feedback at all or just taking um whatever the AI is producing and giving it to the senior person. What um I think the good practices and what people should be learning to do is a lot more upfront collaboration with that person who you're looking for feedback from before you start writing **[08:33]** the code. And this is something that like high performing teams have been doing for, you know, decades. Um, like very very hightouch feedback. Um, having kind of kick off kickoff design conversations right down at the level of individual stories and kind of having low structure and rigor around that. Um, and and now I think just everyone has to learn how to do that because you just can't we can't just keep firing uh all of this. Like if you think about what are we looking for from code review, there's a ton of at **[09:05]** the end of the day we're looking for feedback on decisions. That's the value of code review. Um and some of those decisions are implementation decisions and you can only give that feedback after implementation has been done. The most interesting decisions are the design decisions, the high level architectural decisions and you should be you get way more leverage looking at those decisions and giving feedback and iterating on those decisions before you start writing any code. you can iterate on it with AI in the loop as well. Um, but yeah, if **[09:35]** you've got a staff engineer who's who's kind of suffering under all of this slop, I think the the key is to figure out how to move that feedback to the beginning of the cycle, not not to the end. >> I I I think I agree, Pete. AI can generate a lot of syntactically correct stuff uh and in large volumes. But we were we were doing architecture discussions, product design, RFC's before that as well. And that is I think so valuable because uh that is where you would describe the intent behind the **[10:06]** change uh rather than >> just the syntax of what is being changed. >> Yep. >> Great. Uh all right. All right. Well, H, I'll go to you for the next question, especially since you uh given your customer base and and the um different number of teams that that you work with. Could you share some about the patterns that you see on the teams and what do they get right versus something that **[10:36]** they're still drowning in? Uh whether that be task or format or whatnot. Uh and are there common things that successful teams are doing first? >> Yes. uh I especially I I'll speak from my perspective not necessarily the whole of code rabbit but um I talk to a lot of these enterprise teams and um organizations and there's a common theme that comes out of it and especially the teams that win are the ones looking at **[11:07]** it as a as a as a systems problem rather than just a model shopping problem. Like I see a lot of teams just talking about, oh, we have the best model. Let's just write a quick tool to do a review and we'll be done with it. Um, it's often not that easy, right? Like um so and and Pete kind of mentioned it as well. Um so when you look at it as a as a as a systems problem then you can **[11:40]** narrow down exactly where the bottlenecks are and target that right so now now you're talking about collecting the right context you are talking about doing the right configurations you are talking about um doing a lot of planning up front before you are eventually writing the code and and doing the review. Um on the other hand there are teams that are not doing so well are not doing that and that's that's something that we see a lot in the field as well. **[12:13]** >> I think that like um that framing of of it being a systems problem I think is a is like a really good way of thinking about it. And I think this this when I see teams that that are that are happy and like thriving in this new world, it's teams that have that have got like a really good muscle around kind of continuous improvement and like when there's a when there's something that wasn't caught in code review or like something that showed up in code review, it's like what why am I reviewing this? This is terrible. rather than just kind of saying like go back and fix it, say how do we lay what do we layer into our **[12:46]** system to catch this class of error? Like how do we systematically fix this? And a lot of times that's um kind of pushing some of the responsibility down to to like something a machine can do rather than a human can do. So that might be an LLM like an adversarial review or something like that. It can also be like old-fashioned old-fashioned like linting tools, semantic analysis, uh running some tests before you show it to a human, all these kind of things. But like I think the details of what you **[13:18]** do is less important than than building like a a like a real muscle and culture in your teams to just be constantly saying, "How do we make this more efficient?" Uh and just kind of like chipping away at it over time. >> Yeah, that's that's actually interesting. We we just had an on-site uh in Phoenix with our our global team recently and one of the the common things that came up is with AI we're noticing not so much that the code is wrong but we're actually building the wrong thing because the conversations we would have had when it took 10 times as **[13:48]** long to build the code. >> Yeah. >> They're just not happening. And so you end up with a lot of siloed code that's reproduced and doesn't quite do what >> a cross pod cutting initiative would do. And so I I will say transparently like we're struggling with that as a team and it's something we've talked about a lot on how we address it. So good to hear that we're we're not the only ones struggling with that and and hopefully we'll address it together as a community. >> Yeah, I know. I I think between the panelists and the participants, we have a lot of things that we're working through that are very common. So great that that like you're sharing it and and **[14:20]** definitely want to sort of continue to to peel, you know, pull from there. So, so maybe James, I I want to ask you this because um I I've been looking at the lead dev survey and and some of the things that are really interesting um is is sort of some of the friction and sort of putting guard rails that work or how do you actually achieve that higher level of um multiplier value or however you want to you want to bridge it. I mean ultimately AI is augmenting our **[14:50]** developers. We want to make sure that the output is not again like slop or just volume where I don't think anyone here is you know measuring lines of code. I hope we really need to lean on the value. So given your work um could you share maybe some details about that and and what are some of these guard rails or like what what takes you beyond just you know your typical um when it comes to effectiveness with code reviews. Yeah, we we started pretty early on the **[15:20]** AI code review process. We were essentially using um a CLI tool within an Azure pipeline, experimenting with different rules and models and that probably got us 90% of the way there, it was a huge step forward at that point in time. And then as that's baked and then the commercial tools have gotten significantly better. Uh, one of the things that I've really started to focus on is like number one, don't reproduce a commercial tool. Kind of, you know, just because you can write it with AI, going back to my last point, doesn't mean you should. And I I do think there's like an overconfidence factor when now I can **[15:51]** build this in-house rather than buy something that somebody's spending an entire team's worth of effort on. >> So, we've kind of abandoned our in-house PR review tool. Um, but we are focusing on other parts of the life cycle. Couple examples are we've really refined our code owners file to make sure that the most knowledgeable people are doing the reviews and as team shift responsibilities keeping that updated that's been huge uh getting us towards a point where AI can be one of the reviewers but not all of the review and then the other thing we've done is we focused on the up and downstream **[16:21]** problems that developers weren't paying attention to. So, we're enforcing a QA notes checker to make sure that the developer actually puts what they tested and what they're asking QA to test because that was getting skipped a lot or with the increased velocity, QA just couldn't keep up. No different than the staff engineers keeping up with code review. And then the final thing we're doing is we're building systems that have awareness outside of the repository so that even like your load mill type test suite um that QA uses your unit test any integration test you have within the mobile app that that context **[16:53]** is brought in and we're actually telling the developer you've made changes to these files this is your coverage that you have and letting them know that sooner rather than later. So maybe they put a little bit of a pause before that PR goes from draft to ready for review. Um, or maybe you realize that it's really well covered and instead of spending eight hours of your day reproducing something that's really difficult to test, you go ahead and just move it forward and get that velocity increase. >> Peter has do you want to add anything? I mean, I I just think this this topic **[17:24]** of of guardrails in general is like the is I mean, going back to that kind of systems thinking thing, I think that's just it's it's such a it's such a key thing. And I I think >> taking taking things that you used to do in a broader in a uh further out loop and bringing it into an inner loop. I like feed I'm obsessed with feedback loops. Um but like if you think about kind of like the outer loop of code review, right? like one person writes the code or um person and an agent writes the code, another person or people review **[17:55]** it. That feedback loop is is really long. Um the feedback loop of just kind of submitting a PR maybe and running like the whatever your your continuous integration tooling is against that PR. That's still really long for a lot of people. That's like multiple days kind of feedback loop or multiple days worth of work like of the batch size, right? And maybe it's a 10-minute feedback loop or something or maybe an hour feedback loop. Um the as as we start getting faster and faster and producing more and more um I **[18:27]** think we've just got to figure out like how do we take something that where we were getting feedback on today and move that into like this inner loop of like ideally yeah inner loop of of the the developer so the developer gets the feedback every time they commit some code rather than every time they create PR or ideally we've now got a new loop like the agentic loop where I'm, you know, I give the the agent a task and it's worring and clicking and writing **[18:57]** code and running the llinter and all those kind of things and I'm off, you know, eating bowl of ice cream or whatever. Um, what can I push into that agentic loop? Like teaching the agent how to run the llinter. Like it's kind of wild to me that probably the majority of teams today they're getting feedback on things like linting or like testing only once they kind of create submit a PR or or create a draft PR when you could be getting that feedback without a human and correcting the feedback **[19:29]** without a human ever seeing it. So, I think that's like figuring out where these guard rails are and building them up more and then like pulling them further in or shifting them left is is like the name of the game. >> And and you've worked with a number of teams. What is like something practical that we can all take back to our teams that we can implement this month, quarter week? >> Yeah, I mean I I think it it like maybe sound sound like a broken record, but but building that muscle memory of like what we found something at this stage. **[19:59]** How do we either find it earlier or how do we make it so that a human doesn't have to find it and a machine can find it? So, how can we use and and I think the other thing that I I think people are doing a little too much is rubbing more AI on it when there's some traditional tools that you could use. So, um, an example I think about a lot for people who are writing kind of TypeScript or JavaScript, um, is like an an ESLint rule, um, is just so much **[20:30]** better than an LM can ever do. It's like cheaper, faster, deterministic, um, all all of the things. And turns out that Claude code or your coding agent choice is really good at re at writing custom ESLint rules. So, when you see an error, rather than saying like, "Oh, I need to add to my claude. MD or my agent.mmd or my skills file. You know, never never use an arrow function when you produce whatever. Um or or make sure you follow the rules of hooks for React. There's an ESLint rule for that. You **[21:00]** just like there's the one line you need is always run this command before you commit code. And that command is like all of the things all of those guardrails deterministic. Some of them can be adversarial, LM etc. But get like setting up that that that set of guardrails and and being you know being okay with them not all being uh kind of guidance to the LM but more sensors that are giving the feedback back to the LM. >> Uh I think I have a question here Pete **[21:31]** and James. I'm sure you guys have seen this. uh you emphasize on having guard rails, but I often find it hard to centralize that, right? Like yes, you can you can put it in your repo in a file, but it can like or a pre-commit hook or however you want it. Um but it eventually runs on a developers machine and that can be ignored or it can be elimin um and again the the battle shifts again **[22:02]** at the PR level having central guard rails I think is is the important piece versus just having guard rails >> right uh have you guys seen that in in in your professional experience? Yeah, I mean we've we've seen some funky stuff with um like we have a rule not to run any llinter u unless you modified the file already because we were getting PRs where every file that was open was adjusted and it would just blow up the git blame. >> So yeah, we we've definitely seen things like that that deteriorate the quality I **[22:33]** think of both human and AI reviews. >> Um and it's it's definitely an ongoing battle. I think one of the things that shifted is we're spending more time focusing on fixing that stuff because we know the long-term benefit with AI is there whereas before um yeah whereas before I don't think we spent as much time because the velocity wasn't there to justify it. So I I've definitely seen a lot more investment in those types of guardrails. I think it's also like it there's this is again like one of these things that the kind of concept has been around for **[23:05]** a really long time and and and what I've seen like is high performing teams have been doing this for a really long time of kind of baking. So um years ago um I I used to work at a consulting firm called forworks and we were kind of like big advocates of of what we called go scripts and this was before Golang. unfortunate namespace collision as always. Um, but it was like the idea was you walk up to a repo and rather than reading the read me and kind of, you know, do this, do that, do that, you **[23:36]** know, when to install this to set up the local database. There was a a script, you know, like could be powered nowadays by like a a make file or a just file. Back in the days, it was a rake file cuz we were Ruby nerds. Um, but yeah, the idea of that like it's all baked into the tools and it's very very very consistent. we were like obsessive about it where we would have virtual machine like no developer would be developing on their um just on their OS. They'd be developing on a VM inside of an OS and um that was that was challenging but **[24:06]** like I think now we're we're kind of a happy accident of the fact that a coding agent is like a brand new employee on boarding every time you you start a new session. >> We're we're just having to build this stuff into our system. So the idea of like having a really standard way to run the linting I think 5 years ago two years ago if if you joined if you picked a random software team there probably was the that exact issue you're describing it where like some people do it this way and oh I my my preferred way **[24:37]** of running ES is this way like oh I'm running inside of Docker and I'm using this ID and it just that's just not >> uh that doesn't work for for coding agents because you you've got to give them a consistent set of rules. So I think it's kind of forcing us to do >> um some of some of this kind of standardization which is >> I like there's some downsides for sure but I think that overall it's actually it's actually a real win that to get that consistency um and start having kind of almost like like a local CI right where you're where **[25:09]** you're kind of like got a standard way of doing all these pre-commits and all those guardrails are running running locally. >> Yeah. And and there's something about like the right level of feedback loop and guard rails. I I think that's also the important part is that like if you go write a million test cases, your cycle time shoots up. Nobody can ship anything. So like the goal is not let's make it as complex as possible. And I think that's kind of like bringing it back to the AI code review >> core topic that we're talking about. How **[25:41]** how do you see past that failure mode where it's a lot of nitpicking? The value is not quite as much, but sure, you've got a bunch of comments on your PR. I don't know if H hasn't you want to start maybe with this question? I'd love to hear your thoughts on it. >> Um, yeah. So, do you I mean, do you are you specifically asking in terms of code reviews right now or uh Yeah, I I think it's more and I I **[26:14]** apologize. I don't mean to throw you off here. I was mainly thinking um given that you're seeing how um multip like different teams are adopting your product. Um there there is that fear of now I'm getting a bunch of nitpick comments on my PR and that's really not valuable. >> So so really how do you get past that failure mode? What have you seen work really well for for yourselves at a as a team as an inch team for your customers or or you know the partners that you work closely with? **[26:45]** >> Yeah, I I Okay. Yeah. So, I think um it's it's it's less about asking the question of um how do we do reviews and review AI AI written codes uh code PRs. It's more about now kind of thinking about okay what is sufficient evidence to do this in an like automatic manner right like in and in a in an efficient manner um and of course with safety in **[27:16]** mind like you do not want to just get a rubber stamp from an AI tool to just go forward and um so having a second pair of eyes is important having uh guard rails as we've discussed in the previous topic is important Um a mix of determinism mixed with this AI based code review is also important. Um so continuous verification maybe write set up some policies to make sure that it your code always aderes to **[27:48]** it. Um you know some repo specific memory some knowledge base that is carried on from review to review or from um conversation to conversation. Uh and also like making it machine checkable right like that's again we keep going back to this but like gate is really helping when things are checkable by a machine it can be much faster and much more efficient. Um so yeah that's that's my two cents on that. >> Yeah James did you have something to **[28:19]** add? Yeah. Um I think I think for us like avoiding the nitpicky stuff is more on just code hygiene that wasn't possible in the past. And so one of the things we're doing with our PR review tool is every couple of weeks we go through and we look at the feedback that developers are thumbs uping. Uh it also tracks who's responding to the the PR re I'm sorry, who's responding to the feedback and making the requested code change and who's not. And so we're looking at the rules that aren't being triggered. And if they're not being either they are **[28:50]** being triggered or they're not, but either way, if they're not being acted upon, we go ahead and clean those up because they're not serving any value after a certain amount of time. And then the ones that we're seeing constantly pop up that are getting adjusted, what we're doing is we're actually realizing that a lot of the times the reason they're triggered so often is because our codebase has a mix of patterns. And so we're leveraging tools like cloud agents to go clean those patterns up because theoretically you write a unit test and then now you can go clean it up in a hundred spots and you know that the risk is very low. And then that kind of eliminates the nitpicky thing not just **[29:22]** with AI but with humans too. Like I've noticed a lot of our our more junior developers, they're not sitting in the staff engineering meeting or the team leads meeting and getting feedback on what's going wrong. And so there's feedback that gets missed and doesn't trickle down. And so they're constantly getting feedback on a PR, which is the first time they're hearing about a decision that was made in a room they didn't have access to. And so definitely focusing more on like background cloud agents to do the lowerhanging fruit. We're finding that stopping the nits because people aren't getting it wrong, let alone AI agents that are really good **[29:53]** at following patterns. >> I love that. I I I this is very very practical. And to your point, like AI or not, I think there's a lot of value to be gained for engineering teams um just, you know, keeping up with a practice like this. So maybe let's um I'd love your thoughts. I won't call anyone specifically, but if if you have a metric that has helped you, not necessarily metrics, but I'm very very clear that **[30:24]** they cannot be punitive. But what what I'm looking for here is like a metric that allows you to see when your setup is a bit off. Um or your teams are really sort of getting friction versus value. Is there anything that you found that works well for you or teams that you you know of? Um I can go first at least in the perspective of from the perspective of code rabbit we see um because we are code AI review tool uh a metric that we **[30:56]** look at often is acceptance rates uh we we've always seen that and and we've seen a steady increase as we uh put in more uh bring in more context into the review um with the models of course we benefit from the models getting better as well uh but as our as our harness gets better over time. Uh we we're close to on an average I would say last time I checked was 64 65% acceptance ratios across the board across all companies **[31:28]** which is which is amazing. I think we've gone up from early days of code rabbit lower 20s to now 60 65% which is I think that's a metric we are tracking very closely. Um sometimes we hear time to merge as a metric from other companies and we try to steer away from it because um there is no real control we can have over it. It's that at the end of the day the the engineers should decide when to merge it like a PI can go on for as long **[31:59]** as you want or as short as you want. I think that's kind of also a a tricky or an interesting one to measure because you could be merging fast because you're just like LGTMing or you could be merging which uh which one of my one of my co-workers used to think meant let's get that money which is >> misunderstand there is this there's this false sense of hey everyone needs to run faster because there's so much code being >> right **[32:30]** >> merge sooner so and then big things and then fix later. Uh these are just traps for the future, you know. >> Yeah. Yeah. I think that's that pre that pressure that everyone is is feeling is a is a real thing, right? Like it's a it's a real thing. And figuring out how to >> not incentivize people to to just slam the code into production is is part of the part of the challenge. >> Yeah, this is this is an interesting one. actually went deep down this rabbit hole um because we're I think every **[33:01]** company is investing a significant amount of money right now on AI and um you know especially on the fintech side we're high volume fairly low margin per volume and so we measure everything um in excruciating detail and with AI it's been difficult because we're such a new industry that we don't really know how to market or sorry how to measure the return on that investment and so I reached out to just several peers within the community um and tried to get some feedback. I've heard everything from like measuring um lines of code to token **[33:32]** burn to how long projects took to merge and they all kind of have the same downsides and push back as what the historical dorm metrics have always had. And so I guess like the more things change the more we're back to the same point of we just don't really understand how to measure developer productivity. but around the AI side and and like agentic development. Um Steven Tobe had a really good article um it was 10 months of using AI with the net runtime and one of the things I liked about that article is they broke it down to show the number of PRs that were merged that **[34:02]** were agentic with no extra commits and then they showed the number of commits that were done by the human and then the number of commits that were also done as part of the PR review by the agent. And that's probably the closest of everything that I I spent going down this rabbit hole. That's probably the most effective metric I could come up with of how often are we getting it right the first time. That means in my mind we're doing the work up front. We're we're making sure we give the agent clear instructions. We've cleaned up our code. We've got guard rails. >> And then also seeing how often the human **[34:33]** is in the loop. Um so I think we're going to lean towards measuring it like that as far as AI productivity. um but still in the early stages of setting up those dashboards. So I can't comment on like long-term what the impact of that will be. >> I've seen other teams doing the same thing of of focusing on kind of code review churn like how much of are we kind of thrashing after like the ideal obviously is every code review everything is perfect by the time it gets to code review. But yeah, I think it's a good one one to measure. Yeah, this is really like great ideas and and **[35:04]** like James, maybe I'll follow you and see your progress. Okay, we have a couple of questions in the chat. So, uh maybe let's um take them quickly. Um, the first question is, I'm curious about the practices you found most effective for closing the loop on code review feedback, mainly for turning reviewer comments into actionable improvements in repo context and development standards and also from a context engineering perspective. **[35:38]** Any thoughts? >> Feel free to go for it. I this is kind of not a not a very helpful response but maybe someone else should give a helpful response but I think it just it really depends on the team. I think that the the the most the key thing is is just try just do it. Just setting the time and space to do it like setting a you know bi-weekly Friday like calendar invite to like review like like a continuous improvement hour or **[36:09]** something where you're going to like look for that stuff and review it. Um the way that you the actual kind of mechanisms are just really really different for for for different teams. But the just the key I think particularly as a um if you're a leader in an organization is making it clear to the teams that this is important and asking them to kind of tell you in some way like how are you doing this like I want some kind of visibility like have you been having this meeting tell me how many little changes you've made in the last month figuring out a metric for that churn or something and seeing if it **[36:39]** can go down >> maybe something go ahead James >> oh no go ahead um yeah I I tend to agree with Pete um one of the things that I think is developers, we sometimes struggle with is admitting what we're not good at. And I think a lot of developer frustration around like cleaning up the code base um or you know getting these things that we're constantly flagging on reviews cleaned up is the fact that we're not product owners and we're not used to writing business cases. And that's something that we don't have a lot of exposure to. And so one of the things I've tried to do is lean in on some of my product peers that are helping on the product **[37:11]** side, but say, "Hey, I'm really struggling to get this work prioritized, and I know it's because I'm not pitching a clear vision to engineering leadership. Can you give me some tips on how I can do that?" And you'd be shocked at the difference in presenting it with some data makes versus just saying like, "This annoys me once every couple months at a meeting." >> And and maybe just to closing the the loop uh point as we discussed earlier, right? I if if a pattern emerges again and again and you're getting it wrong again and again, maybe convert it into a **[37:42]** learning like that that goes into your MD file or becomes uh a linting rule or a static analysis rule over a period of time, right? >> Um and and again be mindful because some of these things go out of fashion. So make sure you clean it up after some point if it's not relevant anymore. um that could be uh something that you could do with with your team uh like a retrospect meeting or every two weeks or something. >> Yeah. Yeah. I I like that a lot. I I **[38:14]** will say I encourage and this goes to both engineers and managers and so on. But there there is a beauty in how quickly we can iterate now. So a lot of engineering initiatives have been held back by the amount of investment and then is that investment worth the return on effort and I think now the the cost of that investment is hopefully reduced because we're able to get that leverage. So I'm encouraging my team to to kind of identify these areas that we had assumed **[38:45]** we can't touch it. It's too complicated. It's going to take us forever. I'm like yeah no but really we should revisit those decisions. Um and and I think there is something to that in in sort of this new evolution whether it be from code reviews to the practices to the design which I agree very much with the conversation we had earlier. Um I think like having that opportunity to try things that we had assumed have to be on the back burner or at the bottom of the backlog uh is very exciting. Okay. Uh **[39:15]** let's see if we can fit in this next question. uh do you have different tolerance set in reviews based on the feature maturity? So if you if um if you have this experimental feature flagged, should you have more leniency un sorry I apologize I feel like I'm butchering the question but I think I understand the spirit of it um would anyone care to comment on it? I think this burns down to your your organization's appetite for how okay you **[39:46]** are releasing something that is not very finished. Right there this there's this is not a leniency thing that you would do on a review because you should take every code change with the utmost importance. Code changes are not to be taken lightly. Eventually they are making into production. Um, but yes, you might get something wrong from the business side on a new feature flag feature and that's okay. Uh, you could iterate over that quickly. Um, >> yeah, **[40:17]** >> I'll I'll play a little bit of devil's advocate on this one. Uh, I guess we can have differing views to prove it's not scripted. Um, so I I was a part of a really cool project right before COVID that was essentially getting log books on to aircraft and we did complete unit test, integration test coverage over everything. Took it to the extreme and then ultimately the funding got cut uh once COVID happened and airlines were no longer paying their bills and that really hit home to me that I missed out on what one of the coolest projects I could have had in my career because time to market was too slow. And so now I've **[40:50]** adopted a lot more of like if it can be a feature flag that is at 1% and we can roll it out and we know how to monitor it. I'm actually willing to ship loosely tested or like really fast code with a couple of exceptions like obviously it can't be money movement or it can't be a life or life or death situation. Um, but but I will say one thing as I've gotten further in my engineering career is I am a lot more willing to test or to ship code that is not thoroughly tested to get speed to market because so many features that we've built have been the **[41:20]** wrong thing the first time. >> Yeah, I think I I really agree with that and I will kind of like build on it to say I think that what you've got to be when you're doing like an AB test or a feature flag feature, right? Like I think what you've got to be saying, am I building this? Am I building this to learn? If this is an experiment and I'm building it to learn absolutely time to market like again feedback loops like can I build a quicker iteration of learning? Can I put this out to 1%? Tweak it. Tweak it. Show it to some users. Tweak it. 100% I think **[41:53]** you should be looking at quality through a different lens. But once you've learned and you're building for scale, you this is now real code. So if you if you've kind of punted on all of the code review and then you said okay this is great. You can't just kind of like la like we it was feature flagged so therefore we don't care we'll turn it on. I think you've got to be intentional and saying this was great. We did 20 rounds of iteration where normally we would have done two. We saved so much **[42:23]** time. Now we can use that time to go back and build the feature. Right. and setting that intention up front and having these multip these uh kind of dual tracks of like building to learn versus building for scale. I think that's a really good habit because then you can have that conversation explicitly like oh no this is a build to scale this is a build tolearn feature it's fine if we if it's not perfect or this was a build tolearn feature we've got to rebuild it for scale because it's not no one's actually looked at the quality of this in depth. So having that upfront um language and and and **[42:55]** intentionality lets you get the best of both worlds where you can iterate super fast, learn fast and then um build build to production. And like with with I think James you talked about this earlier like this new world of of AI where um product managers are able to to write code understanding for that product manager understanding you're building to learn. You're not building to scale. If you're building to scale, well, you need some involvement from engineering. If you're building to learn, this is great. Go go go crazy. We'll put we'll have some **[43:27]** guardrails in place. Um, but yeah, iterate, learn. Um, and then once we're ready to scale, we'll bring in the engineers and and go a little bit slower, but build it fast. >> And I I find that much easier in gy projects versus brownie projects. If you're starting from scratch, it's very easy to just like quickly get something out and get feedback from customers. >> All right, y'all. We are at time. This was a fantastic conversation. I hope everyone is excited about some of the takeaways from what you all shared as **[43:57]** well as you know shipping great products and value. Uh I think we're all here because we deeply deeply care about the code review step as part of quality as part of the value that we're delivering with the products that we're building. So amazing panel for all participants. Thank you so much for joining. Please uh move the conversation to the Slack channel. I encourage you to keep the questions coming. Uh both myself and the panelists will hang around there and keep an eye on the channel if you have more thoughts or questions or carry on the conversation. All right, everyone. **[44:28]** Have a great rest of your day. Thank you again for joining us. Bye everyone. >> Everybody. >> Byebye. Thank you. >> Bye.