What to Do When There’s Too Much Code to Review  🔗 **Source:** [What to Do When There’s Too Much Code to Review](https://www.youtube.com/watch?v=0dbMKGS7uWk) > [!note]+ Document Notes > tags: [code-review, ai-assisted-development, engineering-leadership, developer-productivity, software-quality] > author: [MISSING] > source: Lead Dev > reference: https://www.youtube.com/watch?v=0dbMKGS7uWk > date created: {{date:YYYY-MM-DD}} {{time:HH:mm}} > > # 🏷️ Title: What to do when there's too much code to review > <!-- deep-read:sections start --> ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ## 📝 Summary - **What happened**: AI-powered development is dramatically increasing code volume, creating review bottlenecks for senior engineers - **Why it matters**: Teams need new processes to maintain code quality while leveraging AI productivity gains - **Key stakeholders**: Staff engineers, engineering managers, and development teams adopting AI tools - **Risks and tradeoffs**: Quality vs velocity; human oversight vs automation; building wrong things faster - **Bottom line**: Success requires treating code review as a systems problem, shifting feedback left, and implementing intelligent guardrails ## 📖 Main Content The panel addresses a critical challenge facing engineering teams: how to manage exponentially increasing code volumes from AI-assisted development while maintaining quality standards. ### Core Problems Identified **Volume Overload**: AI enables developers to produce significantly more code, but review capacity remains constant. Senior engineers become bottlenecks as they context-switch between numerous PRs. **Quality vs Speed Tension**: Teams risk building "the wrong thing" faster because conversations that previously happened during slower development cycles are being skipped. **Systems vs Tooling**: Many teams focus on "model shopping" rather than addressing the systemic workflow changes needed for AI-augmented development. ### Key Solutions Framework **Shift Feedback Left**: Move architectural and design discussions to the planning phase rather than post-implementation code review. This provides higher leverage for senior engineer feedback. **Guardrails Strategy**: Implement layered quality controls including deterministic tools (linters, static analysis) combined with AI review, rather than relying solely on human review. **Context Engineering**: Maintain repository-specific knowledge bases and policies that carry forward between reviews to improve AI review quality. **Continuous Improvement Culture**: Build team muscle for systematically identifying review pain points and pushing them into automated checks. ## 🗒️ Notes #### ⏱️ [00:00:00](https://www.youtube.com/watch?v=0dbMKGS7uWk&t=0) Panel Introduction - Delia Havens (SVP Engineering at LUSK) moderating - James (Staff Engineer at Tilt) - fintech serving underserved lending market - Hasset (Enterprise Engineering Lead at Code Rabbit) - AI code review tool - Pete Hodgson (Head of Technology at Tribe AI) - AI native transformation consultant #### ⏱️ [00:05:12](https://www.youtube.com/watch?v=0dbMKGS7uWk&t=312) Core Problems with Code Review - **James**: Keeping up with volume while maintaining focus at higher velocities - **Pete**: Too much feedback happening post-coding vs during planning phase - **Hasset**: Senior engineers becoming bottlenecks due to context switching across multiple PRs #### ⏱️ [00:07:30](https://www.youtube.com/watch?v=0dbMKGS7uWk&t=450) Bottleneck Shift to Verification - **Pete**: Bad pattern is "agentic meat puppet" - junior dev takes AI output and passes to senior for review - Good pattern: Upfront collaboration with senior engineers before writing code - Focus on getting feedback on design decisions before implementation - Architecture discussions and RFCs remain critical for describing intent #### ⏱️ [00:12:15](https://www.youtube.com/watch?v=0dbMKGS7uWk&t=735) Successful Team Patterns - **Hasset**: Winning teams treat this as systems problem, not just model selection - Collect right context, do proper configuration, plan before coding - **Pete**: Teams with continuous improvement culture systematically push review issues into automated checks #### ⏱️ [00:15:45](https://www.youtube.com/watch?v=0dbMKGS7uWk&t=945) Building Wrong Things Faster - **James**: With 10x development speed, cross-cutting conversations aren't happening - Results in siloed code that doesn't align with broader initiatives - Need to maintain architectural discussions despite faster implementation #### ⏱️ [00:18:20](https://www.youtube.com/watch?v=0dbMKGS7uWk&t=1100) Guardrails Implementation - **James**: Evolved from custom CLI tool to commercial solutions - Focus on refined code owners files for proper reviewer assignment - QA notes checker to ensure testing documentation - System awareness beyond repository (test coverage, integration impacts) #### ⏱️ [00:24:30](https://www.youtube.com/watch?v=0dbMKGS7uWk&t=1470) Feedback Loop Optimization - **Pete**: Push traditional outer loop feedback into inner loops - Teach agents to run linters and tests autonomously - ESLint rules often better than LLM feedback - deterministic, fast, cheap - Agents good at writing custom ESLint rules for specific patterns #### ⏱️ [00:27:45](https://www.youtube.com/watch?v=0dbMKGS7uWk&t=1665) Centralized Guardrails Challenge - **Hasset**: Difficulty centralizing guardrails when they run on developer machines - Need centralized enforcement at PR level, not just local pre-commit hooks - **James**: Seen issues with overly aggressive linting affecting all open files #### ⏱️ [00:31:10](https://www.youtube.com/watch?v=0dbMKGS7uWk&t=1870) Avoiding Nitpick Reviews - **Hasset**: Look for sufficient evidence for automatic review with safety guardrails - Mix deterministic tools with AI-based review - Continuous verification with repo-specific knowledge base - **James**: Regular cleanup of unused/ignored review rules based on developer response patterns #### ⏱️ [00:35:50](https://www.youtube.com/watch?v=0dbMKGS7uWk&t=2150) Productivity Metrics - **Hasset**: Code Rabbit tracks 64-65% acceptance rate (up from low 20s initially) - Avoid time-to-merge as primary metric - developers should control merge timing - **James**: Researching agentic PR metrics - PRs merged with no extra commits, agent vs human commit ratios #### ⏱️ [00:40:15](https://www.youtube.com/watch?v=0dbMKGS7uWk&t=2415) Feature Maturity and Review Standards - **Hasset**: Code changes should always be taken seriously regardless of feature flags - **James**: Devil's advocate - willing to ship less tested code for feature-flagged experiments at 1% rollout - **Pete**: Distinguish "building to learn" vs "building to scale" - different quality standards appropriate ## 📚 References - [Code Rabbit](https://coderabbit.ai/) - AI code review tool - [Lead Dev](https://leaddev.com/) - Engineering leadership community - [Tilt](https://www.tilt.com/) - Fintech lending platform - [Tribe AI](https://tribe.ai/) - AI transformation consulting ## 🗣️ Paraphrase Engineering teams are experiencing a productivity paradox with AI-assisted development: while individual developers can produce significantly more code, the traditional code review process is becoming a major bottleneck. Senior engineers find themselves overwhelmed by the volume of pull requests requiring review, leading to context switching fatigue and potential quality compromises. The solution isn't simply to review faster or add more reviewers, but to fundamentally restructure the development workflow. Successful teams are shifting critical feedback conversations earlier in the development cycle, implementing intelligent automation for routine checks, and building systems that capture and apply institutional knowledge consistently. The key insight is treating this as a systems engineering problem rather than a tooling problem. Teams need to identify which types of feedback can be automated, which require human judgment, and how to create feedback loops that provide value without creating friction. This includes everything from advanced linting rules to AI-powered contextual reviews to better architectural planning processes. ## 📰 Newsletter 🚀 **AI Code Review Crisis**: Engineering teams are drowning in pull requests as AI boosts developer productivity 10x, but review capacity stays flat. Three engineering leaders share battle-tested strategies: shift architectural feedback left to planning phase, layer smart guardrails (ESLint + AI + human oversight), and treat this as a systems problem, not a tooling one. Key insight: distinguish "building to learn" vs "building to scale" - different quality standards for each. Teams seeing 65% AI review acceptance rates (up from 20%) by focusing on context engineering and continuous improvement culture. 🛠️ **Quick Wins**: Set up bi-weekly "continuous improvement hours" to systematically push review pain points into automation. Your coding agents can write custom ESLint rules better than you can! [Read the full panel discussion](https://www.youtube.com/watch?v=0dbMKGS7uWk) ## ✅ Action Items #### DevRel - Document AI-assisted development best practices for community education - Create workshops on "feedback loop optimization" for engineering teams - Develop templates for architectural planning discussions pre-implementation #### Community - Share measurement frameworks for AI developer productivity - Create peer learning groups around code review process evolution - Document case studies of successful AI integration workflows #### WordPress - **Core Performance**: Establish AI-assisted performance review guidelines for Core contributions - **Community**: Update contribution guidelines for AI-augmented development workflows - **Training**: Develop curriculum on modern code review practices with AI tools ## 🧪 Hypotheses - Teams treating AI code review as a systems problem will achieve higher quality outcomes than those focused solely on tool selection - Shifting architectural feedback to pre-implementation phases will reduce review cycle time by 40% - Combining deterministic tools (linters) with AI review will achieve higher acceptance rates than AI-only approaches - Teams with regular continuous improvement processes will adapt to AI-augmented workflows faster than ad-hoc teams - Feature-flagged experimental code can maintain lower quality standards without impacting production systems ### 📊 Methodology - Survey engineering teams on pre/post AI adoption review metrics (cycle time, acceptance rates, defect rates) - A/B test feedback timing: architectural discussions pre-implementation vs post-implementation - Measure code quality metrics across different guardrail configurations - Track team satisfaction and productivity across different AI integration approaches ### ❓ Questions - How do you maintain code quality culture when AI can generate syntactically correct but architecturally poor solutions? - What's the optimal ratio of human to AI review for different types of code changes? - How should engineering career development change when junior developers can produce senior-level code volume? - What governance models work best for teams mixing human and AI code generation? ### 🔮 Future Research - Long-term impact of AI-augmented development on technical debt accumulation - Effectiveness of different AI models for domain-specific code review (fintech, healthcare, etc.) - Evolution of engineering roles and responsibilities in AI-augmented teams - Cost-benefit analysis of custom vs commercial AI code review tools ### 💡 Implications - **Practice**: Engineering teams must redesign their entire SDLC, not just add AI tools to existing processes - **Policy**: Organizations need governance frameworks distinguishing experimental vs production code quality standards - **Education**: Computer science curricula should include systems thinking about development workflows, not just coding skills - **Funding**: Investment in process automation and workflow optimization may provide higher ROI than additional engineering headcount ### 🧠 Learning Styles Assessment Reading level estimate: Flesch-Kincaid Grade 12 (appropriate for engineering professionals) #### 👀 Visual learners - Present: Workflow diagrams showing feedback loop optimization, before/after process comparisons - Improve: Add architectural decision trees, code review process flowcharts, metrics dashboards #### 🔊 Aural learners - Present: Panel discussion format with multiple perspectives and conversational learning - Improve: Add audio summary of key takeaways, follow-up podcast episodes #### 🗣️ Verbal learners - Present: Clear definitions of terms like "agentic meat puppet," detailed explanations of concepts - Improve: Add glossary of AI development terminology, more analogies for complex workflow concepts #### 🧩 Kinesthetic learners - Present: Practical implementation steps, specific tool configurations - Improve: Add hands-on workshop materials, step-by-step setup guides, interactive exercises #### 🧠 Logical learners - Present: Systematic framework for guardrails, metrics-based decision making - Improve: Add decision matrices for tool selection, ROI calculation templates #### 🧑‍🤝‍🧑 Social learners - Present: Panel discussion format, community learning emphasis - Improve: Add discussion prompts for team retrospectives, peer learning group guides #### 🧘‍♀️ Solitary learners - Present: Individual reflection on team practices, self-assessment opportunities - Improve: Add personal audit checklists, individual skill development paths ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Clear heading structure, descriptive link text - Improve: Add ARIA labels for video timestamps, transcript formatting improvements #### Keyboard navigation - Present: Standard web navigation patterns - Improve: Add skip-to-content links, keyboard shortcuts for video sections #### Contrast and readability - Present: Standard text contrast ratios maintained - Improve: Ensure code examples maintain sufficient contrast, test with high contrast modes #### Media and captions - Present: Video transcript provided - Improve: Add closed captions to video, audio descriptions for visual elements ### 🌍 DEIB Assessment #### Cultural breadth - Present: Global team examples (US, Australia, Brazil, New Zealand) - Improve: Include perspectives from different regional development practices #### Inclusive language - Present: Professional terminology, defined technical jargon - Improve: Expand glossary, avoid assumptions about team structure/resources #### Representation - Present: Multiple speakers with different backgrounds and company sizes - Improve: Include more diverse perspectives on team leadership and decision-making ### ✅ Logical Validity #### Strong reasoning - **Systems thinking approach**: Panelists consistently frame the problem as workflow redesign rather than tool adoption, supported by concrete examples - **Evidence-based metrics**: Discussion of measurable outcomes (65% acceptance rates, commit ratios) provides concrete validation - **Feedback loop theory**: Well-supported argument for shifting feedback earlier in development cycle #### Fallacies - **Appeal to novelty**: Some assumption that AI tools are automatically better than traditional approaches without sufficient comparison - **Survivorship bias**: Examples focus on successful implementations without discussing common failure modes in detail #### Tone of voice Professional and collaborative, with practical problem-solving focus. Examples from real experience provide credibility. ### 🛡️ DARVO Analysis No DARVO patterns detected. Discussion maintains focus on constructive problem-solving rather than defensive responses to criticism. ## 🎯 Conclusion The AI-powered development revolution requires a fundamental rethinking of code review processes, not just adding AI tools to existing workflows. Successful teams treat this as a systems engineering challenge, implementing layered guardrails, shifting critical feedback earlier in the development cycle, and building cultures of continuous improvement. The key practical next steps for readers: establish bi-weekly continuous improvement meetings, implement intelligent automation for routine checks, and clearly distinguish between experimental and production code quality standards. Teams that master these workflow adaptations will gain sustainable competitive advantages in the AI-augmented development era. ## 🔄 Refactor Appendix — What to do when there's too much code to review ### 📚 Literature Atoms #### AI increases code volume but reduces implementation conversation > [!quote] ID:rw_1 > With AI we're noticing not so much that the code is wrong but we're actually building the wrong thing because the conversations we would have had when it took 10 times as long to build the code. **Context & use**: - Understanding how AI acceleration affects product planning and cross-team coordination **Tags**: #ai-development #product-planning #xref #### Code review bottlenecks shift from volume to context switching > [!quote] ID:rw_2 > The staff engineers the lead engineers become the bottleneck after some point because the amount of volume that they see they cannot context switch between all of these amongst all these different PRs. **Context & use**: - Identifying scaling challenges in code review processes for senior engineers **Tags**: #code-review #engineering-management #xref #### Feedback should move from post-implementation to pre-coding phase > [!quote] ID:rw_3 > Rather than that maybe less senior engineer kind of not looking for feedback at all or just taking whatever the AI is producing and giving it to the senior person. What I think the good practices is a lot more upfront collaboration. **Context & use**: - Restructuring development workflows to prevent "slop cannon" from AI-generated code **Tags**: #ai-development #code-review #xref #### Successful AI adoption requires systems thinking over model shopping > [!quote] ID:rw_4 > The teams that win are the ones looking at it as a systems problem rather than just a model shopping problem. Like I see a lot of teams just talking about, oh, we have the best model. **Context & use**: - Evaluating AI tool adoption strategies beyond just choosing the best model **Tags**: #ai-adoption #systems-thinking #xref #### Traditional tools often outperform AI for deterministic code checks > [!quote] ID:rw_5 > An ESLint rule is just so much better than an LM can ever do. It's like cheaper, faster, deterministic, all of the things. **Context & use**: - Choosing appropriate tools for different types of code quality enforcement **Tags**: #code-quality #tooling #xref #### AI code review acceptance rates indicate system effectiveness > [!quote] ID:rw_6 > We're close to on an average I would say last time I checked was 64 65% acceptance ratios across the board across all companies which is amazing. **Context & use**: - Using acceptance rates as a metric for AI code review tool performance and configuration **Tags**: #metrics #ai-tools #xref <!-- deep-read:sections end --> 📄 **Full text:** ![[Full Document Contents/Articles/What to Do When There’s Too Much Code to Review.md|What to Do When There’s Too Much Code to Review by YouTube]] ## Highlights - sure this topic is very very relevant to you. It's amazing how much code we're able to ship. Uh and that's probably still leaves us with the same number of reviewers, the same engineers on your team, uh trying to keep up with it all. So like at least for for me and the enterprise products and code bases that I work in, we're talking about brownfield here. So like hold your objections green field. You can yolo your code if you want to, but really there is intentionality in the care not ([View Highlight](https://read.readwise.io/read/01kv3tdxdq050bddj17p0mfh0p)) - Tags: 👻 ai highlighted - that's changing um how where to put the guard rails and and how to handle the amount of code that's coming to your team. So, let's go ahead and dig in. So I to to kick us off um maybe for the panelists here, I'd love to hear from each of you one s sentence or something, you know, quick on the single biggest thing that's broken about code reviews right now. All right, James, we'll start with you. >> I think the biggest thing for us is keeping up with the code review volume ([View Highlight](https://read.readwise.io/read/01kv3tdxek4ggm3b1wanvn4haz)) - Tags: 👻 ai highlighted - >> So yeah, I I'd like to echo uh what James said and the volume of code is increasing. Not just that the number of PRs is going up. Uh it's especially the the the staff engineers the the lead engineers become the bottleneck after some point because the amount of volume that they see they cannot effectively context switch between all of these amongst all these different PRs to to actually give a meaningful review. ([View Highlight](https://read.readwise.io/read/01kv3tdxf9xbdjqtwcqkw0r67w)) - Tags: 👻 ai highlighted - >> So yeah, I I'd like to echo uh what James said and the volume of code is increasing. Not just that the number of PRs is going up. Uh it's especially the the the staff engineers the the lead engineers become the bottleneck after some point because the amount of volume that they see they cannot effectively context switch between all of these amongst all these different PRs to to actually give a meaningful review. ([View Highlight](https://read.readwise.io/read/01kv3wy0c158zktbcqyj954xvv)) - Note: The review bottleneck is context-switching, not volume per se: staff/lead engineers can't switch between all the incoming PRs fast enough to give meaningful review. - Tags: atom/literature/h1, code-review, engineering-management - where you're looking for feedback from those staff engineers and who's giving that feedback. So rather than that maybe less senior engineer kind of not looking for feedback at all or just taking um whatever the AI is producing and giving it to the senior person. What um I think the good practices and what people should be learning to do is a lot more upfront collaboration with ([View Highlight](https://read.readwise.io/read/01kv3wybe7y6y1v2b6p02hx13e)) - Note: Shift feedback earlier: instead of juniors dumping AI output on seniors at review time, do more upfront collaboration — move feedback from post-implementation to pre-coding. - Tags: code-review, atom/literature/h1, ai-development - that comes out of it and especially the teams that win are the ones looking at it as a as a as a systems problem rather than just a model shopping problem. Like I see a lot of teams just talking about, oh, we have the best model. Let's just write a quick tool to do a review and we'll be done with it. Um, it's often not that easy, right? Like um so ([View Highlight](https://read.readwise.io/read/01kv3wymf43dx8f8yqg0h52a6f)) - Note: Winning teams treat AI as a systems problem, not a "model shopping" problem — having the best model and a quick review tool isn't enough; the surrounding system is what matters. - Tags: atom/literature/h1, systems-thinking, ai-adoption - chipping away at it over time. >> Yeah, that's that's actually interesting. We we just had an on-site uh in Phoenix with our our global team recently and one of the the common things that came up is with AI we're noticing not so much that the code is wrong but we're actually building the wrong thing because the conversations we would have had when it took 10 times as long to build the code. >> Yeah. >> They're just not happening. And so you end up with a lot of siloed code that's reproduced and doesn't quite do what >> a cross pod cutting initiative would do. And so I I will say transparently like ([View Highlight](https://read.readwise.io/read/01kv3wxk42ynb4ncyedr9zza2n)) - Note: AI's risk isn't wrong code — it's "building the wrong thing." The alignment conversations that happened when code took 10x longer stop happening, leaving siloed, reproduced code. - Tags: product-planning, atom/literature/h1, ai-development - the value. So given your work um could you share maybe some details about that and and what are some of these guard rails or like what what takes you beyond just you know your typical um when it comes to effectiveness with code reviews. Yeah, we we started pretty early on the AI code review process. We were essentially using um a CLI tool within an Azure pipeline, experimenting with different rules and models and that probably got us 90% of the way there, it ([View Highlight](https://read.readwise.io/read/01kv3tdxjvmy54fyp7qppc61z6)) - Tags: 👻 ai highlighted - better than an LM can ever do. It's like cheaper, faster, deterministic, um, all all of the things. And turns out that Claude code or your coding agent choice is really good at re at writing custom ESLint rules. So, when you see an error, rather than saying like, "Oh, I need to add to my claude. MD or my agent.mmd or my skills file. You know, never never use an arrow function when you produce whatever. Um or or make sure you follow the rules of hooks for React. There's an ESLint rule for that. You ([View Highlight](https://read.readwise.io/read/01kv3wyzdy7cr7n0n92c712yca)) - Note: Use deterministic tools where they win: ESLint is cheaper/faster/deterministic than an LLM for rule-checking. Have your coding agent write a custom ESLint rule instead of stuffing more "never do X" lines into CLAUDE.md/skills. - Tags: code-quality, atom/literature/h1, tooling - we've always seen that and and we've seen a steady increase as we uh put in more uh bring in more context into the review um with the models of course we benefit from the models getting better as well uh but as our as our harness gets better over time. Uh we we're close to on an average I would say last time I checked was 64 65% acceptance ratios across the board across all companies which is which is amazing. I think we've ([View Highlight](https://read.readwise.io/read/01kv3wzapdbpf0b707cc65afkz)) - Note: AI-review acceptance rate is the effectiveness signal: ~64–65% acceptance across companies, rising as the harness brings in more context (not just from better models). - Tags: atom/literature/h1, metrics, ai-tools