My Thoughts on Building Software Using AI Agents  🔗 **Source:** [My Thoughts on Building Software Using AI Agents](https://adamadam.blog/2026/05/13/my-thoughts-on-building-software-using-ai-agents/) > [!note]+ Document Notes > tags: #artificial-intelligence #software-development #productivity #code-quality > author: Adam Zieliński > source: Adam's Perspective > reference: https://adamadam.blog/2026/05/13/my-thoughts-on-building-software-using-ai-agents/ > date created: 2024-12-19 10:30 > > # 🏷️ Title: My thoughts on building software using AI agents > <!-- deep-read:sections start --> ## 🧰 Action Inbox - [H1] Implement work-in-progress limits for AI agent usage — avoid context switching fatigue #productivity - [H2] Experiment with autonomous loops for complex tasks — set clear outcomes and continuous validation #automation - [H3] Establish human-driven vs LLM-driven project policies — prevent understanding erosion #governance ## 🧭 Table of Contents - [[#📝 Summary]] - [[#📖 Main Content]] - [[#🗒️ Notes]] - [[#📚 References]] - [[#🗣️ Paraphrase]] - [[#📰 Newsletter]] - [[#✅ Action Items]] - [[#🧪 Hypotheses]] - [[#📊 Methodology]] - [[#❓ Questions]] - [[#🔮 Future Research]] - [[#💡 Implications]] - [[#🧠 Learning Styles Assessment]] - [[#📊 Accessibility Assessment]] - [[#🌍 DEIB Assessment]] - [[#✅ Logical Validity]] - [[#🛡️ DARVO Analysis]] - [[#🎯 Conclusion]] ### 🔗 References - [Adam's Perspective Blog](https://adamadam.blog/) - [Programming as Theory Building](https://adamadam.blog/2026/05/13/my-thoughts-on-building-software-using-ai-agents/#fn:1) ## 📝 Summary - A developer's reflection on practical challenges when using LLMs for software development - Identifies three key problems: human attention limits, understanding erosion, and quality degradation - Emphasizes work-in-progress limits and autonomous loops as potential solutions - Argues for clear separation between human-driven and LLM-driven projects - Highlights the ongoing reliability issues with current AI coding tools ## 📖 Main Content Adam Zieliński shares his experience using Large Language Models extensively for software development, particularly after gaining unlimited access to commercial models. He identifies three critical challenges that emerge from heavy LLM usage in development workflows. The first challenge centers on human attention as the limiting factor. Despite the temptation to have AI agents work on multiple tasks simultaneously, Zieliński argues this creates context-switching fatigue and prevents meaningful completion of work. He draws on Toyota's work-in-progress limit principles, suggesting developers should focus on driving one or at most two agents at a time. His proposed solution involves autonomous loops where developers define expected outcomes once and let models run continuously, picking up results on their own schedule. However, he notes that models often report premature success, requiring loops to continue working even after claiming completion. The second major issue is the erosion of software understanding. Traditional development allows humans to build mental models through time, repetition, and multiple approaches to problems. LLMs generate code too quickly for humans to meaningfully engage with, leading to a loss of system comprehension that no amount of summarization or automated reviews can replace. Quality degradation represents the third challenge, with LLM-generated code often requiring extensive testing and debugging cycles. The author reports that his agents spend most time running tests and fixing bugs, while he spends most time finding defects in their work. He specifically criticizes Claude for ignoring directions and solving different problems than requested. ## 🗒️ Notes #### Opening Context - Key idea: Unlimited LLM access increased usage and revealed fundamental limitations - Evidence: "I even got an unlimited access to a popular commercial model recently, which increased my usage even more" #### Human Focus Scarcity - Key idea: Human attention becomes the bottleneck when managing multiple AI agents - Evidence: "I'm very fatigued by switching between multiple agents and re-prompting them. I lose focus, and I can't meaningfully drive anything to completion" #### Toyota WIP Principles - Key idea: Work-in-progress limits from manufacturing apply to AI agent management - Evidence: "having more than n projects in progress at any given time means increases the total time required to ship them" #### Autonomous Loop Strategy - Key idea: Define outcomes once and run continuous loops to manage multiple projects - Evidence: "I'm defining the expected outcome once and run the model in a while(true) bash loop" #### Premature Success Reporting - Key idea: Models incorrectly claim completion before tasks are actually finished - Evidence: "I've noticed models often report success prematurely, so the loop is instructed to continue working" #### Understanding Erosion Mechanics - Key idea: Human understanding requires time and repetition that LLMs bypass - Evidence: "Human developers use time to turn code + requirements into understanding of the system. LLMs use electricity to turn code + requirements into large volumes of text" #### Code Volume Problem - Key idea: LLM-generated code comes too fast for human comprehension - Evidence: "The more new code produced per day, the faster maintainers give up on even reading it" #### Human vs LLM Project Division - Key idea: Projects should be primarily driven by either humans or LLMs, not both - Evidence: "Dennis Snell said projects must be primarily driven by either humans or LLMs, but not by both. I agree" #### WordPress Core Concerns - Key idea: LLM-heavy development unsuitable for critical infrastructure projects - Evidence: "I'd rather not adopt that modus operandi in WordPress core" #### Quality Degradation Evidence - Key idea: Reliable software increasingly fails due to LLM-driven development practices - Evidence: "GitHub throws errors and changes the content of people's repositories, major companies have daily outages" #### LLM Review Chains - Key idea: End-to-end LLM processes lack human oversight until customer impact - Evidence: "LLM-produced code is reviewed by LLM reviewers, then passes LLM-generated checks, gets tested by LLM testers" #### Economic Uncertainty - Key idea: Token pricing changes could dramatically alter LLM viability - Evidence: "tokens may increase in price 100x in the next few years" #### Agent Time Distribution - Key idea: Most AI agent time spent on testing and bug fixing rather than initial development - Evidence: "My agents spend most of their time running tests and fixing bugs" #### Test Coverage Limitations - Key idea: LLM-generated tests focus on surface-level scenarios despite specific guidance - Evidence: "LLM-generated tests tend to cover surface-level happy paths in many of my long-running loops, even when I'm very specific" #### Claude-Specific Issues - Key idea: Claude consistently ignores directions and solves different problems - Evidence: "Claude, especially, is very prone to ignoring my directions and solving a different problem than the one I wanted solved" ## 📚 References - [Toyota Production System - Work in Progress Limits](https://en.wikipedia.org/wiki/Toyota_Production_System) - [Programming as Theory Building by Peter Naur](http://pages.cs.wisc.edu/~remzi/Naur.pdf) - [Dennis Snell's WordPress Development Insights](https://profiles.wordpress.org/dmsnell/) ## 🗣️ Paraphrase - Managing multiple AI coding agents simultaneously creates attention fragmentation that prevents meaningful task completion, similar to how Toyota discovered work-in-progress limits improve manufacturing efficiency - LLMs generate code faster than humans can understand it, leading to a fundamental erosion of system knowledge that automated tools cannot replace through summaries or reviews - Current AI-generated software shows increasing quality problems because the entire pipeline from code generation to testing lacks meaningful human oversight until customers encounter issues ## 📰 Newsletter **AI Agents in Development: Lessons from the Trenches** One developer's candid experience with unlimited LLM access reveals three critical challenges for WordPress developers considering AI integration: **The Attention Trap**: Just because AI can handle multiple tasks doesn't mean you should manage multiple agents. Work-in-progress limits that improved Toyota's manufacturing apply to AI workflows too. **The Understanding Gap**: When code generates faster than humans can comprehend it, we lose the deep system knowledge that prevents technical debt and security issues. **The Quality Slide**: End-to-end AI pipelines often lack human oversight, contributing to the reliability issues we're seeing across major platforms. **WordPress Takeaway**: Consider designating projects as either human-driven or AI-driven from the start. For WordPress Core and mission-critical plugins, human understanding remains essential. Resources: [Learn WordPress](https://learn.wordpress.org/), [Developer Docs](https://developer.wordpress.org/), [WP-CLI](https://wp-cli.org/) ## ✅ Action Items #### DevRel - Developer advocacy: Create guidelines for responsible AI tool usage in WordPress development - Education: Workshop on AI agent work-in-progress limits and context switching - Developer experience: Document when to choose human-driven vs AI-driven approaches - Developer marketing: Case studies on sustainable AI integration practices - Community management: Survey developers on AI tooling challenges and solutions #### OSPO - Policy & Governance: Draft AI-generated code review and attribution policies - License Compliance: Establish scanning protocols for AI-generated contributions - Security Management: Create vulnerability assessment processes for AI-assisted code - Software Inventory: Track AI tooling dependencies in project SBOMs - Developer Support: Training on AI tool limitations and quality controls - Community Engagement: Engagement with AI companies on developer tooling reliability - Strategic Alignment: KPIs for code quality in AI-augmented development workflows #### Community - Maintainers: Templates for AI usage policies in open source projects - Contributors: Guidelines for disclosure of AI assistance in contributions - Sponsors: Transparent reporting on AI tooling investments and outcomes - Ecosystem: Cross-project coordination on AI governance best practices #### WordPress - Core: Performance budget considerations for AI-generated code contributions - Core Performance: Profiling methodology for AI vs human-written code efficiency - Core AI: Guardrails for AI tooling used in WordPress development itself - Core Program Team: Coordination on AI usage policies across teams - Community: Meetup resources on responsible AI development practices - Training: Lesson plans on AI tool limitations and quality control - Documentation: Style guides for documenting AI-assisted development - Plugins: Security review protocols for AI-generated plugin code - Themes: Quality standards for AI-assisted theme development #### The WPCC - Programs: Fellowships focused on sustainable AI development practices - Projects: Documentation sprints on AI governance in open source - Funding: Grants for research on AI impact on software quality #### FAIR - Technical independence: Metadata standards for AI-generated code provenance - Policies: Transparency requirements for AI tooling usage - Governance artifacts: Charters addressing AI integration in fair software ## 🧪 Hypotheses - Work-in-progress limits for AI agents will improve task completion rates and reduce developer fatigue compared to concurrent multi-agent management - Projects with clear human-driven vs AI-driven designation will maintain better code quality and system understanding than mixed approaches - Autonomous loop strategies with continuous validation will produce more reliable outcomes than interactive prompting sessions - Token pricing increases will force a return to more selective, strategic AI usage patterns in software development ### 📊 Methodology - A/B testing of WIP-limited vs unlimited AI agent usage measuring completion rates and developer satisfaction - Longitudinal study tracking code comprehension and maintenance velocity in human-driven vs AI-driven projects - Quality analysis comparing autonomous loop outputs with interactive development sessions - Economic modeling of development costs under different token pricing scenarios ### ❓ Questions - How do we measure and maintain system understanding in an AI-augmented development world? - What quality gates can effectively catch AI-generated code issues before customer impact? - How should open source projects balance AI development speed with community comprehension? ### 🔮 Future Research - Development of AI tooling that preserves human understanding through better abstractions - Investigation of hybrid development workflows that maintain code quality - Economic analysis of long-term sustainability for AI-heavy development practices ### 💡 Implications - Practice: Developers need explicit strategies for managing AI tool cognitive load - Policy: Organizations require clear guidelines for AI vs human responsibility in code - Security: AI-generated code may require enhanced review processes and testing - Education: Developer training must include AI tool limitations and quality control - Funding: Investment in AI tooling should account for hidden quality and comprehension costs ### 🧠 Learning Styles Assessment Reading level estimate: 12th grade #### 👀 Visual learners - Present: Clear section breaks and conceptual organization - Improve: Diagrams showing AI agent workflow patterns, quality degradation visualizations #### 🔊 Aural learners - Present: Conversational tone and personal narrative structure - Improve: Podcast-style audio version discussing real-world AI development experiences #### 🗣️ Verbal learners - Present: Analogies to Toyota manufacturing, clear definitions of technical concepts - Improve: More manufacturing metaphors, comparison tables for different approaches #### 🧩 Kinesthetic learners - Present: Specific strategies like autonomous loops and WIP limits - Improve: Step-by-step implementation guides, hands-on exercises for testing approaches #### 🧠 Logical learners - Present: Clear cause-and-effect relationships between AI usage and outcomes - Improve: Decision trees for choosing AI strategies, metrics for measuring success #### 🧑‍🤝‍🧑 Social learners - Present: References to community insights (Dennis Snell) and shared challenges - Improve: Discussion prompts for development teams, collaborative evaluation rubrics #### 🧘‍♀️ Solitary learners - Present: Personal reflection format and individual experience sharing - Improve: Self-assessment questions, private journaling prompts for AI usage patterns #### 🌿 Naturalistic learners - Present: Ecosystem thinking about software quality and interconnected impacts - Improve: Analogies to natural systems, sustainability frameworks for AI tool usage ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Good heading hierarchy and semantic structure - Improve: Add landmark roles for main sections, descriptive alt text for any future diagrams #### Keyboard navigation - Present: Standard blog navigation and focus patterns - Improve: Skip links for long-form content, logical tab order for interactive elements #### Contrast and readability - Present: Clean text layout with adequate white space - Improve: Verify color contrast ratios meet WCAG AA standards #### Text resizing and reflow - Present: Responsive design supporting text scaling - Improve: Test reflow behavior at 200% zoom, ensure no content clipping #### Media and captions - Present: Text-based content with clear structure - Improve: Provide audio transcripts if audio version created #### Forms and interactions - Present: Standard blog commenting interface - Improve: Ensure form labels and error handling meet accessibility standards #### Standards reference - Align with WCAG 2.2 AA Level success criteria for long-form content ### 🌍 DEIB Assessment #### Cultural breadth - Present: Focus on universal development challenges applicable across contexts - Improve: Examples from different development cultures, non-Western software practices #### Inclusive language - Present: Generally accessible technical writing, personal experience framing - Improve: Define all technical acronyms, provide plain-language summaries #### Neurodiversity support - Present: Clear section breaks and consistent structure - Improve: Add content warnings for stress/anxiety-inducing topics, provide multiple reading paths #### Representation - Present: Personal perspective acknowledging individual experience limitations - Improve: Include voices from different developer backgrounds and experience levels #### Psychological safety - Present: Honest discussion of challenges without blame or judgment - Improve: Acknowledge different comfort levels with AI tooling, provide multiple valid approaches ### ✅ Logical Validity #### Strong reasoning - Deductive strength: Sound application of manufacturing principles to development workflows - Inductive strength: Reasonable generalizations from personal experience with caveats - Causal reasoning: Clear connections between AI usage patterns and observed outcomes - Analogical strength: Appropriate Toyota manufacturing comparison with relevant parallels #### Fallacies - Hasty generalization: Some conclusions drawn from limited personal experience - addressed by acknowledging scope - Appeal to authority: Dennis Snell reference appropriately contextualized within expertise area #### Tone of voice - Overall tone: Reflective and cautionary, balancing enthusiasm with realistic assessment of limitations ### 🛡️ DARVO Analysis - Deny: No denial patterns present - Attack: No attacks on critics of AI tooling - Reverse victim/offender: No victim/offender reversals identified - Emotional cues and rhetoric: Appropriately cautious tone about AI limitations without inflammatory language ## 🎯 Conclusion The article provides valuable practical insights into the real challenges of AI-augmented development, emphasizing that human cognitive limitations often become the bottleneck rather than AI capabilities. The key takeaway is that successful AI integration requires deliberate workflow design, clear project governance, and honest assessment of quality tradeoffs. For WordPress developers and the broader open source community, this suggests the need for explicit policies distinguishing human-driven from AI-driven development approaches, particularly for critical infrastructure projects. ## 🧱 Refactor <!-- deep-read:sections end --> 📄 **Full text:** ![[Full Document Contents/Articles/My Thoughts on Building Software Using AI Agents.md|My Thoughts on Building Software Using AI Agents by adamadam.blog]] ## Highlights - Once LLMs are involved, humans can’t and don’t meaningfully engage with new code. It comes too fast, it lacks soul, and it requires too much effort to go through. The more new code produced per day, the faster maintainers give up on even reading it. ([View Highlight](https://read.readwise.io/read/01krk75bvq8h5xssz04v5wfjzg)) - Tags: 👻 ai highlighted - My agents spend most of their time running tests and fixing bugs. Even with less frequent test runs, the tests must run eventually. LLM-generated tests tend to cover surface-level happy paths in many of my long-running loops, even when I’m very specific about the desired depth of coverage. And even then, most test runs fail, leading the agent to change the code and re-run the tests. Which means more waiting. ([View Highlight](https://read.readwise.io/read/01krk79454bzr9e8hky1d6n681)) - I spend most of my time finding defects in the agent’s work and getting them to fix them. ([View Highlight](https://read.readwise.io/read/01krk79c03z8gystbs8nbzm5n8))