AI mistakes you're probably making - CourtneyR.dev's 2nd Brain

AI mistakes you're probably making 🔗 **Source:** [AI mistakes you're probably making](https://www.youtube.com/watch?v=Jcuig8vhmx4) > [!note]+ Document Notes > tags: [ai-coding, developer-tools, claude-code, cursor, context-management] > author: Theo - t3․gg > source: YouTube > reference: https://www.youtube.com/watch?v=Jcuig8vhmx4 > date created: 2025-01-27 23:06 > > # 🏷️ Title: AI mistakes you're probably making >  ## 🧰 Action Inbox - [H1] Test AI tools on problems you already know how to solve — validate solutions against your knowledge #validation - [H2] Create frozen reproducible test cases from past bugs — use for evaluating new models #testing - [H3] Fix broken environments before blaming AI performance — eliminate type errors and config issues #environment ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [G2I Hiring Platform](https://soyv.link/2i) - [Repo Mix Project](https://repomix.com) ## 📝 Summary - Developers are making systematic mistakes when using AI coding tools, leading to poor experiences - Common errors include selecting wrong problems, poor context management, using outdated tools, broken environments, and over-configuration - AI tools work best on problems you already understand, with minimal but precise context - Context rot occurs when too much information is provided - models perform worse with excessive context - Keeping environments clean and using current tools dramatically improves AI coding assistance - Bottom line: Start simple, fix fundamentals, use AI as a skilled new engineer rather than a last resort ## 📖 Main Content The core argument is that most developers are approaching AI coding tools incorrectly, leading to disappointing results that reinforce skepticism. The speaker identifies five critical mistakes: **Problem Selection**: Most people only use AI when they've exhausted other options and don't understand the problem - exactly when AI performs worst. AI excels at problems you already know how to solve. **Context Management**: Dumping entire codebases into AI context creates "context rot" - performance degrades significantly with too much irrelevant information. Tools like "repo mix" that flatten codebases are counterproductive. **Tool Currency**: AI capabilities are advancing rapidly (months, not years). Perspectives based on older tools or models may be drastically outdated. **Environment Issues**: Broken type configurations, linting errors, and development setup problems confuse AI agents and waste cycles on ghost issues. **Over-Configuration**: Adding numerous MCP servers, skills, and plugins typically makes performance worse through context bloat rather than better. The recommended approach treats AI like onboarding a skilled new engineer - provide precise context about the specific problem without overwhelming detail. ## 🗒️ Notes #### ⏱️ [00:00](https://youtu.be/Jcuig8vhmx4?t=0) — [H1] Introduction: Common AI skepticism - Key idea: Many developers see AI as useful for small projects but useless for real codebases - Evidence: "There's a lot of reasons most people should feel that way, and I did myself just a few weeks ago" #### ⏱️ [02:43](https://youtu.be/Jcuig8vhmx4?t=163) — [H2] Mistake 1: Selecting the wrong problems - Key idea: People use AI as a last resort when they don't understand the problem - Evidence: "Most people don't try a new solution on a problem they already know how to solve" #### ⏱️ [04:15](https://youtu.be/Jcuig8vhmx4?t=255) — [H3] Use AI on problems you already know how to solve - Key idea: Start with known problems to compare AI solutions against your own - Evidence: "If I know how I would fix this hydration error...start there. It's a great place to test these things out" #### ⏱️ [06:30](https://youtu.be/Jcuig8vhmx4?t=390) — [H1] Create reproducible test cases for evaluation - Key idea: Freeze code states before fixes to test new AI capabilities - Evidence: "Those types of tests are gold because benchmarks suck" #### ⏱️ [08:45](https://youtu.be/Jcuig8vhmx4?t=525) — [H2] Mistake 2: Context management failures - Key idea: Repo mix and similar tools that dump entire codebases are harmful - Evidence: "This project is a fucking scourge and should be wiped off of the internet" #### ⏱️ [10:12](https://youtu.be/Jcuig8vhmx4?t=612) — [H3] Understanding AI as autocomplete - Key idea: AI is next token prediction - too much context creates noise - Evidence: "The more the model knows, the dumber it gets" #### ⏱️ [12:34](https://youtu.be/Jcuig8vhmx4?t=754) — [H1] Context rot: Performance degradation with too much information - Key idea: Models perform worse with excessive context, even high-capacity ones - Evidence: "Sonnet 4.5...once you break 50k tokens of context, the model starts to perform worse" #### ⏱️ [15:20](https://youtu.be/Jcuig8vhmx4?t=920) — [H2] Proper context management: Less is better - Key idea: Give AI search tools rather than entire codebase - Evidence: "They give the model tools to find what it's looking for in the codebase" #### ⏱️ [18:45](https://youtu.be/Jcuig8vhmx4?t=1125) — [H3] Claude.md and agent.md best practices - Key idea: These files should capture gotchas and mistakes, not comprehensive docs - Evidence: "The role of this file isn't to describe every single thing about the codebase" #### ⏱️ [22:10](https://youtu.be/Jcuig8vhmx4?t=1330) — [H1] Mistake 3: Using outdated tools - Key idea: AI capabilities advance in months, not years - old experiences misleading - Evidence: "Things went from 'AI can never solve this' to 'Wait, what the fuck?' in literally a 3-month window" #### ⏱️ [25:30](https://youtu.be/Jcuig8vhmx4?t=1530) — [H2] Mistake 4: Broken environments - Key idea: Type errors, config issues, and broken setups confuse AI agents - Evidence: "If your type check involves a CD, an actual directory change, your environment is broken" #### ⏱️ [28:15](https://youtu.be/Jcuig8vhmx4?t=1695) — [H3] Ghost errors waste AI cycles - Key idea: Persistent environment errors cause agents to chase irrelevant problems - Evidence: "If you tell an AI agent about a ghost, it will chase it forever" #### ⏱️ [31:00](https://youtu.be/Jcuig8vhmx4?t=1860) — [H1] Mistake 5: Over-configuration and MCP bloat - Key idea: Adding multiple MCP servers and skills typically makes performance worse - Evidence: "People spend so much time trying to set the agents up to succeed that they lose the plot" #### ⏱️ [33:45](https://youtu.be/Jcuig8vhmx4?t=2025) — [H2] Successful developers use minimal configuration - Key idea: High-performing developers use stock tools with minimal customization - Evidence: Pete "uses stock codecs" and "puts out 500 plus commits a day regularly" #### ⏱️ [36:20](https://youtu.be/Jcuig8vhmx4?t=2180) — [H3] Plan mode and iteration strategies - Key idea: Fix bad outputs by improving initial context, not appending corrections - Evidence: "Rather than try to fix it with a better input being appended, revert, go back" ## 📚 References - [G2I Developer Network](https://soyv.link/2i) - [Claude Code Documentation](https://claude.ai/docs) - [Cursor IDE](https://cursor.sh/) - [React 19 Hydration Error Tracing](https://react.dev/) ## 🗣️ Paraphrase - Developers typically reserve AI tools for their hardest, most poorly understood problems, which is exactly when these tools perform worst - they excel at problems you already know how to solve - Flooding AI with entire codebases through tools like repo mix creates "context rot" that degrades performance - models need focused, relevant information with search capabilities, not everything at once - The AI coding landscape changes so rapidly (months, not years) that experiences from even six months ago may be completely outdated, requiring developers to stay current with latest tools and models ## 📰 Newsletter **Stop Making These AI Coding Mistakes - Your Development Workflow Needs This** Hey WordPress developers! 👋 If you've been skeptical about AI coding tools or frustrated with poor results, you're probably making some common mistakes that are sabotaging your experience. **The Game-Changer**: Don't save AI for your hardest problems. Instead, use it on problems you already know how to solve. This lets you compare solutions and build trust in the tool's capabilities. **Context is King**: Never dump your entire codebase into AI. Create focused claude.md or agent.md files that capture specific gotchas and project quirks - think of it as onboarding documentation for a skilled new engineer. **Stay Current**: AI capabilities advance in months, not years. That frustrating experience from six months ago? Completely outdated. Perfect for WordPress development workflows where you need reliable code assistance for theme customization, plugin development, and debugging. Resources: [Learn WordPress](https://learn.wordpress.org/), [Developer Docs](https://developer.wordpress.org/), [WP-CLI](https://wp-cli.org/) ## ✅ Action Items #### DevRel - Developer advocacy: Create content series on AI coding best practices, measurable through engagement metrics - Education: Workshop curriculum on context management and prompt engineering for development teams - Developer experience: Documentation improvements for AI tool integration workflows - Developer marketing: Case studies showcasing proper AI tool usage vs common mistakes - Community management: Moderate discussions about AI tool experiences, gather feedback on pain points #### OSPO - Policy & Governance: Guidelines for AI tool usage in open source contribution workflows - License Compliance: Review AI-generated code for license compliance and attribution requirements - Security Management: Evaluate AI coding tools for security vulnerabilities in generated code - Developer Support: Training materials on AI coding best practices and tool selection - Community Engagement: Fund research on AI impact on open source development patterns #### Community - Maintainers: Templates for claude.md/agent.md files that help AI understand project conventions - Contributors: Onboarding guides that include AI-assisted development workflows - Sponsors: ROI measurement of AI tools in open source development efficiency #### WordPress - Core: AI-assisted code review processes for core contributions and performance optimizations - Core Performance: Use AI for profiling analysis and optimization suggestions with human validation - Community: Meetup presentations on AI coding best practices for WordPress development - Training: Lesson plans integrating AI tools into WordPress development curricula - Documentation: DevHub examples showing AI-assisted plugin and theme development workflows - CLI: Command development using AI assistance with proper testing and validation - Themes: Review processes that account for AI-generated theme code quality standards - Plugins: Security review protocols for AI-assisted plugin development #### The WPCC - Programs: Fellowships focused on AI-assisted WordPress development methodologies - Projects: Documentation sprints creating AI-friendly project setup guides - Funding: Grants for research on AI impact on WordPress development productivity ## 🧪 Hypotheses - Developers using AI on known problems first will develop better intuition for tool capabilities than those using it as a last resort - Codebases with clean environments and minimal context will see 3x better AI assistance performance than those with configuration issues - Teams using minimal AI tool configuration will have higher productivity than heavily customized setups - Context rot occurs predictably at around 50k tokens regardless of model architecture improvements ### 📊 Methodology - A/B testing comparing AI performance on known vs unknown problems with same developer cohorts - Measurement of AI success rates across different context window sizes and codebase configurations - Longitudinal study tracking developer productivity with stock vs heavily customized AI toolchains ### ❓ Questions - At what context size does diminishing returns become negative returns for different model architectures? - How do we quantify the ROI difference between proper and improper AI tool usage in development teams? - What environmental factors beyond type errors significantly impact AI coding assistance quality? ### 🔮 Future Research - Automated detection of "ghost errors" that confuse AI agents in development environments - Development of standardized benchmarks for real-world AI coding assistance evaluation - Investigation of optimal context management strategies for different programming languages and frameworks ### 💡 Implications - Practice: Developers should audit their development environments and fix configuration issues before implementing AI tools - Policy: Engineering teams need guidelines for AI tool usage that emphasize environmental hygiene and minimal configuration - Security: AI-generated code requires validation processes, especially when context management is poor - Education: Developer training must emphasize context management and problem selection over tool maximization - Funding: Investment in AI coding tools should prioritize environmental compatibility over feature breadth ### 🧠 Learning Styles Assessment Reading level estimate: Grade 12-14 (technical content with specialized terminology) #### 👀 Visual learners - Present: Code examples, configuration screenshots, performance graphs - Improve: Flowcharts showing context management decisions, visual comparisons of good vs bad AI setups #### 🔊 Aural learners - Present: Video format with clear narration and examples - Improve: Audio-only version focusing on key concepts and decision frameworks #### 🗣️ Verbal learners - Present: Clear definitions, analogies (AI as new engineer, autocomplete metaphor) - Improve: Glossary of AI coding terms, comparative explanations of different approaches #### 🧩 Kinesthetic learners - Present: Step-by-step configuration examples, hands-on debugging scenarios - Improve: Interactive checklist for environment setup, practice exercises with broken configurations #### 🧠 Logical learners - Present: Clear problem-solution structure, performance metrics, decision trees - Improve: Flowchart for context management decisions, quantified performance impacts #### 🧑‍🤝‍🧑 Social learners - Present: Community examples, developer testimonials, team scenarios - Improve: Discussion prompts for development teams, peer review guidelines for AI-assisted code #### 🧘‍♀️ Solitary learners - Present: Individual reflection on current AI usage patterns - Improve: Self-assessment checklist, personal experimentation framework #### 🌿 Naturalistic learners - Present: Ecosystem thinking about AI tools in development environments - Improve: Systems thinking about tool interactions, environmental impact considerations ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Good heading hierarchy, descriptive timestamps - Improve: More detailed alt text for code examples, ARIA labels for technical concepts #### Keyboard navigation - Present: Standard video controls, logical tab order in references - Improve: Skip links for major sections, keyboard shortcuts for timestamp navigation #### Contrast and readability - Present: High contrast code examples, readable font sizes - Improve: Dark mode considerations, larger text options for code samples #### Text resizing and reflow - Present: Responsive layout that scales well - Improve: Ensure code examples remain readable at 200% zoom #### Media and captions - Present: Video with clear audio, transcript provided - Improve: Detailed captions including technical terms, downloadable transcript #### Forms and interactions - Present: Clear reference links, accessible video controls - Improve: Better focus indicators, accessible timestamp --- ## Full Transcript [00:00] I have a feeling I can guess what your [00:01] stance is on AI right now. It seems [00:03] pretty useful and especially helpful in [00:05] small projects, side projects, and early [00:07] stage startups. But when it comes to [00:09] real problems in real code bases, these [00:11] things just aren't that useful. And you [00:13] don't understand why everybody's getting [00:14] so hyped and excited about those things. [00:16] If that's your perspective, I [00:18] understand. There's a lot of reasons [00:19] most people should feel that way, and I [00:21] did myself just a few weeks ago. But the [00:23] harsh reality is that things are [00:25] changing, and I want to talk a bit about [00:27] that. But more importantly, the mistakes [00:29] that you might be making when you use [00:30] these tools that result in the [00:32] lackluster experience that you're [00:34] having. This stuff has changed so much [00:36] and my own understanding of how I use it [00:37] has too. I keep seeing other people who [00:40] are struggling and when I talk to them, [00:41] I found this set of mistakes that almost [00:43] all of them seem to be making that once [00:45] corrected result in really, really good [00:48] things being built using these tools. So [00:50] whether you're a skeptic that doesn't [00:52] see any value in these things yet, or [00:54] you're somebody who's really deep in the [00:55] weeds using them every day and you just [00:56] want to double check you're not making [00:58] obvious mistakes, hopefully this video [01:00] will be helpful. I have a ton of sources [01:02] and things I want to talk about here, [01:04] and I'm really excited to do all of [01:05] that. But the first thing I want to tell [01:07] you guys about is today's sponsor. It's [01:09] 2026. Everything about how we code [01:11] changed last year, yet nothing about how [01:13] we hire did. That's kind of insane [01:15] because hiring is worse than ever now [01:17] that we're getting swarmed with terrible [01:19] AI generated resumes constantly. If [01:21] you're tired of reading that AI slop, [01:23] check out G2I. These guys have it [01:25] figured out. They have a network of over [01:26] 8,000 engineers ready to go. And these [01:28] aren't just people who are fresh out of [01:30] college. These are experienced [01:31] developers, many of which have worked at [01:33] fang companies in the past. And they've [01:34] all been onboarded with the best modern [01:36] AI tools. So, you're not going to have [01:37] to catch them up the whole time that [01:39] they're employed. They even offer 7-day [01:41] work trials to make sure that the [01:42] employee that you get is a good fit for [01:44] the company. Do you know how much easier [01:46] my life would have been when I was doing [01:47] job hunting if I could have tested a [01:49] company for 7 days? I honestly think [01:51] both sides benefit from this. Nobody [01:53] wants to work somewhere that isn't a [01:54] good fit. Nobody wants an employee that [01:55] isn't either. It just makes sense to me. [01:57] And if you don't think 7 days is a lot, [01:58] I have good news for you. Their process [02:01] to get them to file their first PR from [02:03] when you start can take as little as 7 [02:05] days. After you sign up for G2I, you'll [02:07] meet with their team. You'll set up a [02:08] shared Slack channel, decide how often [02:10] you want to meet in chat, and within [02:12] five or less days from that point, you [02:14] will have your first candidates, many of [02:16] which are ready to go on the spot. And [02:18] don't worry, you get to interview the [02:19] candidates yourself, too. They do a [02:21] first pass. They'll even take questions [02:22] that you write for them to go ask the [02:24] candidates and send you video responses [02:26] so you can see how the candidate [02:28] actually interacts. This is a person [02:30] you're going to be employing. You want [02:31] to know what they're like, not just what [02:33] their credentials are on a piece of [02:34] paper. It's 2026, guys. Stop hiring like [02:36] it's 95. fix it now at soyv.link/2i. [02:40] So the first thing I want to talk about, [02:42] one of the biggest mistakes I see is [02:44] selecting the right problem. So we start [02:46] with I have a problem. Step one, you [02:49] validate the problem. Is it really a [02:51] problem? Maybe this is a thing I [02:52] misunderstand or doesn't matter. But [02:54] once you've decided that this is really [02:56] a problem and it does matter, the next [02:58] thing you probably do is the obvious [03:00] solution, if there is one. If you're [03:02] looking at this problem and it's a space [03:03] you understand and the solution seems [03:05] obvious, you can go try that. But if [03:08] that fails, you have to do the harder [03:11] solution, which is something less [03:13] obvious. Maybe you have to debug more to [03:14] get more information. Maybe you have to [03:16] think about it more, read more code, [03:18] check more logs, find some way to [03:20] reproduce it. You have to put more time [03:22] in. But once that fails, you go to step [03:25] four, which is try something new. This [03:28] is the point where once your existing [03:31] solutions, your existing way of doing [03:33] things has failed you, you then go try [03:35] something else to see if that will solve [03:37] it. So let's say you have a database [03:39] performance problem. Maybe when you do [03:42] too much throughput on your database, [03:43] people start having errors in their [03:45] experience. First thing to do is make [03:47] sure it's really a problem. Are the [03:48] errors actually happening? Are they only [03:50] happening when there is too much [03:52] throughput? Are they happening for some [03:54] other reason? Are they happening because [03:55] the user has a weird extension or they [03:56] shipped something wrong? Once you [03:58] validate the problem, you look at the [04:00] error. You see what line of code is on. [04:01] And then you go make the obvious fix. [04:04] You ship it. Nothing changes. You never [04:06] reproduce the issue because you couldn't [04:08] figure out how to. So then you sit down, [04:10] you put in the effort, you try to [04:11] reproduce it. You think you find the [04:13] repro for it. You ship some changes. You [04:16] put it out. Still have the same issue. [04:19] At this point, you've concluded this [04:20] database I'm using just cannot handle [04:22] the thing that I'm doing. So, you go [04:25] explore new solutions. Maybe you look at [04:27] a different database. You look at other [04:28] technologies or systems or you try [04:30] something else. Or maybe you ask the AI [04:33] for help at that point. And this is a [04:35] problem I see a lot. Most people don't [04:37] try a new solution on a problem they [04:40] already know how to solve. If you know [04:42] how to solve the problem, you just solve [04:44] it. You don't use another tool to try to [04:47] solve it. And this is a very common [04:49] mistake I see. People aren't using these [04:51] tools to solve problems they already [04:53] know and understand. They are using them [04:55] as almost like a safety net type thing [04:58] once everything else they've tried has [04:59] failed. And that's often the first time [05:01] they tried the thing. This is not meant [05:03] to pick on Adam, but I think he's [05:05] actually a really good example of this [05:07] particular problem here. Come on AI, [05:09] give me that 10x power that I saw out of [05:12] Davis 7. Ben Davis is my channel [05:14] manager, also a fellow YouTuber, and is [05:16] having a ton of fun building with all of [05:17] these things. He wants to see how [05:19] powerful this stuff can be. Here's the [05:21] example he gave asking a question to [05:24] Opus 4.5 in cursor. There's a hydration [05:27] error in this app. I traced it to this [05:30] file provider. TSX, can you help me find [05:32] the problem? This is for a lot of [05:35] different reasons the wrong way to use [05:38] AI. And thankfully, I'm already seeing [05:40] people in chat realizing they're doing [05:42] the same thing. They aren't reaching for [05:43] these tools to solve known problems more [05:46] easily. They're reaching for them when [05:49] they've exhausted the other things [05:50] within reach and then they're giving the [05:52] model something that's really, really [05:54] hard or poorly understood enough to seem [05:57] really, really hard. This is a really [05:59] important thing I want to push. I talked [06:01] about this a little bit in the you're [06:02] falling behind video and I'm going to [06:04] try to not let these things overlap too [06:05] much, but one of the best things to try [06:08] these tools on is problems you already [06:10] know how to solve because then you can [06:13] compare your solution to the solution [06:15] the tool would have done. If I know how [06:17] I would fix this hydration error or this [06:19] database error and I'm curious if the AI [06:21] can do it too, start there. It's a great [06:24] place to test these things out. Or [06:27] another way of thinking about it, what [06:29] context would you give another engineer [06:31] on your team? When I am handing problems [06:34] out to earlier career engineers at my [06:36] businesses, I'm not giving them the [06:38] problems that I don't understand. I'm [06:40] giving them the problems that I already [06:42] understand and know how to write the [06:44] code for, knowing that if I give it to [06:46] that employee, it's going to take longer [06:48] and maybe come out worse and maybe have [06:50] to do some back and forth with them to [06:51] get it in the right shape. But I'm doing [06:53] that knowingly to get it off my plate [06:55] and to spread the knowledge of the [06:56] codebase around the team. Learning how [06:58] to do that with AI is a skill you'll [07:00] pick up over time and you'll get an [07:02] intuition not just of what problems the [07:04] AI can solve, but also what things that [07:07] it needs to know it solved it right. And [07:10] a fun thing you can do here is gify it a [07:13] bit and automate it a bit. If you have a [07:15] problem and you know how to solve it and [07:17] you give it to the agent with the same [07:19] information you have, like you saw a log [07:21] that referenced a specific line of code [07:23] that you're pretty sure is the problem [07:25] because of this blog post you read. Give [07:27] it all of that information, all the [07:28] things that led you to the solution. [07:30] Maybe just go through a PR where you fix [07:32] something. Check out the commit before [07:34] that fix. Hand all of what you knew to [07:37] the agent and see if it can solve that. [07:39] Chances are it will. And if it doesn't, [07:42] you have a thing that is as good as [07:43] gold. you have a reproducible test to [07:46] see the capabilities of the model. If [07:48] you have a pull request that fixes a [07:50] problem that an agent was unable to fix, [07:53] go grab the code state before you push [07:55] that fix, freeze it somewhere, grab all [07:58] of the information you need to solve it, [08:00] put it in a markdown file in that place, [08:02] and then every time a new AI tool comes [08:04] out, throw it at that problem with that [08:07] frozen state and see if it can solve [08:09] that. Those types of tests are gold [08:11] because benchmarks suck. Now, and if you [08:14] can collect a handful of these types of [08:16] problems, the few things that the models [08:18] can't do, but you could, you have a [08:21] combination of things that's useful. You [08:23] have a realworld reproduction test. You [08:26] have the information needed to solve the [08:28] problem and you have knowledge of what [08:30] the right solution looks like and a way [08:32] to validate the solution was correct. [08:34] That is a hard combination of things to [08:36] get and you should be hunting for as [08:38] many of those as you can find because if [08:40] you can sit on them and use them to run [08:41] evals and you can tell your team or [08:44] people on Twitter or the company that [08:46] made the model, hey, this update made it [08:48] so these five things that the AI [08:51] couldn't do before, it can now. That is [08:53] huge. And if you can sit there and say [08:55] confidently, it can't solve any of these [08:57] things, it still can't solve any of [08:58] these things. That is even better. That [09:00] is really useful. And if you can hit me [09:02] up and share some of that with me, I [09:03] will gladly pay you to have these [09:05] reproductions that I can use to validate [09:07] and test new things when they come out. [09:09] That is as good as gold. That said, what [09:11] I think you'll discover when you do this [09:13] is that when you have the right context [09:15] and the right instructions and you had [09:17] enough information to solve the problem, [09:19] if you give the same exact information [09:21] to the AI, it will solve the problem in [09:23] roughly the exact same way. But that's [09:25] just the top of it. That's just [09:27] selecting the right problem. I was [09:29] touching on context management a bit [09:30] there. So I'm going to go deeper on this [09:32] for the next piece. This is a project [09:34] called repo mix. Its goal is to take a [09:37] given codebase and flatten all of it [09:40] into a single AI friendly file so that [09:42] you could include it as contacts with [09:44] chat bots. This project is a [ __ ] [09:47] scourge and should be wiped off of the [09:49] internet. Not only has this project [09:51] personally cost me and T3 chat in the [09:54] entrance we are running probably [09:55] multiple hundreds of thousands of [09:57] dollars, it also virtually guarantees [10:00] you're going to get [ __ ] output. It [10:02] turns out that AI shouldn't have access [10:04] to your whole codebase at once in the [10:07] context. I feel like I have to talk [10:09] about this multiple times a day at this [10:11] point because it happens so often, but [10:12] all AI is is autocomplete. If you hand [10:16] it the capital of the US is and then [10:19] stop, it is doing a bunch of crazy math [10:22] across a bunch of things called [10:23] parameters and those parameters linked [10:26] some text to other text. When you have [10:29] the early text of the capital of the US [10:32] is it increases the likelihood that the [10:35] next pointer is pointing towards [10:37] Washington DC. The whole model that we [10:40] use for LM is next token prediction. [10:43] Based on everything currently in [10:45] context, all of the information passed [10:47] to the model. What does it think the [10:49] next most likely 3 to six characters [10:51] are? And it does that over and over [10:53] again. The fact that these things can [10:56] generate real meaningful contributions [10:58] to real meaningful code bases that [11:00] effectively through this autocomplete [11:02] strategy is genuinely unbelievable and [11:05] is an accident that it was discovered in [11:06] the first place. But you have to be [11:08] conscious of this when you use the [11:10] tools. If you hand a model too much [11:13] context so that the majority of the [11:16] context is just nonsense about your [11:18] codebase, it's just going to generate [11:20] more nonsense. Sure, there are some [11:22] models that can handle way more tokens [11:24] than others. Like Opus 4.5, I think is [11:27] capped at 100, maybe 200k tokens, but [11:29] Gemini can do 1 to2 million. We should [11:32] just all use the models with higher [11:33] context windows. No. No, we shouldn't. [11:36] The more the model knows, the dumber it [11:38] gets. The problem isn't that it needs to [11:40] know everything. Because the more it [11:42] knows, the less likely it is to [11:44] autocomplete the solution. Because if [11:46] the problem exists in the codebase, [11:48] giving it the whole codebase does not [11:50] give it the solution. And if you get to [11:52] the context limits, even with something [11:53] like opus, once you break 50k tokens of [11:56] context, the model starts to perform [11:58] worse. There is a very important concept [12:00] of context rot which is when you have [12:03] too much context and it is distracting [12:05] you from the thing that matters. Just to [12:07] visualize it, this is the average score [12:11] that the models get for finding things [12:13] in a set of repeated words. And as you [12:16] increase the number of tokens the model [12:18] has, the success rates plummet fast. For [12:22] example, sonnet 4. Up to a certain [12:24] point, it's 100% successful. And then [12:26] once you have too much context, it goes [12:28] down to less than 60%. If the model has [12:31] too much information, it will perform [12:34] worse. Stop doing this. Imagine if in [12:37] every Jira ticket or linear issue you [12:39] got, the actual thing you cared about, [12:42] like the problem the user had was stuck [12:44] in the middle of paragraphs upon [12:47] paragraphs of [ __ ] And you have to [12:49] go deep into all of it to just find the [12:52] actual problem. That's effectively what [12:54] you're asking the AI to do when you [12:56] compress the whole codebase in to the [12:59] history. If you copy paste your codebase [13:02] into a chat product like T3 chat or [13:04] chatgbt or anything like that, it's [13:07] going to confuse the hell out of it. [13:08] It's not going to perform well. Don't do [13:11] that. The reason all of these new tools [13:13] like cloud code, open code, and even to [13:15] an extent things like cursor are doing [13:17] so well at this is because they don't [13:19] give the whole codebase to the model. [13:21] They give the model tools to find what [13:23] it's looking for in the codebase. Things [13:26] like an agent MD or a claude MD file [13:28] that briefly describes what exists where [13:30] in the codebase. Things like a GP tool [13:33] to find the specific line that matters [13:35] so that it can address that and only [13:37] pull in the section that matters instead [13:39] of the whole codebase. Just imagine how [13:41] you solve a problem. If you see some [13:43] typo on a web page, you don't go read [13:46] every line of the codebase trying to [13:48] find the typo. you use GP or command [13:50] shift F to find it in the codebase and [13:52] then you go fix it. Models aren't that [13:54] different from people. If you give it a [13:56] search tool, it will use it. If you [13:57] don't and you expect it to read every [13:59] line of the codebase, you're just as bad [14:00] as the junior engineer that would go and [14:02] do that. People often ask, "How do you [14:04] learn a codebase?" I tell them to go [14:05] check the poll requests. Now, my [14:07] thoughts are a little different. But you [14:09] certainly shouldn't do it by reading [14:10] through it file by file. You should do [14:12] it by searching around the codebase, [14:14] doing things in the app, trying to [14:15] figure out where the thing you're doing [14:17] exists in the code. in building the [14:19] mappings and the relationships between [14:20] those things because you can't keep the [14:22] whole codebase in your brain. Don't [14:24] expect the model to do it. They don't [14:25] work that way. So, we've established [14:27] that giving it your whole codebase is [14:29] bad. But what is good context? This is a [14:33] more complex question that you'll get [14:35] varying answers about. Generally [14:38] speaking, less is best. If you can [14:41] simply describe the thing that is wrong [14:43] and what needs to be done, you can [14:45] usually trust the model to go find what [14:47] it needs to from there. This is actually [14:49] one of the things that makes codeex [14:51] slightly better than Opus right now with [14:52] cloud code is that Codeex will do a very [14:55] good job of reading all of the files [14:57] that might be relevant, only pulling in [15:00] the ones that matter, and then going and [15:02] solving the problem. it is slower as a [15:04] result because it'll spend a ton of time [15:06] looking into each potential file that [15:08] might be relevant, but once it does, [15:10] it'll make a precise change with [15:11] confidence. Whereas a model like Opus [15:14] isn't quite as eager to search the [15:16] codebase and is more eager to change it. [15:19] As soon as it finds something that it [15:20] thinks is the problem, it immediately [15:22] goes into editing mode. So with [15:24] something like Opus, it is better to put [15:26] a little more time up front to say the [15:29] problem is here. Don't touch this. Don't [15:32] touch that. Just solve here. This is [15:34] also an important thing to do with your [15:36] claude MD file or agent MG if you're [15:39] doing other tools. I'm very thankful [15:41] everyone else is following a standard [15:42] and anthropic is just special snowflake. [15:45] They are what they are. They'll never [15:46] embrace standards that they didn't make. [15:47] It's how they are. One of the most [15:49] interesting things I learned about the [15:50] Claude Code team is that whenever they [15:52] notice the model doing something poorly [15:54] in the Claude Code codebase, they [15:56] immediately go and change the Claude MD [15:59] file to help steer the model in the [16:01] right direction. I actually just made [16:03] some changes to the T3 chat codebase for [16:05] the same reason. I noticed a couple [16:07] mistakes that were happening as a result [16:10] of this file not steering the model in [16:13] the right direction. There are certain [16:14] things that we do every day as devs that [16:17] the model probably shouldn't. One of the [16:18] ones that pisses me off the most is PNPM [16:21] dev. I don't want the model to run a dev [16:24] server. I have a dev server running. I [16:26] want the model to make changes to know [16:28] that I have a dev server running and to [16:29] just run a type check when it's done [16:31] with its work. So I updated this file [16:34] where I describe the PNPM scripts and [16:36] specify now note don't use this unless [16:40] otherwise told to. and all of a sudden [16:42] the model randomly running dev commands [16:44] stopped entirely. I also wanted it to be [16:47] able to check the types from convex [16:50] because if I wasn't running a dev server [16:52] or it was running in a cloud environment [16:53] where I didn't have the dev server [16:55] running, it would make changes to the [16:56] convex files and then not have the types [16:58] update and then get really confused [16:59] about the type errors. So I added [17:01] another command here, pnpm generate, and [17:03] told it generate convex types after [17:05] schema changes. Now it knows when it [17:08] makes a schema change, it should run [17:10] this command and then the types will be [17:12] correct again. Constantly changing this [17:14] file based on the mistakes you see it [17:16] making seems like you're babysitting the [17:19] thing. It feels like you're doing way [17:20] more work than you have to. I promise [17:22] you it's not. This took 5 seconds and [17:24] it's already saved me hours. And it also [17:26] helps with building the intuition for [17:28] what these can and can't do. Funny [17:30] enough, when I was at Twitch, I [17:32] experienced very similar things even [17:33] though this was preai. When I onboarded [17:36] into the codebase for the web app when [17:38] we were rebuilding it in React, I was [17:39] still learning React and TypeScript. I [17:41] put up my first two PRs and had some [17:44] dumb mistakes in them. Another engineer [17:46] who was much more experienced hit me up [17:48] and asked me, "Hey, not trying to pick [17:50] on you or anything. Why did you do this [17:52] though? Why did you write this this [17:53] way?" So, I showed him the page in [17:55] Twitch's docs that made me think [17:57] incorrectly that I should do things a [17:59] specific way. I don't remember what it [18:00] was. I just remember this order of [18:01] events. I linked him the docs page that [18:04] led to me doing it this way. He said, [18:06] "Oh, I see why it would make you think [18:08] that. I'm going to go fix it." And [18:10] before my PR even merged, they had [18:12] updated the docs to make it less likely [18:14] that the next person would make the [18:16] mistakes I made. The issue with AI is [18:18] that it doesn't remember things when it [18:20] makes mistakes. That's your job. You [18:23] have to build the memory for it. Not in [18:25] the sense that you have to manage every [18:27] single parameter inside of the model, [18:29] but in the sense that when you notice it [18:30] making stupid mistakes the way a new [18:32] engineer might in your codebase, you [18:34] need to document that so it's less [18:36] likely to happen to the next person [18:38] because the AI agent is a new engineer [18:40] every time it runs. The role of this [18:42] file is to take this really skilled [18:44] engineer who just joined your company [18:46] and make sure they know all of the [18:47] things that are special about your [18:49] company and your codebase. If you're [18:50] copy pasting a cloudmd file from [18:52] somebody else or some template repo [18:54] somewhere, you're doing it wrong. Even [18:56] if you're running the init command that [18:58] a lot of these CLIs have, you're [18:59] probably doing it wrong. The role of [19:01] this file isn't to describe every single [19:03] thing about the codebase. It's not just [19:05] docs. It is specifically a gotchas pile [19:08] almost like listing the things you've [19:10] seen it do wrong to steer it away from [19:13] that. This file should start really [19:15] small and simple and slowly have small [19:18] additions added and tuning done to it to [19:21] steer the model away from the things you [19:23] don't want it to do. And this file is [19:25] always included as context. So it's [19:27] basically guaranteeing that everything [19:30] in it will or won't happen depending on [19:32] how you describe it. So less code, more [19:34] markdown. This is one of the most [19:35] important pieces for the big code bases, [19:37] too. By the way, if you're noticing [19:39] problems with agents in really big code [19:41] bases, the problem isn't the size of the [19:43] codebase so much as the number of [19:46] opinions and expectations that have been [19:48] encoded. As a result, as the codebase [19:50] gets bigger, the things that are weird [19:52] about that codebase increase, too. Your [19:54] expectations around how people operate [19:56] in that codebase grow. So, you need to [19:58] encode those. Another fun side effect of [20:00] this is I've noticed when I look at a [20:03] new codebase and I read through this [20:04] cloud MD or agents MD file, I learn a [20:07] lot about how the team builds and how [20:08] they think about the codebase. It's [20:10] actually a really useful way to learn. [20:12] Honestly, context management is complex [20:14] enough that I could do a whole series of [20:16] videos on it. It's not complex in the [20:19] sense that it's like super super hard to [20:21] figure out. It's complex in the sense [20:23] that there are many different solutions [20:24] to the weird set of problems and you can [20:27] for the most part ignore them and be [20:29] fine. So don't overthink this one. Don't [20:32] try to hack this one either. It's a [20:34] common mistake. That's kind of that [20:35] stupid repo mix thing I was sharing [20:36] earlier. It's kind of the problem with [20:38] things like this. It's trying to hack [20:40] the context instead of use the context. [20:43] And hacking the context doesn't work. [20:46] Don't try to work around this reality. [20:48] Try to embrace it. recognize the fact [20:50] that these problems shouldn't require [20:53] knowing everything about the codebase. [20:55] Again, to go back to the selecting the [20:56] right problem thing, if you had a really [20:58] smart friend that you were giving access [21:00] to this codebase, what information [21:03] should they have in order to solve the [21:04] problem that you're describing? Assuming [21:06] they know nothing about the codebase, [21:08] what is the smallest amount of info you [21:09] can give them for them to meaningfully [21:11] understand and solve the problem? That [21:13] is what you should give to the agent. [21:15] We'll probably talk a little more about [21:16] context management in other pieces here [21:18] and I might even do more videos about it [21:20] in general in the future. But you have [21:21] enough of an idea now about the common [21:23] mistakes I see. Hopefully that'll help [21:25] out enough that you can get more value [21:27] from these tools. Speaking of which, one [21:29] of the biggest problems I see using [21:32] outdated tools or even worse holding [21:35] your perspective based on experiences [21:37] you had with outdated tools. Let's be [21:40] real. If your perspective on how good AI [21:42] is at coding is based on your experience [21:44] using sonnet 3.5 in wind surf in the [21:48] early days of agentic coding where you [21:50] handed it the whole codebases context [21:52] and didn't have a cloudmd or an agent md [21:54] file to help it out. You're in a [21:56] different world than us. That's [21:58] legitimately like comparing notepad.exe [22:01] to vim. Like the gap here is insane. To [22:05] be fair to the people who feel this way, [22:08] this is not just a cope. This is a [22:11] learned behavior from a history of doing [22:13] stuff with software and in the [22:15] development world. If you tried adopting [22:17] GraphQL right when it started getting [22:19] hype and it went really poorly for you [22:22] and then you tried again four years [22:23] later, it didn't get much better. And [22:26] that's the case a lot of the time. There [22:27] are things that have a lot of hype that [22:29] you go try, they just don't work well [22:32] for you. then going back a few years [22:34] later is very unlikely to change [22:36] anything. That has been the case in the [22:38] dev world for a while. If you tried [22:39] React a couple months after it dropped [22:41] and you just hated the syntax, the way [22:43] it worked and no matter how hard you [22:44] tried, you couldn't get over it, coming [22:46] back a year later isn't going to be any [22:48] better. Cuz like as much as React has [22:50] changed, it hasn't changed in ways that [22:52] would turn a skeptic into a supporter. [22:54] It just improved in ways that take [22:55] supporters and make them more powerful. [22:57] So, if you used these tools two years [22:59] ago and got nothing out of it and you've [23:02] based your way of evaluating things on [23:04] how you've evaluated them your whole [23:05] lives, chances are this thing isn't [23:08] going to be any better. I understand why [23:10] you're avoiding it. Things are changing [23:12] absurdly fast. I had problems that [23:15] nothing in the AI world could solve 6 [23:17] months ago that started being solved [23:19] three months ago. Like, they went from, [23:21] "Oh, haha, AI can never solve this," to, [23:24] "Wait, what the fuck?" in literally a [23:26] 3-month window. That is insane. This [23:28] level of oh [ __ ] has not happened in the [23:30] software dev world almost ever before [23:33] where a thing could go from incapable of [23:35] something to beyond capable at the exact [23:38] same thing in months. So if you're not [23:40] using the state-of-the-art stuff, you [23:42] don't know where it is today. And I know [23:44] a lot of you guys aren't using it [23:46] because I've seen the numbers. Not the [23:48] numbers of how much you're using certain [23:50] models or how many tools you're using [23:51] and when you use them. I'm just talking [23:53] about my sub count. Did you know less [23:55] than half of you guys are subscribed to [23:57] my channel? How are you expecting to [23:58] keep up on all of this [ __ ] I can [24:00] barely do it and it's my job to. If you [24:02] want good summaries of what's going on [24:03] and what the hot [ __ ] is and what you [24:05] can learn from it and whether or not to [24:06] try it, hit that red button. It costs [24:08] you nothing and it helps you a ton with [24:10] keeping up. I guarantee you that that [24:12] engineer on your team that is way ahead [24:14] that is killing it with all these AI [24:16] tools, they're probably subscribed and [24:17] you're not. Fix it. Anyways, I could sit [24:20] here and tell you what the hottest tools [24:22] are right now, but I want this video to [24:24] stay useful for a while, so I'm not [24:27] going to do that. You can go check [24:29] Twitter or you can go to my channel and [24:31] sort by most recent and get a rough idea [24:33] of what is good enough today. I will say [24:36] chances are it's probably not co-pilot. [24:40] And also important thing, if your [24:42] company doesn't let you try the new cool [24:45] hot things, this is harder to do. And I [24:47] actually do empathize with that. If you [24:49] applied to get co-pilot approved at the [24:51] start of 2024 and it finally got [24:54] approved halfway through 2025 and [24:57] everybody's moved over to Cursor and [24:58] Cloud Code and Codeex, you now have to [25:00] wait a year and a half for it to be [25:01] approved again. That sucks. There are [25:04] reasons that you're on outdated tools [25:05] that aren't your fault. And if that's [25:07] the case, first off, I'm sorry. Second [25:09] off, you should start ignoring those [25:10] rules and doing it anyways. If you get [25:11] fired, you have an awesome story. Ask [25:13] forgiveness, not permission. And third, [25:15] you can probably find a better company [25:17] to do these things at because this stuff [25:18] like like don't fall behind because your [25:20] company doesn't let you try new things. [25:22] And you'd also be amazed how many big [25:24] companies are embracing this. There are [25:26] medium-sized companies like Sentry that [25:28] are big enough to buy Super Bowl ads [25:30] that will let you bring in whatever [25:32] tools you want. And then there are giant [25:34] companies like Microsoft that will also [25:36] do the same. I can't tell you how many [25:37] of the startups I've invested in that [25:39] have Microsoft as a customer because [25:40] they're actually willing to try out [25:41] these new tools. I think that's really [25:43] cool, especially because they compete [25:44] with a handful of them. But then there's [25:45] places like Amazon where they are [25:47] constantly being told they can't use [25:49] these tools because they really want to [25:51] make their crappy vibe coding, PRD [25:54] building. I think it's called Kira is [25:56] their VS Code fork. There are a lot of [25:58] people at Amazon that are forced to use [26:00] that and can't use better tools because [26:02] Amazon really wants the information in [26:04] order to make their tool better. So that [26:06] sucks. And if you're in one of those [26:07] boats, figure it out. I'm sorry. It [26:09] sucks. I get it. But the fact that your [26:12] company is forcing you to use worse [26:13] tools does not mean the tools are [26:15] useless. The ones you're using might be [26:17] useless. And I know you guys don't know [26:19] many people using Kira, but all the Kira [26:21] users I know are my ex-coorkers that are [26:22] still on Amazon and Twitch cuz they're [26:24] all forced to use it. Yeah. So if [26:26] they're skeptical of AI tools, I get it [26:29] because Yeah. But you should find [26:32] opportunities to try the best things. I [26:34] am regularly amazed. Like when I started [26:36] using Cloud Code with Opus, after using [26:38] cursor with Opus, my brain just kept [26:41] opening to the things you could do with [26:42] it. Next common mistake, and I really [26:45] hope people aren't making this, but [26:46] honestly, I I know you are because I've [26:49] seen it a whole bunch and I've made this [26:51] mistake myself too in a handful of [26:52] places. Broken environments. This one is [26:55] so common. Good example here. I've been [26:58] in code bases that are mono repos where [27:00] if you open the repo at a root level in [27:03] your editor, the type system breaks. you [27:05] have to open the sub packages in order [27:07] for the TS config to work. That's a [27:09] broken environment. If I can't in the [27:12] root of the project run a type check [27:14] command and know that everything in the [27:16] project is handled properly, your [27:18] environment is broken. And this is a [27:21] very common thing. I've seen this so [27:22] many times. If your type check involves [27:25] a CD, an actual directory change, your [27:28] environment is broken. And you can go [27:30] try to teach the agent about all of [27:32] these things through the MD files [27:34] through agent MD or cloud MD, but it's [27:36] much better to just fix your [ __ ] [27:37] environment. I just had this in a Vibe [27:39] coding project. I was working in the [27:41] Vibe canban codebase and they had a [27:43] mistake in their ESLint config that [27:45] meant that whenever you opened a file in [27:47] the editor and you were in the root, not [27:49] the subpackage, every single file [27:52] wouldn't be type checked and threw an [27:54] error on the first line because it [27:56] expected you to be using it at the root. [27:58] fix those things. Not just for the [28:00] agents, but for your co-workers, too. If [28:02] I open up your project in my codebase [28:04] and I go to a file and it's getting a [28:06] type error from main because your [28:09] configuration is bad, fix it. A good [28:12] rule of thumb here is if you have a good [28:14] engineer that has never touched your [28:16] codebase and you ask them to spin up on [28:18] it and they run into these types of [28:20] things, you shouldn't tell them why [28:21] they're wrong or how to fix it. You [28:23] should go fix the codebase so these [28:24] don't happen at all. Especially because [28:26] again these agents effectively have [28:28] their memory wiped every time you run [28:30] them again. So if you have this type of [28:32] type error where every file has a bad [28:35] eslint config every single time the [28:37] agent runs it's going to see the error. [28:39] It's going to try and fix the error. [28:41] It's going to realize the error isn't [28:42] because of their changes. And I can tell [28:43] you how many times we've seen this. The [28:45] agent finds the file that it needs to [28:47] touch to fix the problem. It makes the [28:49] changes. It fixes the problem. It [28:50] verifies the fix. It runs the type [28:53] check. It gets a type error. It freaks [28:55] the [ __ ] out. It tries a billion random [28:57] things to fix it. Eventually, it reverts [29:00] the change. It sees the same error. It [29:02] then says to itself, "Oh, I guess this [29:05] error already existed and is unrelated [29:07] to my changes. It reapplies the change [29:09] and then it finally ships." And then you [29:11] run the agent with a different problem [29:12] and the exact same thing happens over [29:14] and over and [ __ ] over. If you tell [29:17] an AI agent about a ghost, it will chase [29:20] it forever. You need to get rid of the [29:22] ghosts. You need to get those skeletons [29:24] out of the closet. And do you know [29:26] what's really crazy? If you see these [29:28] issues, you can tell the agent about [29:30] them and it'll fix them really well. Do [29:32] you know how I fixed the problem in the [29:33] Vibe canband codebase? So my issue is [29:35] when I went in a file, I had a type [29:37] error. So what I did to solve this [29:39] problem is I didn't feel like dealing [29:40] with the TS config [ __ ] I literally [29:42] clicked in cursor the little button that [29:44] appears that says oneclick fix this [29:47] error. Ask the agent to fix the error. I [29:50] clicked that. It put this in the text [29:52] here. For this code present, we get the [29:54] following error. Parsing error. Cannot [29:56] read the file tsconfig.json. Fix it. [29:59] Verify. And then give a concise [30:00] explanation. And it did it. It realized [30:03] the problem is that the project expected [30:05] the tsconfig to be at dot slash. So [30:08] where we currently are, but it's a monor [30:09] repo, so it doesn't actually exist there [30:12] if you're using it at the top level. If [30:13] I had opened my editor in the front-end [30:16] folder, it would have been fine. But I [30:17] opened my editor at the root, so it [30:19] wasn't. So what we need to do is use the [30:22] path helper to figure out where we are [30:25] so we can actually link it correctly. [30:27] And it did this with no issues in one [30:29] shot. I then pushed it up, got it [30:32] merged, and now the agents will behave [30:34] better when they work in this codebase. [30:35] So the very problem you are giving [30:37] agents can actually be solved by those [30:39] very same agents. You need to fix your [30:41] environments. I I have played in far too [30:45] many big code bases that have lots of [30:47] problems like this. You can fix them [30:48] yourself. You can fix them with an [30:50] agent, but please [ __ ] fix them. The [30:52] next time I see somebody who can't get [30:54] AI to solve their problem because their [30:56] environment is broken, I'm going to flip [30:58] a [ __ ] Obviously, if you have type [31:00] errors everywhere because you suck at [31:01] setting up the environment, things are [31:03] going to break. Fix it. Okay. The next [31:06] mistake I see is actually one I'm scared [31:08] about because I have a feeling this [31:10] video might contribute to it. I almost [31:12] want to put this earlier, but I'm going [31:14] to put it here because this is where it [31:15] starts to matter. MCP Helen over [31:17] configuration. I cannot tell you how [31:19] many times I have seen this. People [31:22] spend so much time trying to set the [31:24] agents up to succeed that they lose the [31:27] plot. They bloat the context with dozens [31:30] of MCP servers and then nothing works. [31:33] As you know, I've been using Cloud Code [31:35] a lot. Want to see all the MCP servers I [31:37] have configured? Oh, literally none. [31:40] Zero. Same deal with cursor. Same deal [31:43] with every other tool I use. Stop [31:45] loading up your agents with these [31:47] useless [ __ ] things. Just don't. It's [31:50] so common, too. I've seen this a ton. [31:53] Chad immediately says, "Now, let's see [31:54] your skill section." Okay, I have one [31:57] skill. The front-end design skill. Do [32:00] you know what the front-end design skill [32:01] does? It is a single markdown file that [32:04] tells the model to not make AI slop. It [32:07] literally says exactly that. This skill [32:10] guides creation of distinctive [32:11] production grade front-end interfaces [32:13] that avoid generic AI slop aesthetics. [32:15] Implement real working code with [32:17] exceptional attention to aesthetic [32:18] detail and creative choices. Never use [32:20] generic AI aesthetics like overused font [32:23] families, clichéed color schemes, [32:25] particularly purple gradients on white [32:27] backgrounds. This is all that skill is. [32:30] That's all I have for configuration in [32:32] cloud code. I have a custom runner CC [32:36] that is a oneline addition to my zish [32:39] file that adds a specific environment [32:42] variable in front to hide my email [32:45] address and also appends the [32:46] d-dangerously [32:48] skip permissions. I am tempted to put a [32:51] piece of advice that is always use mode [32:54] and dangerously skip permissions, but [32:55] you'll get there yourself. I don't need [32:57] to tell you this. Stop adding all of [32:59] this [ __ ] to your stuff, though. skills [33:01] are literally just markdown files and [33:03] they're often way longer than they [33:04] should be. They are for the most part [33:06] context bloat and context rot. Avoid [33:08] them unless they solve very specific [33:10] problems that you are specifically [33:11] having after you've tried solving them [33:13] other ways. And I've talked enough about [33:15] why MCP servers suck. You should already [33:17] understand that by now. If not, go watch [33:18] my other videos. If you are skeptical of [33:21] AI being useful in your day-to-day work [33:23] and you are touching MCP at all, you are [33:25] [ __ ] up. If you don't see the value [33:27] in these things yet, don't configure [33:29] [ __ ] Don't go into MCP. Don't add a [33:32] whole bunch of skills. Don't add [33:33] hundreds of rules to cursor. Make subtle [33:35] adjustments to your cloud MD and your [33:37] agent MD file. And try to give a bit [33:39] more clarity in your prompt as you talk [33:42] to the model. One more thing, and I feel [33:44] bad calling this out cuz it seems like a [33:45] pretty good and very well-intentioned [33:47] project. Oh, my open code. This seems [33:50] fine if you're already deep in open [33:53] code, getting a lot of value from it, [33:56] and you want to see what it looks like [33:58] to thoroughly configure it. That's the [34:00] point of oh my open code. But we're [34:03] already seeing in chat, oh my open code [34:05] is so bloated, slop. Yeah, [34:10] it's a mixed bag at best. You don't [34:12] solve problems with AI coding tools by [34:15] adding more things to them. More [34:17] features, more MCPs, more plugins, more [34:20] skills, none of that's going to make you [34:22] go from this thing is useless to this [34:24] thing is useful. If anything, they're [34:25] going to make it feel worse. I promise [34:27] you if you take somebody who isn't sure [34:29] about these tools and you clone them, [34:32] you give one version of this person [34:34] stock cla or codecs and tell them to [34:36] solve problems, you give the other one a [34:39] super customized oh my open code setup [34:41] with all the fancy MCPS, all of the [34:43] skills built in and everything, the [34:45] person with stock codec is going to have [34:46] a much better time and going to enjoy it [34:47] a lot more. These things can be useful [34:50] in small doses once you have specific [34:52] problems that they solve, but nobody [34:54] talking about them is there. Everybody [34:56] talking about this [ __ ] is just AI [34:58] maximalist trying to squeeze every [35:00] single possible thing. It's the same [35:02] people who would buy Android phones [35:04] because they cared so much about every [35:05] single line in the specs. I'm saying [35:07] this cuz I was one of those people. I [35:09] used to tell you very excitedly how many [35:11] gigahertz were in my phone. I was so [35:14] hyped when I got my first dual core [35:15] phone back in the day. I don't even know [35:17] how many [ __ ] cores my iPhone has. I [35:19] have literally no idea. I know it's [35:21] fast. I know the GPU is even faster, but [35:23] I have no [ __ ] idea what the specs of [35:25] this phone are. I have better [ __ ] to [35:26] do, and you do, too. Stop adding all [35:28] these [ __ ] tools. The tool maximalist [35:31] people are the same ones who care too [35:32] much about specs when your grandma's [35:34] buying a computer. It's the exact same [35:36] people. Ignore them and go get real work [35:38] done. I feel like Pete comes up in [35:40] almost all of my videos at this point, [35:42] but there's a reason why. He is building [35:44] an absurd amount of things in parallel, [35:47] all for free and open source, because [35:48] he's already made his money, and he's [35:50] killing it. He's doing really, really [35:52] cool [ __ ] And he's vibe coding almost [35:54] all of it. And he's vibe coding so hard [35:56] that he's putting out 500 plus commits a [35:58] day regularly. So he must be maximizing [36:01] all of this stuff, right? All the [36:03] customization, all the MCPs, all the [36:05] skills, all the things. He's probably [36:07] running his own forked version of [36:08] whatever. No. Ready to see what he uses? [36:12] He uses stock codecs. Here is his config [36:16] for codecs. Defaults to GBD 5.2 Codeex [36:19] high. defaults to high reasoning, has a [36:21] limit for how many tokens it can output, [36:23] has a model autocompaction limit, and [36:25] then in the features, ghost commits are [36:27] off, unified exec is true, free form [36:29] patch is true. It's just a tool they [36:32] have for doing patching better. I think [36:33] it's on by default now. Web search [36:35] request is true because for whatever [36:37] [ __ ] reason, the search tool for [36:39] searching the web is off by default in [36:41] codeex. This is like the only part in [36:43] here that really matters. Skills is true [36:45] cuz he's playing with skills. And then [36:46] the shell snapshot is true. I don't even [36:48] know what that does. This allows the [36:49] model to read more in one go. The [36:51] defaults are a bit small and can limit [36:52] what it sees. It fails silently, which [36:54] is painful and something they'll [36:55] eventually fix. Also, web search is [36:57] still not on by default. Yeah, the [36:59] unified exec apparently replaces using [37:01] T-mucks for handling execution. Not [37:03] much, but like that's it. No crazy [37:05] plugins, no crazy forks, no absurd [37:08] orchestration stuff. Keep it simple. If [37:11] the thing isn't useful in the simple [37:13] form, it's not going to magically get [37:14] useful by adding a bunch of [ __ ] Please [37:16] stop. I actually really like this [37:18] article he did called just talk to it. [37:21] This is as people realize over time what [37:24] they can do with these things. They [37:26] start with please fix this and as they [37:28] get more in they do crazy things. Eight [37:30] agents running at the same time, complex [37:32] orchestration with multiple checkouts, [37:34] chaining agents together, custom sub [37:35] agent workflows, libraries with tons of [37:38] slash commands, full stack features [37:40] being built in MCPs, all that [ __ ] And [37:42] once you get through all that, you [37:43] realize the best thing to do is just [37:45] say, "Hey, look at these files." and [37:46] then do this thing. Life is much better [37:48] when you realize that's all you need. [37:50] That said, even he admits he'll [37:52] sometimes just take a screenshot of a [37:54] thing that's wrong, paste it to the [37:55] model, and say, "Fix this," and it will. [37:58] Last important piece here, plan mode and [38:00] iteration. A very common mistake I will [38:03] see is someone asks the agent to do [38:06] something, usually compounded with the [38:07] other mistakes, like with a bunch of [38:09] weird tools on, nowhere near enough [38:10] context on the problem, and then it [38:12] struggles. And rather than fix the [38:15] context, they keep asking more things [38:18] and adding on. They keep appending more [38:21] stuff as the mistakes compound. And [38:23] remember, this is all autocomplete. So [38:26] if the history that it has is a bad [38:28] instruction, incorrect implementation, [38:30] and then a correct instruction, there is [38:33] still more bad information than good. If [38:35] the whole thing is autocomplete based on [38:37] the history, having a bunch of bad [38:40] things in the history is going to result [38:42] in bad output. even put a good thing at [38:44] the end. Obviously, it can work through [38:46] this and there are plenty of examples of [38:48] it doing it. But it is significantly [38:50] better to not do this this way. When you [38:53] notice that the output came out bad [38:55] rather than try to fix it with a better [38:57] input being appended, revert, go back, [39:01] make the better input the start. Because [39:03] if you have a better input as the start, [39:05] the likelihood that the output is better [39:07] too is much much higher. The more good [39:12] context exists in your history, the more [39:14] likely the next thing is good as well. [39:16] Obviously, there's a limit here. If you [39:18] have too much context, chances are the [39:20] context has become less relevant, which [39:21] makes it worse, which makes the output [39:23] worse, too. Fix it. And you know what? [39:25] One of the best ways to fix this is plan [39:28] mode. Plan mode is great because instead [39:31] of the bad input resulting in a bad [39:33] output, the output will be a confused [39:36] output. In fact, it'll be confused [39:39] questions where instead of it doing a [39:41] whole bunch of things it shouldn't, it [39:42] will add a little bit of additional [39:45] context. That is, hey, I'm not sure [39:47] about these three or four things. Can [39:49] you answer these questions for me? And [39:51] you might realize, oh, I should have put [39:53] that at the start. And you can kill it [39:54] and restart if you want. Or you can [39:56] answer the questions. Those answers to [39:58] those questions result in better [40:02] answers. And then the model can output a [40:05] good plan. [40:06] And now the majority of your context and [40:09] the majority of your history is useful. [40:11] So the likelihood that it builds [40:13] something good is much higher. This is [40:16] why plan mode is good. It gives the [40:18] right relevant context instead of [40:20] handing the model your whole codebase or [40:22] a bunch of tools to find things in the [40:23] codebase. The plan mode creates a [40:26] perfect prompt almost. The goal of plan [40:29] mode is to write the exact thing you [40:32] want to hand to the model that it can [40:33] then go use to solve the problem. But a [40:36] really important piece here is what [40:37] happens if it goes wrong. If you make [40:40] this plan, you think it's good, you run [40:42] it with the model and it comes out with [40:44] a bad output. If it's just a few things [40:46] that are wrong, sure, tell it that. Tell [40:49] it to go fix those small things and it [40:51] will. But if it gets the whole thing [40:53] wrong, reflect on why. Read a little bit [40:56] of what it did. Read the reasoning [40:58] traces for why it made the change you [41:00] don't like or the thing that you don't [41:01] want. If it's something that was wrong [41:03] with the plan, go fix the plan. If it's [41:06] something that was wrong with the [41:07] understanding of the code base, go fix [41:09] your claw MD or your agent MD file. This [41:11] is the delicate balancing act you have [41:13] to do. And it's an intuition you build [41:14] as you do it more. You'll write a plan, [41:17] the model will start executing on it. [41:18] You catch something it's doing wrong and [41:20] you'll just know where to go to fix it. [41:22] Just to go back to my example earlier in [41:23] the T3 recheck codebase when I ran a [41:26] plan and it got the code right, but then [41:28] it ran a dev server and it couldn't [41:29] verify things because the dev server [41:31] went to the wrong port, caused errors, [41:32] and broke [ __ ] I knew the solution [41:34] wasn't to add to the plan. Don't run dev [41:37] commands. I know that cuz I used to do [41:38] that. Instead, I went and put it in the [41:41] agent MD file and then cloned it as the [41:42] cloud MD file and the problem [41:44] disappeared. Figuring out where to put [41:47] the thing to stop the bad behavior is an [41:49] intuition you'll build and you'll build [41:51] it way quicker than you think. The same [41:53] way you build intuitions about how to [41:54] use a tool, right? Like in React, you [41:56] don't put hooks after an early return. [41:58] You just get why after you use it for a [42:00] bit. Those types of things are [42:02] intuitions you build as you use the [42:03] tools, but you have to build them, not [42:06] just keep saying fix it over and over [42:07] and over again. Generally speaking, if [42:10] you're not oneshotting things often, [42:12] that's because there are problems in [42:14] your prompting, problems in your context [42:16] management, problems in your agent MD [42:17] and cloudmd files, problems in the [42:19] environment you have given the model to [42:21] solve things in. And as you catch [42:23] mistakes on the other side, there are [42:25] more solutions there, too. If you notice [42:26] the model makes code that works, but the [42:28] types are always erroring, go give it a [42:30] command to check the types so that it [42:33] knows they are good. A lot of things now [42:35] have type-checking plugins built in with [42:36] the LSPs. I've had mixed success with [42:38] those personally, but giving it a [42:40] type-checking command can solve almost [42:42] all of these problems. Don't [42:44] overengineer it. Just identify the [42:46] common mistakes and fix those in the [42:48] agent MD or in a given plan. If a [42:50] mistake happens, don't tell it to [42:52] correct it. Go back and adjust the plan [42:54] and rerun it. So, to wrap things up, I [42:57] want to go back to this tweet from Adam [42:58] in the back and forth we had on it. He [43:01] couldn't get it to solve this error. He [43:03] had done a little bit of work. He found [43:04] the file that it is likely coming from. [43:07] He knew it was from this provider tsx [43:09] file, but he couldn't figure out why. He [43:12] also wasn't very clear in the ask. He [43:14] had an ask mode and ask it if it could [43:17] help him find the problem. It's not a [43:19] big deal. Models can work through that [43:20] type of thing, but being more concise [43:22] and specific about what you want from it [43:24] helps a lot. The bigger problem here [43:26] though is that he didn't know what the [43:27] problem was and hadn't found the right [43:29] context for it. The feedback I gave was [43:32] that he should copy paste the exact [43:33] error. give it any useful context like [43:35] does it happen to logged in or logged [43:37] out users other information that could [43:39] make it easier to root cause the bug [43:41] also with a case like this because it's [43:43] happening in the browser a tool that I [43:44] can use to verify its fixes like [43:46] playright or access to the browser [43:48] directly can be really helpful I also [43:50] specified put in less qualifiers instead [43:52] of can you help me find it it's what is [43:54] the cause of this error and if you think [43:55] the model can fix it just skip straight [43:57] to build mode turns out the reason that [44:00] Adam couldn't do this is because he [44:01] didn't know that there was an exact [44:03] error. His immediate response was there [44:05] is no exact error. Is there isn't the [44:07] only framework that manages to actually [44:09] give you a diff when the HTML is broken [44:11] with a hydration error? And this is [44:13] again to go back to the start why he was [44:15] using the model. He was using a model to [44:17] solve this problem because he had tried [44:19] all of the other things he knew of and [44:21] they didn't work. So he was using the [44:22] model for the first time as a last [44:25] fallback and that's not going to work [44:27] great. If you're only using these models [44:29] because you don't have enough [44:30] information to solve the problem, you're [44:31] not using them right. And what made this [44:33] one really funny is that I effectively [44:35] played the role of what the you wanted [44:37] the model to do here. I told him that [44:40] this was added in React 19 that you have [44:42] traces that now show you in React where [44:45] the hydration error occurs. Ah, I'm an [44:48] idiot. The diff was right in front of my [44:50] face. What's crazy though is that it [44:52] appeared to understand perfectly what a [44:54] hydration error is, but it diagnosed the [44:57] cause wrongly, even absurdly. Almost [45:00] exactly backwards. Yeah, because it [45:02] didn't have enough context because you [45:04] as well didn't have enough context. And [45:07] once he found the right error and handed [45:09] that to the model, sure as [ __ ] it [45:12] could solve it perfectly. So again, if [45:14] you apply the lessons from what I just [45:17] explained here, you have to select the [45:19] problem well. You have to not select a [45:21] problem that you don't know how to [45:22] solve. Ideally, you have to give the [45:24] right context and find the context if it [45:27] doesn't exist. Help the model work on it [45:29] with you if it doesn't and you want to [45:30] build it. He wasn't using outdated [45:32] tools. All that stuff was fine. The [45:34] environment was kind of broken in the [45:35] sense that there was no way for it to [45:36] know the right way. But if he was to go [45:39] add everything, he would quickly end up [45:40] in MCP hell. So finding the right [45:41] balance here, a hard thing to do, but if [45:43] you do it right, it's great. But with [45:44] the right application of a handful of [45:46] these tips, he would have had a much [45:47] better time. And after that back and [45:49] forth, he is now enjoying these tools [45:50] much more. I hope I've done an okay job [45:52] of explaining why I like these things [45:54] and the mistakes that I've seen others [45:56] having with them. And maybe, just maybe, [45:58] you'll be a bit more of a vibe coder as [46:00] a result. At the very least, you have a [46:02] whole new set of things to flame me for [46:03] in the comments. Let me know how you [46:04] feel about this one. And until next [46:06] time, peace nerds.