Anthropic Study Shows AI Makes Devs Dumb - CourtneyR.dev's 2nd Brain

Anthropic Study Shows AI Makes Devs Dumb ![rw-book-cover](https://i.ytimg.com/vi_webp/ZINQTR6H5dI/maxresdefault.webp) 🔗 **Source:** [Anthropic Study Shows AI Makes Devs Dumb](https://www.youtube.com/watch?v=ZINQTR6H5dI) > [!note]+ Document Notes > tags: [ai-coding, developer-skills, anthropic-study] > author: Theo - t3․gg > source: YouTube > reference: [https://www.youtube.com/watch?v=ZINQTR6H5dI] > date created: 2024-12-19 10:30 > > # 🏷️ Title: Anthropic study shows AI makes devs dumb >  ## 🧰 Action Inbox - [H1] Review Anthropic's full paper on AI impact on skill formation — study methodology #research - [H2] Implement Code Rabbit AI code review in development workflow — productivity tool #devtools - [H3] Create AI guidance document for junior developers similar to HTMX approach — teaching aid #education ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [Anthropic Paper on AI and Skill Formation](URL) - [Code Rabbit AI Code Review](https://coderabbit.ai) ## 📝 Summary - Anthropic published research showing AI-assisted developers completed tasks 2 minutes faster but scored 17 percentage points lower on comprehension tests (50% vs 67%) - Study involved 52 mostly junior developers using Python's Trio library in unfamiliar web coding environment - Results suggest AI may accelerate productivity while potentially hindering skill acquisition, particularly debugging abilities - Key concerns include cognitive offloading and reduced independent problem-solving development - Time savings measurement questioned due to 30-minute constraint and unfamiliar tooling environment ## 📖 Main Content Anthropic conducted a controlled study examining how AI coding assistance affects developer skill formation. The research divided 52 junior developers into control and treatment groups, with the treatment group using AI assistance for coding tasks involving Python's Trio asynchronous programming library. The study found that while AI-assisted developers finished slightly faster (statistically insignificant 2-minute improvement), they demonstrated significantly worse understanding on post-task assessments. The AI group averaged 50% on comprehension quizzes compared to 67% for developers who coded without assistance - equivalent to nearly two letter grades difference. The largest performance gaps appeared in debugging questions, suggesting AI assistance may particularly impact developers' ability to understand code failures and error resolution. Researchers identified different AI interaction patterns, from complete delegation to conceptual inquiry, with varying impacts on learning outcomes. However, the study's methodology faces criticism for its limited scope: 30-minute tasks in unfamiliar environments with junior developers. Critics argue this doesn't reflect real-world AI usage patterns where experienced developers use familiar tools over extended periods on complex projects. ## 🗒️ Notes #### ⏱️ [00:00](https://youtu.be/ZINQTR6H5dI?t=0) — [H1] Study overview and speed results - Key idea: AI makes developers faster but questions remain about skill impact - Evidence: "AI group finished about 2 minutes faster, although the difference was not statistically significant" #### ⏱️ [01:15](https://youtu.be/ZINQTR6H5dI?t=75) — [H2] Comprehension score differences - Key idea: Significant gaps in understanding between AI and non-AI groups - Evidence: "AI group averaged 50% on the quiz compared to 67% in the handcoded group" #### ⏱️ [02:45](https://youtu.be/ZINQTR6H5dI?t=165) — [H1] Code Rabbit sponsor segment - Key idea: AI code review tools can augment rather than replace human review - Evidence: "Quality of the reviews is great... better than humans at code review because it'll do much more meticulous stuff" #### ⏱️ [04:30](https://youtu.be/ZINQTR6H5dI?t=270) — [H2] Study methodology breakdown - Key idea: 52 junior developers, unfamiliar with Trio library, 30-minute constraint - Evidence: "Recruited 52 mostly junior software engineers... encouraged to work as quickly as possible" #### ⏱️ [06:15](https://youtu.be/ZINQTR6H5dI?t=375) — [H3] Environment setup time critique - Key idea: New environment learning curve invalidates speed measurements - Evidence: "Half the time is going to be spent figuring out the environment... would take me more than 30 minutes" #### ⏱️ [08:45](https://youtu.be/ZINQTR6H5dI?t=525) — [H2] AI interaction time analysis - Key idea: Participants spent up to 30% of time composing AI queries - Evidence: "Several took up to 11 minutes... composing up to 15 queries" #### ⏱️ [11:20](https://youtu.be/ZINQTR6H5dI?t=680) — [H1] Real-world productivity gains - Key idea: Industry studies show consistent 30% improvement year-over-year - Evidence: "Pretty consistent 30% improvement in code output year-over-year for developers who are using AI" #### ⏱️ [13:50](https://youtu.be/ZINQTR6H5dI?t=830) — [H3] Manual retyping behavior - Key idea: Half of AI users manually retyped generated code instead of copy-pasting - Evidence: "Half of the AI using cohort spent most of their time literally retyping the AI generated code" #### ⏱️ [15:30](https://youtu.be/ZINQTR6H5dI?t=930) — [H2] Interaction pattern analysis - Key idea: Different AI usage patterns correlated with different learning outcomes - Evidence: "AI delegation... completed the task the fastest... Progressive AI reliance... scored poorly on the quiz" #### ⏱️ [17:45](https://youtu.be/ZINQTR6H5dI?t=1065) — [H3] High-scoring patterns with small sample sizes - Key idea: Beneficial AI usage patterns identified but with very small groups (n=2-3) - Evidence: "Generation then comprehension... n equals 2... Two people did this in the study" #### ⏱️ [20:15](https://youtu.be/ZINQTR6H5dI?t=1215) — [H1] "Correct code" critique - Key idea: Study assumes single correct solution, which doesn't reflect real development - Evidence: "There is no such thing as the correct code... hundreds of ways to solve most problems" #### ⏱️ [22:30](https://youtu.be/ZINQTR6H5dI?t=1350) — [H2] HTMX professor's AI guidelines - Key idea: Educational approach to AI that promotes learning over cheating - Evidence: "AI agents should function as teaching aids... not by solving problems for them" #### ⏱️ [25:45](https://youtu.be/ZINQTR6H5dI?t=1545) — [H1] Anthropic's acknowledgment of limitations - Key idea: Study authors recognize different contexts for AI usage - Evidence: "AI both accelerates productivity on well-developed skills and hinders the acquisition of new ones" #### ⏱️ [26:30](https://youtu.be/ZINQTR6H5dI?t=1590) — [H1] Skateboarding motivation analogy - Key idea: Early success experiences crucial for long-term engagement - Evidence: "If you can give them a taste of the reward earlier, they will survive longer" #### ⏱️ [30:15](https://youtu.be/ZINQTR6H5dI?t=1815) — [H2] Hope and persistence research - Key idea: Early positive experiences exponentially increase persistence - Evidence: "Rats would swim for more than 60 hours instead of 15 minutes... once they developed a sense of hope" ## 📚 References - [Anthropic Paper on AI and Skill Formation](URL) - [Code Rabbit AI Code Review](https://coderabbit.ai) - [HTMX AI Guidelines](URL) - [Sean's Article Analysis](URL) ## 🗣️ Paraphrase - Anthropic's study found junior developers using AI tools completed coding tasks marginally faster but demonstrated significantly worse comprehension of the code they created - The research methodology has limitations including short time constraints, unfamiliar environments, and small sample sizes that may not reflect real-world AI usage patterns - Different approaches to using AI showed varying impacts on learning, with delegation patterns showing poor comprehension while conceptual inquiry approaches maintained better understanding ## 📰 Newsletter **AI Coding Study Sparks Developer Skill Debate** New research from Anthropic reveals a concerning trade-off: while AI tools help developers work faster, they may be hurting our understanding of the code we create. Junior developers using AI scored 17 percentage points lower on comprehension tests, particularly struggling with debugging skills. The study has limitations - short timeframes, unfamiliar tools, small sample sizes - that may not reflect how experienced developers actually use AI in practice. Industry data shows consistent 30% productivity gains for AI-using developers over time. The key insight? Context matters enormously. AI that helps beginners get early wins and stay motivated could be valuable, while AI that replaces learning entirely is counterproductive. The goal should be augmentation, not replacement. **Practical Takeaway**: Consider implementing AI code review tools like Code Rabbit to catch issues before human review, and create learning-focused AI guidelines for junior team members. Resources: [Learn WordPress](https://learn.wordpress.org/), [Developer Docs](https://developer.wordpress.org/), [WP-CLI](https://wp-cli.org/) ## ✅ Action Items #### DevRel - Developer education: Create AI usage guidelines that promote learning over delegation - Developer experience: Review how AI tools impact onboarding and skill development - Community management: Facilitate discussions on balanced AI adoption practices #### OSPO - Developer Support: Provide training on effective AI tool usage for different skill levels - Strategic Alignment: Monitor how AI adoption affects codebase quality and maintainability #### Community - Contributors: Develop onboarding programs that balance AI assistance with skill building - Maintainers: Consider AI review tools for code quality while preserving learning opportunities #### WordPress - Core: Evaluate AI tools for code review and quality assurance processes - Training: Develop lesson plans addressing AI tool usage in development workflows - Community: Create guidelines for AI usage in educational contexts and skill development - Documentation: Update developer resources with AI tool recommendations and best practices ## 🧪 Hypotheses - AI tools provide greater productivity benefits for experienced developers than beginners due to existing mental models and debugging skills - Short-term comprehension gaps from AI usage may resolve as developers gain more experience with both AI tools and programming concepts - Different AI interaction patterns (delegation vs. inquiry) create different learning outcomes that persist over time - Early positive experiences with AI-assisted development increase long-term persistence in programming careers ### 📊 Methodology - Longitudinal study comparing skill development over 6-12 months between AI-assisted and traditional learning groups - Analysis of code quality metrics and bug rates for AI vs. non-AI generated code across experience levels - Controlled experiments measuring debugging skill retention after different types of AI assistance ### ❓ Questions - How do comprehension gaps from AI usage evolve as developers gain more experience with both programming and AI tools? - What specific AI interaction patterns maximize learning while maintaining productivity benefits? - Do early positive experiences with AI tools increase or decrease long-term programming skill development? ### 🔮 Future Research - Long-term longitudinal studies tracking skill development over multiple years with different AI usage patterns - Analysis of AI impact on different types of coding skills (debugging, architecture, testing, etc.) - Investigation of optimal AI usage guidelines for different experience levels and learning contexts ### 💡 Implications - Practice: Need for balanced AI usage that maintains skill development while leveraging productivity benefits - Policy: Educational institutions should develop guidelines for AI tool usage in programming curricula - Education: Training programs must address both AI tool proficiency and fundamental programming skills - Funding: Investment needed in research on optimal AI-human collaboration patterns for skill development ### 🧠 Learning Styles Assessment Reading level estimate: Grade 10-12 #### 👀 Visual learners - Present: Code examples, performance comparisons, interaction pattern diagrams - Improve: Flowcharts showing different AI usage approaches, visual timelines of skill development #### 🔊 Aural learners - Present: Video format with clear explanations and examples - Improve: Podcast version focusing on key insights and implications #### 🗣️ Verbal learners - Present: Clear analogies (skateboarding), defined terminology, contextual explanations - Improve: Glossary of technical terms, more real-world examples #### 🧩 Kinesthetic learners - Present: Practical recommendations, actionable takeaways - Improve: Hands-on exercises comparing different AI interaction patterns #### 🧠 Logical learners - Present: Statistical analysis, methodology critique, cause-effect relationships - Improve: Decision trees for AI usage, more quantitative comparisons #### 🧑‍🤝‍🧑 Social learners - Present: Community implications, team collaboration aspects - Improve: Discussion prompts for team meetings, collaborative exercises #### 🧘‍♀️ Solitary learners - Present: Individual reflection opportunities, personal development focus - Improve: Self-assessment rubrics for AI usage, personal learning plans #### 🌿 Naturalistic learners - Present: Ecosystem thinking about developer communities - Improve: More connections to natural learning processes and environments ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Clear heading structure, descriptive video content - Improve: More detailed alt text for any visual elements, transcript availability #### Keyboard navigation - Present: Video player supports keyboard controls - Improve: Ensure all interactive elements are keyboard accessible #### Contrast and readability - Present: High contrast video content, clear audio - Improve: Closed captions for better comprehension #### Media and captions - Present: Audio content with clear narration - Improve: Professional captions, downloadable transcript ### 🌍 DEIB Assessment #### Cultural breadth - Present: Technology-focused examples relevant globally - Improve: More diverse geographic examples of AI adoption #### Inclusive language - Present: Generally neutral technical language - Improve: Avoid assumptions about educational background or experience #### Neurodiversity support - Present: Clear section breaks, consistent structure - Improve: Content warnings for sensitive topics (animal research) #### Representation - Present: Mix of junior and senior developer perspectives - Improve: More diverse voices and experiences in examples ### ✅ Logical Validity #### Strong reasoning - Deductive strength: Logical critique of study methodology and scope limitations - Inductive strength: Drawing patterns from personal experience with AI tools - Causal reasoning: Connecting AI usage patterns to learning outcomes - Analogical strength: Skateboarding analogy effectively illustrates motivation and persistence #### Fallacies - Appeal to personal experience: Overreliance on individual anecdotes about AI productivity, but acknowledged as such - Hasty generalization: Study sample size critique valid but could be more precise about statistical requirements #### Tone of voice - Overall tone: Passionate, opinionated but substantive, mixes technical analysis with personal perspective ### 🛡️ DARVO Analysis - Deny: No evidence of denial patterns - Attack: Strong criticism of study methodology but focused on technical limitations rather than personal attacks - Reverse victim/offender: No evidence of victim/offender reversal - Emotional cues and rhetoric: Passionate delivery but grounded in technical arguments and data ## 🎯 Conclusion The Anthropic study reveals important tensions between AI-assisted productivity and skill development, particularly for junior developers. While the research has methodological limitations that restrict its broader applicability, it raises valid concerns about cognitive offloading and the need for balanced AI adoption strategies. The key insight is contextual: AI tools can accelerate experienced developers while potentially hindering skill acquisition for beginners. The solution isn't avoiding AI but using it thoughtfully --- ## Full Transcript [00:00] I think we can finally put the debate [00:01] about how useful AI is for coding to [00:03] rest. It is. We don't know to what level [00:06] yet, but we know it can make you way [00:07] faster as a developer. What we don't [00:09] know yet is the impact on us. How does [00:12] this affect ourselves and our skills and [00:13] the things that we're capable of [00:15] building? Are we running to a future [00:16] where everyone is too dumb about code [00:18] stuff to make changes and actually fix [00:20] things because we just get used to the [00:21] agents doing it? These are actual [00:23] interesting and kind of scary questions. [00:25] And Anthropic is now taking the time to [00:27] research it themselves. They just [00:28] published a new paper. how AI impacts [00:30] skill formation. And it's really, really [00:32] interesting. They took multiple groups [00:34] of developers and had them build the [00:36] same project, some with AI, some [00:38] without, and then checked to see how [00:41] much they understood what they built and [00:42] how well they could answer questions [00:44] about what they built afterwards. And [00:46] the results are fascinating. On average, [00:48] participants in the AI group finished [00:49] about 2 minutes faster, although the [00:51] difference was not statistically [00:53] significant. There was, however, [00:55] significant differences in test scores. [00:57] The AI group averaged 50% on the quiz [00:59] compared to 67% in the handcoded group, [01:02] or the equivalent of nearly two letter [01:04] grades. The largest gap in scores [01:06] between the two groups was on debugging [01:08] questions, suggesting that the ability [01:09] to understand when code is incorrect and [01:11] why it fails may be a particular area of [01:14] concern if AI impedes coding [01:16] development. That is scary. We all might [01:19] be getting much dumber really fast. [01:22] There's a lot to dig into here, and I [01:24] obviously have a lot of feelings about [01:25] it. Some of those feelings are bad, but [01:27] some of those feelings are good. Do you [01:29] know what else I feel good about? [01:30] Today's sponsor. Today's sponsor is a [01:31] product that I use every single day. I [01:33] can only imagine how many bugs I would [01:35] have shipped without it. It's Code [01:36] Rabbit, the AI code review that my team [01:38] chose for T3 chat, ping, and all of the [01:40] other crazy things that we build. We've [01:41] tried all of them, and this is what the [01:43] team settled on. And honestly, I [01:44] understand why. The quality of the [01:46] reviews is great. The integrations are [01:47] great. Its understanding of us is great. [01:50] You tell it you want to do something [01:51] different, and it remembers and does [01:52] things differently just like a human [01:54] would. It's actually better than humans [01:56] at code review because it'll do much [01:57] more meticulous stuff. But I'm not going [01:59] to sit here and say you should replace [02:00] your engineers with an AI for code [02:02] review. You shouldn't do that. You [02:03] should use these tools to augment your [02:05] teams. How many times have you read a [02:06] PR, found obvious mistakes, and then to [02:08] wait a day or two for somebody to go fix [02:10] it? Flip that. Now imagine if instead of [02:12] waiting days for a review, you're [02:13] waiting minutes. You get the feedback, [02:15] you go make the changes, and you don't [02:17] bother a human until you know the code [02:18] is good. This can get down to a much [02:20] more granular level. Why even push until [02:22] you've had the review done? That's why [02:24] they now have a CLI and IDE plugin. [02:26] Whether you're in VS Code, Cursor, or in [02:28] your terminal, you have tools now to use [02:31] Code Rabbit to review your changes [02:32] before you even ship. And if you do try [02:34] out the CLI reviews, you can include [02:36] this in the loop for your codegen. Use [02:38] something like Cloud Code, Codeex, or [02:39] Open Code, add in the reviewer CLI, and [02:42] now when it makes a mistake, it will [02:44] catch it and fix it before you even get [02:45] bothered. It's so weirdly powerful to [02:47] add code review to your loop when you're [02:49] building with agents because the quality [02:50] of the output code just goes up a ton as [02:53] a result. There's a reason everyone's [02:54] using Code Rabbit, even the Linux [02:56] Foundation. Yes, that Linux Foundation. [02:59] Also, Bun, Clerk, Cheg, Ashb, and so [03:02] many other companies. There's a reason [03:04] we all pick Code Rabbit. And it's not [03:05] just because it's free for open source, [03:06] which by the way, free for open source. [03:08] Figure it out now at [03:09] soyf.link/codrabbit. [03:11] Before we dig too deep into the numbers, [03:12] I think it's important to better [03:14] understand how the study worked and was [03:16] structured. It's very important to get [03:18] those things before you just go share [03:19] numbers around because I've already seen [03:21] people saying, "Look, this is proof AI [03:23] makes you 15 to 20% dumber." Not that [03:26] simple. We recruited 52 mostly junior [03:28] software engineers, each of whom had [03:29] been using Python at least once a week [03:31] for over a year. We made sure they were [03:33] at least somewhat familiar with AI [03:35] coding assistance and were unfamiliar [03:37] with Trio, the Python library on which [03:39] our tasks were based. We split the study [03:41] into three parts. A warm-up, the main [03:43] task consisting of coding two different [03:45] features using Trio, which requires [03:47] understanding concepts related to [03:48] asynchronous programming, a skill often [03:50] learned in a professional setting, and a [03:52] quiz. We told participants that the quiz [03:54] would follow the task, but encourage [03:56] them to work as quickly as possible. We [03:58] designed the coding task to mimic how [04:00] someone might learn a new tool through a [04:02] self-guided tutorial. Each participant [04:04] was given a problem description, starter [04:06] code, and a brief explanation of the [04:08] trio concepts needed to solve it. We use [04:10] an online coding platform with an AI [04:11] assistant in the sidebar which had [04:13] access to participants code and could at [04:15] any time produce the correct code if [04:17] asked. Interesting. They're using a web [04:19] interface for this. There's a lot of [04:21] layers of things that are interesting [04:23] here from the mostly junior to using a [04:26] library they were unfamiliar with to [04:28] specifically being told and encouraged [04:30] to work as quickly as possible, but also [04:32] that a quiz would occur later on. So to [04:35] see just like what the flow of this [04:36] looks like, you have two groups, [04:38] treatment and control. Control has no AI [04:40] throughout. Treatment has no AI for the [04:42] first task, which is a warm-up coding [04:43] task. They do have AI for the second one [04:46] which is the core task of creating this [04:48] project or solving these problems they [04:50] were given. Then they have posttask quiz [04:52] which they 25 minutes to do and then a [04:54] survey at the end about things they did, [04:56] what they thought etc. In our evaluation [04:58] design, we drew on research in computer [05:00] science education to identify four types [05:02] of questions commonly used to assess [05:04] mastery of coding skills. Debugging, [05:06] code reading, code writing, and general [05:08] concepts. Our assessment focused most [05:10] heavily on debugging, code reading, and [05:12] conceptual problems as we considered [05:13] these to be the most important for [05:15] providing oversight of what is [05:16] increasingly likely to be AI generated [05:18] code. And again, the results, as [05:21] mentioned before, on average, [05:22] participants in the AI group finished [05:24] slightly faster, but also perform [05:26] meaningfully worse on the test. There [05:28] are important pieces to measure here, [05:30] and this is where I'm going to start my [05:31] push back. I want to talk about the [05:33] twominut thing. When you're first [05:35] learning a new environment of any form, [05:38] it takes some amount of time to ramp up [05:40] on it to figure out how it works, what [05:42] goes where, what key bindings function. [05:44] If I have to use a computer or an [05:45] environment that isn't mine, I have to [05:47] spend the first 5 10 minutes just [05:49] adjusting. So, I have a bunch of weird [05:50] key bindings on my computer. I hate that [05:52] I do this. I try to be as stock as [05:53] possible on my machines, but there's a [05:55] few I just can't give up. I use [05:57] Carabiner heavily. It's a way to rebind [05:59] your keys on Mac, and I have a handful [06:01] that I I live on. For example, I have [06:03] caps lock bound as hyper and then other [06:05] number keys for switching between apps. [06:07] So one is helium, two as my terminal, [06:10] three is my editor or as notion. And [06:12] being able to switch between my apps [06:13] with one key press like this is very [06:15] very useful when I'm doing my day-to-day [06:16] work. I also have caps lock and U and O [06:19] as page up page down and JKL for my [06:21] D-pad. So I don't have to move my hand [06:23] all the way down to the arrow keys to [06:25] move around in my editor. So, like if I [06:27] want to just go up to, I can do it [06:28] without having to move my hand over to [06:30] the arrow keys or the trackpad. Just [06:32] little things like that that I've built [06:33] up over the years that I I can't really [06:36] use a computer without. And I've had [06:38] this problem when I was doing job [06:40] interviews, by the way, where I would go [06:42] in and they would give me a computer to [06:43] use during the interview and I could [06:45] barely use it cuz I could barely use [06:47] their keyboard cuz I just it's not mine. [06:48] I'm not used to it's not how I'm set up. [06:50] If you give all of these people a brand [06:52] new environment, at least half of the [06:54] time is going to be spent figuring out [06:56] the environment, especially in a [06:57] 30-minute window, it would take me more [06:58] than 30 minutes to figure out some weird [07:00] AI vibe coding tool in the browser. So, [07:02] if half the time is setting up, some of [07:05] the time is reading and figuring out the [07:07] code. The only piece that's left for [07:09] typing is like 5 to 10 minutes of the [07:11] actual time, obviously, the AI is not [07:13] going to be that helpful there. But, if [07:14] you're using environments you're already [07:16] familiar with, the time savings are [07:18] crazy. I have had many a time where [07:21] legitimate difficult work that would [07:23] have taken me hours if not days took an [07:25] agent 10 to 20 minutes. And I've had [07:27] times where work that would have taken [07:28] me weeks took an agent 3 to four hours [07:31] of an overnight run. Like you cannot [07:34] measure this in a study so simply when [07:37] it's in a 35minut window and half the [07:39] time is spent figuring out the tool. The [07:41] number measurement for time here is we [07:43] should just throw it out. It is not [07:45] valuable at all. Just let's be real. The [07:47] percentage gap in their understanding, [07:49] however, that is somewhat real and we [07:51] definitely want to dig more into that. [07:52] One of the pieces they do dig into here [07:54] that I think is really important to [07:55] learn from is the AI interaction modes. [07:58] This isn't like a button they hit in the [07:59] settings. This is how they actually used [08:01] the AI. We were particularly interested [08:03] in understanding how participants went [08:04] about completing the tasks that we [08:06] designed. In our qualitative analysis, [08:08] we manually annotated screen recordings [08:09] to identify how much time participants [08:11] spent composing queries, what types of [08:13] questions they asked, the types of [08:15] errors they made, and how much time they [08:16] spent actively coding. One surprising [08:18] result was how much time participants [08:20] spent interacting with the AI assistant. [08:22] Several took up to 11 minutes, which is [08:24] 30% of the total time allotted, [08:25] composing up to 15 queries. This helped [08:28] explain why on average participants [08:29] using AI finished faster, although the [08:31] productivity improvement was not [08:32] statistically significant. This sent [08:34] should not even be in here. The fact [08:35] that they're trying to measure the [08:36] productivity win in a 30-minute task [08:39] means the people who built this study [08:40] and threw it at a bunch of juniors [08:42] fundamentally do not understand how we [08:43] operate with these tools. And I want to [08:45] really emphasize this point from my [08:47] understanding of the study that I have [08:49] read separately. My understanding is [08:51] that this is interesting in the limited [08:53] scope of how students and really really [08:56] really early developers when first [08:59] introduced to an AI tool similar to like [09:01] the first time they open their IDE. How [09:03] does using it with AI compare to using [09:04] it without in like the limited one to [09:07] two hour starting window? I do not like [09:09] the fact that people are trying to [09:11] project the study beyond that, [09:12] especially on the speed side. I've [09:14] already seen people say, "Look, there's [09:16] proof that AI doesn't actually speed you [09:18] up as a dev." Recent studies are showing [09:19] a pretty consistent 30% improvement in [09:22] code output year-over-year for [09:24] developers who are using AI in real [09:26] companies. So, no, this is [ __ ] [09:29] There is parts here that we should dig [09:30] into though, so I'm not trying to [09:31] discount the whole thing. I'm just [09:32] trying to limit the scope of how much we [09:34] use this to prescribe what's going on. [09:37] They do call us out a little bit. We [09:38] expect AI would be more likely to [09:40] significantly increase productivity when [09:42] used on repetitive or familiar tasks. It [09:44] goes further. It improves a lot more too [09:46] when you're also familiar with the AI [09:48] tools you're using in the development [09:50] environment that you like. Once you get [09:52] yourself in the harness in the tool set [09:54] that you feel good about and get good at [09:56] it, you start shipping at speeds that [09:58] don't make sense. I wish I had my other [10:01] laptop plugged in right now so I could [10:02] show you guys. I was last night working [10:03] on four projects at the same time with [10:05] codecs. It's insane. I'm building like a [10:08] new app every day right now to just test [10:10] out ideas. Obviously, I'm not shipping [10:11] all of these things, but I'm putting [10:13] time in to experiment with different [10:14] ideas to see what is and isn't possible. [10:16] And I've also seen some of the best [10:17] engineers I know, people building crazy [10:19] libraries, having AI agents go out they [10:23] have for a feature and being not just [10:25] surprised that it could do it, but by [10:27] decisions it made that informed them on [10:28] how they would go write it by hand [10:30] later. There's a story that's going to [10:31] come out soon about this from a really, [10:33] really talented full stack front-end [10:35] framework person that y'all already know [10:37] and love. A lot of the AI haters know [10:39] and love, and I cannot wait for him to [10:40] share the story because it is crazy. [10:42] Back to the study, though. [10:43] Unsurprisingly, participants in the no [10:45] AI group encountered more errors. These [10:47] included errors in syntax and in trio [10:49] concepts, the latter of which mapped [10:51] directly to topics tested on the [10:52] evaluation. Reminder, this is a Python [10:56] test. These people are making syntax [10:59] errors in Python. I think I've said all [11:01] I have to about the quality of the [11:02] people being tested. Our hypothesis is [11:04] that the participants who encountered [11:06] more trio errors, namely the control [11:07] group, likely improved their debugging [11:09] skills through resolving these errors [11:11] independently. Sure. Just to emphasize [11:14] the like weirdness of how people were [11:16] using these tools in the study and the [11:18] just to be frank the junioress of the [11:21] people who were in this study. The [11:23] researchers noted that half of the AI [11:25] using cohort spent most of their time [11:26] literally retyping the AI generated code [11:29] into their solution instead of copy [11:31] pasting or manual coding. They wrote [11:33] their code from scratch with light AI [11:34] guidance. If you ignore the people who [11:36] spent most of their time retyping, the [11:38] AI users were 25% faster. I confess this [11:41] baffles me. What kind of person manually [11:42] retypes AI generated code? Did they not [11:45] know how to copy paste? Unlikely since [11:47] the study was mostly composed of [11:48] professional or hobby developers. It [11:50] certainly didn't help them on the quiz [11:51] score. The retypers got the same low [11:53] score as the pure copy pasters. In any [11:55] case, if you know how to copy paste or [11:57] use an AI agent, I wouldn't use this [11:58] paper as evidence that AI will not be [12:00] able to speed you up. Thank you, Sean, [12:01] for the observations here. Great [12:03] article. If you want to read other [12:04] opinions on this, it'll be linked in the [12:05] description. But back to my thoughts, [12:07] cuz I have so much to say. They try to [12:10] break down the different patterns in [12:11] which people interact with the AI here [12:12] to give you an idea of what the results [12:15] were from how they used it. The patterns [12:17] that result in the lowest scores were [12:18] ones where the users the test subjects [12:20] would rely heavily on the AI either [12:23] through code generation or debugging. [12:24] The average quiz score in that group was [12:26] less than a 40%. They showed less [12:28] independent thinking and more cognitive [12:29] offloading. We further separated that [12:31] into a few different groups. AI [12:33] delegation which was participants in the [12:35] group who relied on AI to write the code [12:37] and complete the task. They completed [12:38] the task the fastest and encountered few [12:39] or no errors in the process. Progressive [12:41] AI reliance participants in the group [12:43] started by asking one or two questions [12:45] but eventually delegated all codew [12:47] writing to the AI assistant. They scored [12:49] poorly on the quiz largely due to not [12:50] mastering the concepts in the second [12:52] task. And then the iterative AI [12:53] debugging group. Participants in this [12:55] group relied on AI to debug or verify [12:57] their code. They asked more questions [12:58] but relied on the assistant to solve the [13:00] problem rather than clarify their own [13:02] understanding. They scored poorly as a [13:03] result and were also slower at [13:05] completing the two tasks. High-coring [13:07] interaction patterns. We considered [13:09] high-coring quiz patterns to be [13:11] behaviors where the average quiz score [13:12] was 65% or higher. Participants in these [13:14] clusters used AI both for codegen and [13:17] conceptual queries. Generation then [13:19] comprehension. Participants in this [13:20] group first generated code and then [13:22] manually copied or pasted the code into [13:24] their work. After their code was [13:26] generated, they asked the AI assistant [13:27] follow-up questions to improve their [13:29] understanding. These participants were [13:30] not particularly fast when using AI, but [13:33] they showed a higher level of [13:34] understanding on the quiz. [13:35] Interestingly, this approach looked [13:37] nearly the same as that of the AI [13:38] delegation group, except they used the [13:40] AI to check their own understanding. One [13:42] other thing of note here, n equals 2. [13:45] The other three ones before, n equals 4. [13:49] Two people did this in the study. This [13:51] is not some crazy comprehensive group of [13:53] people testing all sorts of theories. [13:55] This is like 50 people. And out of the [13:58] 50 people, 25 got to use AI and two of [14:01] them happen to use it in this way. Two [14:03] out of 25 is bad. That's less than 10%. [14:05] But do you know why so few people use it [14:07] to clarify their understanding? Remember [14:09] earlier we encourage them to work as [14:11] quickly as possible. How much effort do [14:13] you expect the AI developers who are [14:15] still insecure about their own [14:17] abilities, who are still learning how [14:18] code works when told to write code fast, [14:22] do you actually honestly expect them to [14:24] use those same tools to better their [14:25] understanding? And to be clear, I'm not [14:27] sitting here pretending that every [14:29] developer uses AI to teach themselves [14:31] things, but I like to think it'd be a [14:33] lot higher than 2% if you told them to [14:34] be quizzed after. Like, what the [ __ ] [14:37] It's two people. It's not a real useful [14:39] thing. I like they're being so [14:40] transparent about the numbers here, but [14:41] that's two people. You can't drive home [14:44] any real learnings from this. This is [14:46] not a statistically significant study. [14:48] Hey, Claude, can you contact some people [14:49] and get a paper together? Yeah, actually [14:53] though again there are interesting [14:54] things to talk about here, but like we [14:57] need to be realistic. There were also [14:58] the hybrid code explanation people. [15:01] There were three of them. Participants [15:02] in this group composed hybrid queries [15:04] which they asked for code generation [15:05] along with explanations of the generated [15:07] code. Reading and understanding the [15:09] explanations they asked for took more [15:10] time but helped in their comprehension. [15:12] And then there was the conceptual [15:14] inquiry. Participants in this group only [15:16] asked conceptual questions and relied on [15:18] their improved understanding to complete [15:20] the task. Although this group [15:21] encountered many errors, they also [15:23] independently resolved them. On average, [15:25] this mode was the fastest among [15:27] highscoring patterns and the second [15:28] fastest overall after AI delegation. You [15:31] get the idea. They did their best to [15:33] break these people into groups, but [15:35] there just weren't enough people to get [15:37] meaningful insights. When they created a [15:39] group out of behavior of two people, it [15:41] says all that you need to. A follow-up [15:43] study I would love to see for something [15:45] like this is, okay, there's a lot of [15:47] things I would want to see. I would want [15:49] more experienced developers. I would [15:50] want a wider variety of experienced [15:52] developers. I'd want a much larger [15:53] number of developers. And I would want [15:54] much harder tasks, ideally without the [15:57] same time cap put on them. It would be [15:59] so cool to see an idea for a general [16:02] project. Like I'm working on a Frame.io [16:04] alternative because I'm tired of dealing [16:05] with it for managing my team and I want [16:07] to fully exit Adobe. It's the only piece [16:08] of Adobe software we're still using. If [16:10] you're not familiar, Frame is like code [16:11] review plus Dropbox for video and it's [16:14] really cool. It also sucks to use more [16:16] and more every day because the site's [16:17] just slow as [ __ ] and like every time I [16:20] go to it, it signs me out. I sign back [16:21] in. I have to wait through three loading [16:23] screens. I have to go through a bunch of [16:24] really slow, miserable [ __ ] It's not [16:26] great. I built a fully functioning [16:27] alternative in under an hour. And I [16:29] spent half of that time evaluating [16:31] different storage providers to see which [16:33] ones would provide the best speeds on [16:35] good internet connections because I have [16:37] gigabit, my editor is 5 gig. I want to [16:39] make sure we can saturate our [16:40] connections with this. It'd be really [16:42] cool to see a study that takes a project [16:43] like that, has a group with no AI and a [16:46] group with AI, and see how long it takes [16:48] both to complete it, see how good the [16:51] products they complete function, how [16:52] solid the tech decisions they make are, [16:54] and how good their understanding of the [16:56] code is. I would bet for something like [16:57] that, the results would vary wildly. I [17:00] am promising you as somebody who is [17:02] known specifically for building [17:03] applications fast and building a whole [17:05] tech stack around making application [17:07] development safer and faster. That's why [17:09] I built T3 stack. The goal was to make [17:11] it easier to ship fast without running [17:13] into errors in your final outputs. I [17:16] would literally guarantee that at most [17:18] skill levels, the AI generated versions [17:20] would not just come out faster, but [17:22] would come out more stable and with less [17:24] bugs and probably be better experiences [17:25] overall. And for the junior devs, they'd [17:27] have way better tech decisions, too. The [17:29] best devs I know are telling me about [17:31] how the AI is making better decisions [17:32] than them. We are past that point. And [17:35] this notice nothing in here, not a [17:37] single thing in here has mentioned the [17:39] quality of the code that was output. The [17:41] word quality doesn't even come up. They [17:43] also had one framing here that I have [17:45] feelings on. The agent could at any time [17:48] produce the correct code if asked. The [17:52] correct code. It is possible the person [17:54] who wrote this does not code much if at [17:57] all and doesn't understand how we work [17:59] as developers. If you've written code [18:01] for any amount of time, you know that [18:02] there is no such thing as the correct [18:04] code. There are very few problems in [18:06] software that can't be solved dozens if [18:07] not hundreds of different ways. If there [18:09] is one correct code answer to the [18:12] problem they're asking, then we should [18:14] honestly throw this entire [ __ ] study [18:16] out cuz it's useless. Happy you guys get [18:18] this chat. There can be multiple correct [18:19] codes. That's interesting. Correct code. [18:21] There are hundreds of ways to solve most [18:23] problems. Yeah, I have feelings on this. [18:25] I don't think this study was done in a [18:28] way that maps well enough to real world [18:31] use for me to care about it beyond the [18:34] potential in slight differences in use [18:36] characteristics resulting in different [18:38] behaviors and different learnings as a [18:39] result. I do want to be careful when I'm [18:41] talking about things that affect like [18:42] the junior developers and people who are [18:44] learning how to code right now because [18:45] it's very different from how I use AI [18:47] and how I talk about it. I have been [18:49] writing code for a very long time. I [18:51] know a lot of things that can go right [18:52] and wrong. And I've had even in [18:54] languages and ecosystems I don't [18:55] understand. I'm able to get a rough idea [18:57] of where things can go wrong. I'm deep [18:59] in building multiple native apps with [19:01] AppKit and Swift, which is a stack I've [19:03] never used before. And I still have a [19:05] good enough intuition of where a thing [19:06] might be going wrong to steer the agent [19:08] the right way, even without reading the [19:09] code. That is not what you should be [19:11] doing if you're still learning how to [19:12] code. That is certainly not what you [19:13] should be doing if you've never been [19:14] paid to code yet. You still have to [19:16] build up the skills. And to be clear, [19:18] this does not mean you shouldn't use AI. [19:19] You shouldn't use it to complete your [19:20] homework for you, but you should use it [19:22] to better your own understanding of [19:24] things and also at times reach beyond [19:26] your own capabilities to experiment with [19:27] ideas. I have a whole video about junior [19:29] dev specifically that didn't perform too [19:31] great. So, I will not be talking about [19:33] y'all anymore because you don't actually [19:34] care to get better. You just complain [19:36] when I don't give you enough credit. Not [19:37] every junior, just the ones who are in [19:38] my comment section. There's a lot of [19:40] y'all in the comments section. You're [19:42] certainly writing a lot more comments [19:43] and probably not writing that much code. [19:44] But, I do want to touch on one point [19:46] from this video because I think it's [19:47] really important. The htmx.org org [19:49] account might seem like a weird thing to [19:51] site here, but the HTMXorg creator is a [19:54] professor who teaches computer science [19:56] in a real college. He's one of the best [19:58] computer science professors I've ever [20:00] met, talked to, or seen the work of. [20:02] He's genuinely great. I love Carson. One [20:05] of the cool things he did is give his [20:06] students a custom agent MD file for [20:08] steering their agents in a way that will [20:10] help them learn instead of cheat. AI [20:12] agent guidelines. This file provides [20:14] instructions for AI coding assistants [20:15] working with students in this course. [20:17] Primary role teaching assistant not code [20:20] generator. AI agents should function as [20:22] teaching aids that help students learn [20:24] through explanation, guidance, and [20:25] feedback, not by solving problems for [20:27] them. What agents should do, explain [20:29] concepts, point students to relevant [20:31] lecture materials, review code that [20:32] students have written. You get the idea. [20:34] Agents shouldn't write entire functions [20:36] or complete implementations, generate [20:37] full solutions for assignments, complete [20:39] to-do sections in assignment code, [20:40] refactor large portions of student code, [20:42] provide solutions to quiz or exam [20:44] questions, etc. very much steering the [20:47] models away from cheating and helping [20:49] more towards teaching and learning. This [20:51] is great. I want to see more of this [20:54] because the student case, the junior [20:56] case, and the experienced developer [20:58] case/team/enterprise [21:00] case are so far separated that the [21:03] attempts to talk about them like they're [21:04] the same are almost insulting. Anthropic [21:06] even calls this out in their conclusion [21:08] section here. Prior studies have found [21:10] mixed results on whether AI helps or [21:12] hinders code productivity. Our own [21:14] research found that AI can reduce the [21:15] time it takes to complete some work [21:17] tasks by 80%. A result that may seem [21:20] intention with the findings presented [21:21] here. Yeah, no [ __ ] But the two studies [21:24] ask different questions and use [21:25] different methods. Our earlier [21:26] observational work measured productivity [21:28] on tasks where participants already had [21:30] the relevant skills also were using the [21:32] tools that they are familiar with. While [21:34] this study examined what happened when [21:36] people are learning something new, it's [21:38] possible that AI both accelerates [21:40] productivity on well-developed skills [21:41] and hinders the acquisition of new ones, [21:44] though more research is needed to [21:45] understand this relationship. There's [21:47] also one more key piece that this study [21:49] doesn't touch on at all that I think is [21:51] really important is motivation. The [21:54] harsh reality of building things and [21:56] doing hard stuff is that it's hard to [21:59] stay through. I'm going to do my [22:00] favorite analogy that you guys who have [22:02] been here for a while are very familiar [22:04] with my skateboard. A lot of people have [22:07] tried to skate. It's fun. It's cool. [22:09] It's rad. Skateboarding is also really, [22:12] really hard. Skateboarding is why I had [22:14] to have my whole left hand [22:15] reconstructed. And to this day, I have [22:17] this brutal scar for my entire left [22:20] thumb having to be reattached to my hand [22:22] just a few months ago. Yet, I still do [22:24] it. I love my skateboard. It is a thing [22:28] that is very important to me. It is one [22:29] of the coolest, best, most rewarding [22:31] skills I have taken the time to teach [22:33] myself. But there's a thing that sucks [22:35] about skateboarding. It hurts. [22:38] Literally, every time you try a trick, [22:41] there's a risk. It physically hurts you. [22:43] At the very least, it will physically [22:45] exhaust you some amount. The amount of [22:46] energy a skateboard takes from you is [22:48] unbelievable. It also hurts you mentally [22:51] a bit, too, because every time you fail, [22:53] you feel the failure. You see people do [22:56] the tricks so elegantly, so easily, and [22:58] then you go try it yourself and you can [22:59] barely get the board off the ground. 95% [23:02] of skaters will never even ollie. [23:03] They'll never get the board off the [23:04] ground the first time because even that [23:07] can take a year to learn and more than [23:09] that to learn. Well, even just jumping [23:11] on a skateboard is so difficult. But [23:14] when you can get over that, you'll do [23:16] incredible things. Skateboarding has a [23:18] very strange problem where the interest [23:21] in skateboarding can grow, but [23:24] skateboarding itself does not. Over co a [23:27] lot of people were looking for new [23:28] hobbies and skateboard sales went up in [23:31] a way we've never seen before. It was [23:33] literally five times more sales than [23:35] we've ever seen in the history of [23:36] skateboarding. It was crazy. The issue [23:38] was that all the people who bought a [23:39] skateboard weren't motivated to learn [23:41] how to skate. They were motivated to get [23:43] something to do during co skateboard [23:45] sales rely a lot on buying new boards. [23:48] The wooden part, the deck on a [23:49] skateboard can be broken relatively [23:51] easily when you start skating harder. I [23:53] can break one in a few hours after [23:54] setting it up. Usually it takes me about [23:56] a week or two on certain brands from [23:58] certain things they do like carbon fiber [23:59] and [ __ ] inside. They can last a couple [24:01] months, but I at my peak probably went [24:04] through a hundred decks in a year. That [24:06] is a big part of how skateboarding gets [24:08] funded is people like me having to buy [24:10] replacements for the parts that we [24:11] break. And then that money is used to [24:14] continue to pay skateboarders to do [24:16] harder things to encourage more people [24:17] to skate and get the flywheel. It was [24:19] assumed when this huge surge in [24:21] skateboard sales happened that there [24:23] would be some continued interest. But [24:25] 99% of the people plus who picked up [24:28] skateboarding during the COVID break [24:30] never got good enough to break a board. [24:33] That huge surge in board sales resulted [24:35] in no long-term sales. And a ton of [24:38] skateboard companies that I've loved my [24:40] whole life went out of business. I've [24:42] rode the exact same board for 15 years. [24:44] And you can no longer buy them because [24:46] the company foreclosed because they [24:48] produce too much stock thinking that the [24:50] sales would continue and then they [24:51] didn't and now they're dead. So why the [24:52] [ __ ] am I so deep in talking about [24:54] skateboards? It's why I do hard things. [24:57] the fact that when I was young, I [24:58] learned that to do really cool, [25:00] rewarding [ __ ] that makes you feel this [25:02] special way, you had to be willing to [25:04] get hurt. You had to be willing to feel [25:06] stupid. And my skateboard taught me that [25:08] feeling dumb is actually good because it [25:11] means you'll feel smart soon. And if you [25:13] don't feel dumb, you're not pushing [25:14] yourself. But there are limits to this. [25:17] If it takes you a month to learn how to [25:19] oie, you're much more likely to continue [25:21] skating than somebody who it takes 3 [25:23] months for. because the person who it [25:24] takes three months for is three times [25:26] more likely to give up before they get [25:27] there. This is a thing I think about a [25:29] lot. On one hand, I love things that [25:32] have these hard filters where it's hard [25:34] enough to get into that everyone doing [25:36] it is really good. I love the fact that [25:38] this existed for so long in programming. [25:40] Anybody with real open source work got [25:42] there because they cared too much and [25:45] got over the moments where they felt [25:46] dumb to get there. Anyone who can do a [25:49] trick down a stair set or a handrail got [25:51] there because they pushed through the [25:52] feeling of this is too hard. I'm going [25:54] to get hurt. This isn't worth it. And [25:55] they pushed through it until they could [25:57] do that thing. On one hand, this means [25:59] the people who make it through are [26:00] exceptional. On the other hand, there [26:02] are lots of people who could be [26:04] exceptional that give up because they [26:06] don't have a good feeling early enough. [26:08] And I do think for a lot of things, [26:10] especially places where we don't have [26:11] enough people doing the thing, making it [26:13] easier to have that first, oh [ __ ] I [26:16] might actually be able to do this moment [26:18] is valuable. If it was easier to land [26:21] your first oie and harder to land your [26:23] first kick flip, there would be way more [26:25] skaters. Because if we have the timeline [26:27] of how long it takes to feel good about [26:29] skateboards, it's probably going to be [26:31] the first oie. And if we say the first [26:33] oie happens in 3 months and we say the [26:36] first kick flip happens in I don't know [26:38] 6 months. Think about all of the people [26:41] who fail before they get here. By the [26:43] time you get to the first oie, you've [26:45] already lost 70% of the people, but the [26:47] first kick flip, a lot more survive. I [26:50] personally don't know any person who can [26:52] oie but can't kick flip. I know a few [26:55] that are like they've done a few ollies [26:56] that are sketchy but work and they got [26:58] close to a few kick flips, but I don't [27:00] know a whole lot of people that made it [27:01] to this point that didn't make it to [27:04] this point. So imagine this hypothetical [27:06] world where getting the board off the [27:07] ground was easier than it actually is. [27:09] Let's say the first oi didn't take 3 [27:11] months. Let's say it took 2 weeks. [27:14] Suddenly this gets moved way way way way [27:17] further to the front. And if that drop [27:20] off rate was consistent, I would suspect [27:22] the result there is that you get a much [27:25] higher percentage making it all the way [27:27] through. And even if you move the [27:28] kickflip way further along, let's say [27:30] you move this to 9 months, the fact that [27:32] you had that first, oh [ __ ] maybe I can [27:34] do it moment so early means that that [27:37] huge drop off curve gets smoothed out [27:40] and a lot more people make it to the [27:41] end. Now, let's replace oie and kickflip [27:44] with hello world and usable app. Do you [27:47] understand? Now, if AI tools allow a [27:50] developer to not feel stupid for as long [27:53] initially, the likelihood that they make [27:56] it through goes up exponentially. I can [27:59] tell you from experience, knowing a lot [28:01] of people who were really into tech, [28:02] that like built their own computers that [28:03] loved that stuff, that tried like [28:05] installing Linux on the computer or like [28:07] building a small app for a game they [28:09] played that gave up because the initial [28:12] learnings was just too much and they [28:14] felt too dumb for too long. If you can [28:16] give them a taste of the reward earlier, [28:18] they will survive longer. Trigger [28:20] warning for those who are sensitive to [28:22] animals being hurt. We're about to talk [28:24] about a rat study that involves rats [28:26] dying, but I think it's actually really [28:27] useful for showing biologically that we [28:30] are built to use hope. The experiment [28:34] was done in the 1950s where domesticated [28:36] rats were placed in a bucket of water to [28:38] see how long they would swim before they [28:39] drowned. On average, they would give up [28:41] and drown within 15 minutes. But they [28:43] did a separate trial where just before [28:45] the rats were about to drown, the [28:46] researchers rescued them, dried them [28:48] off, let them rest for a while, and then [28:49] placed them back in the water. When [28:51] those rats were pulled out once and put [28:53] back in, they would swim for more than [28:55] 60 hours instead of the 15 minutes [28:58] before they gave up before. Once the [28:59] rats were saved, they developed a sense [29:01] of hope. And I want to make sure this is [29:04] as grim as it is that this is the point [29:05] that I emphasize here. If you get a [29:07] little bit of hope early on where you [29:09] were about to give up, the likelihood [29:11] that you keep going is exponentially [29:14] higher. It just is. That's a fact. And [29:17] if AI tools are the equivalent of being [29:19] pulled out of the water before you drown [29:21] the first time, that is dark. But that [29:23] is the world. The world's a dark place, [29:25] and you need hope and motivation to push [29:28] through it. If the AI keeps you from [29:29] developing motivation, then it's not [29:31] useful. But if the AI keeps you from [29:33] losing motivation, it is incredible. And [29:35] quick shout out to the lab mice that [29:37] have made the world as much better as it [29:38] is. As we were saying, if you compare [29:41] two people who built an app, one who [29:43] vibe coded it and one who wrote it by [29:45] hand, obviously the one who didn't vibe [29:46] code is going to be more likely to [29:48] succeed in the field. But if you take [29:49] two people who are starting programming, [29:51] one used AI to get something useful [29:54] early and the other didn't, the one who [29:57] didn't is more likely to give up. And I [29:58] want to make sure we find a balance when [30:00] we talk about these things. Like on one [30:02] hand, you shouldn't use AI tools to [30:03] avoid learning, but on the other, you [30:05] shouldn't spend so much time in that [30:06] space where you feel dumb that you burn [30:08] out and quit. If AI can unblock you, if [30:10] AI can accelerate you a bit, if AI can [30:12] get you excited about code again, it's [30:15] probably worth doing. That doesn't mean [30:16] you should replace everything with it. [30:18] It doesn't mean you should outsource [30:18] your brain. It means you should reach a [30:20] little further than you might be capable [30:22] of so that you can see things that you [30:24] might not have been able to and feel [30:26] things that you might have given up [30:27] before you could. Everyone needs to feel [30:29] good about the work they're doing [30:30] sometimes. And if AI helps you feel [30:32] better faster, use it in moderation. [30:34] Thank you for once again letting me rant [30:36] about skateboards a bit. It's my [30:37] favorite thing. I'm always happy when [30:38] you do it. Hope this was helpful for [30:40] somebody. And until next time, peace [30:42] nerds.  [[3. Resources/Readwise/Full Document Contents/Articles/Anthropic Study Shows AI Makes Devs Dumb|Full Document Contents]] ## Document Notes --- tags: #ai-coding #developer-skills #anthropic-study author: Theo - t3․gg source: YouTube reference: [https://www.youtube.com/watch?v=ZINQTR6H5dI] date created: 2024-12-19 10:30 --- # 🏷️ Title: Anthropic study shows AI makes devs dumb ## 🧰 Action Inbox - [H1] Review Anthropic's full paper on AI impact on skill formation — study methodology #research - [H2] Implement Code Rabbit AI code review in development workflow — productivity tool #devtools - [H3] Create AI guidance document for junior developers similar to HTMX approach — teaching aid #education ## 🧭 Table of Contents - [[#📝 Summary]] - [[#📖 Main Content]] - [[#🗒️ Notes]] - [[#📚 References]] - [[#🗣️ Paraphrase]] - [[#📰 Newsletter]] - [[#✅ Action Items]] - [[#🧪 Hypotheses]] - [[#📊 Methodology]] - [[#❓ Questions]] - [[#🔮 Future Research]] - [[#💡 Implications]] - [[#🧠 Learning Styles Assessment]] - [[#📊 Accessibility Assessment]] - [[#🌍 DEIB Assessment]] - [[#✅ Logical Validity]] - [[#🛡️ DARVO Analysis]] - [[#🎯 Conclusion]] ### 🔗 References - [Anthropic Paper on AI and Skill Formation](URL) - [Code Rabbit AI Code Review](https://coderabbit.ai) ## 📝 Summary - Anthropic published research showing AI-assisted developers completed tasks 2 minutes faster but scored 17 percentage points lower on comprehension tests (50% vs 67%) - Study involved 52 mostly junior developers using Python's Trio library in unfamiliar web coding environment - Results suggest AI may accelerate productivity while potentially hindering skill acquisition, particularly debugging abilities - Key concerns include cognitive offloading and reduced independent problem-solving development - Time savings measurement questioned due to 30-minute constraint and unfamiliar tooling environment ## 📖 Main Content Anthropic conducted a controlled study examining how AI coding assistance affects developer skill formation. The research divided 52 junior developers into control and treatment groups, with the treatment group using AI assistance for coding tasks involving Python's Trio asynchronous programming library. The study found that while AI-assisted developers finished slightly faster (statistically insignificant 2-minute improvement), they demonstrated significantly worse understanding on post-task assessments. The AI group averaged 50% on comprehension quizzes compared to 67% for developers who coded without assistance - equivalent to nearly two letter grades difference. The largest performance gaps appeared in debugging questions, suggesting AI assistance may particularly impact developers' ability to understand code failures and error resolution. Researchers identified different AI interaction patterns, from complete delegation to conceptual inquiry, with varying impacts on learning outcomes. However, the study's methodology faces criticism for its limited scope: 30-minute tasks in unfamiliar environments with junior developers. Critics argue this doesn't reflect real-world AI usage patterns where experienced developers use familiar tools over extended periods on complex projects. ## 🗒️ Notes #### ⏱️ [00:00](https://youtu.be/ZINQTR6H5dI?t=0) — [H1] Study overview and speed results - Key idea: AI makes developers faster but questions remain about skill impact - Evidence: "AI group finished about 2 minutes faster, although the difference was not statistically significant" #### ⏱️ [01:15](https://youtu.be/ZINQTR6H5dI?t=75) — [H2] Comprehension score differences - Key idea: Significant gaps in understanding between AI and non-AI groups - Evidence: "AI group averaged 50% on the quiz compared to 67% in the handcoded group" #### ⏱️ [02:45](https://youtu.be/ZINQTR6H5dI?t=165) — [H1] Code Rabbit sponsor segment - Key idea: AI code review tools can augment rather than replace human review - Evidence: "Quality of the reviews is great... better than humans at code review because it'll do much more meticulous stuff" #### ⏱️ [04:30](https://youtu.be/ZINQTR6H5dI?t=270) — [H2] Study methodology breakdown - Key idea: 52 junior developers, unfamiliar with Trio library, 30-minute constraint - Evidence: "Recruited 52 mostly junior software engineers... encouraged to work as quickly as possible" #### ⏱️ [06:15](https://youtu.be/ZINQTR6H5dI?t=375) — [H3] Environment setup time critique - Key idea: New environment learning curve invalidates speed measurements - Evidence: "Half the time is going to be spent figuring out the environment... would take me more than 30 minutes" #### ⏱️ [08:45](https://youtu.be/ZINQTR6H5dI?t=525) — [H2] AI interaction time analysis - Key idea: Participants spent up to 30% of time composing AI queries - Evidence: "Several took up to 11 minutes... composing up to 15 queries" #### ⏱️ [11:20](https://youtu.be/ZINQTR6H5dI?t=680) — [H1] Real-world productivity gains - Key idea: Industry studies show consistent 30% improvement year-over-year - Evidence: "Pretty consistent 30% improvement in code output year-over-year for developers who are using AI" #### ⏱️ [13:50](https://youtu.be/ZINQTR6H5dI?t=830) — [H3] Manual retyping behavior - Key idea: Half of AI users manually retyped generated code instead of copy-pasting - Evidence: "Half of the AI using cohort spent most of their time literally retyping the AI generated code" #### ⏱️ [15:30](https://youtu.be/ZINQTR6H5dI?t=930) — [H2] Interaction pattern analysis - Key idea: Different AI usage patterns correlated with different learning outcomes - Evidence: "AI delegation... completed the task the fastest... Progressive AI reliance... scored poorly on the quiz" #### ⏱️ [17:45](https://youtu.be/ZINQTR6H5dI?t=1065) — [H3] High-scoring patterns with small sample sizes - Key idea: Beneficial AI usage patterns identified but with very small groups (n=2-3) - Evidence: "Generation then comprehension... n equals 2... Two people did this in the study" #### ⏱️ [20:15](https://youtu.be/ZINQTR6H5dI?t=1215) — [H1] "Correct code" critique - Key idea: Study assumes single correct solution, which doesn't reflect real development - Evidence: "There is no such thing as the correct code... hundreds of ways to solve most problems" #### ⏱️ [22:30](https://youtu.be/ZINQTR6H5dI?t=1350) — [H2] HTMX professor's AI guidelines - Key idea: Educational approach to AI that promotes learning over cheating - Evidence: "AI agents should function as teaching aids... not by solving problems for them" #### ⏱️ [25:45](https://youtu.be/ZINQTR6H5dI?t=1545) — [H1] Anthropic's acknowledgment of limitations - Key idea: Study authors recognize different contexts for AI usage - Evidence: "AI both accelerates productivity on well-developed skills and hinders the acquisition of new ones" #### ⏱️ [26:30](https://youtu.be/ZINQTR6H5dI?t=1590) — [H1] Skateboarding motivation analogy - Key idea: Early success experiences crucial for long-term engagement - Evidence: "If you can give them a taste of the reward earlier, they will survive longer" #### ⏱️ [30:15](https://youtu.be/ZINQTR6H5dI?t=1815) — [H2] Hope and persistence research - Key idea: Early positive experiences exponentially increase persistence - Evidence: "Rats would swim for more than 60 hours instead of 15 minutes... once they developed a sense of hope" ## 📚 References - [Anthropic Paper on AI and Skill Formation](URL) - [Code Rabbit AI Code Review](https://coderabbit.ai) - [HTMX AI Guidelines](URL) - [Sean's Article Analysis](URL) ## 🗣️ Paraphrase - Anthropic's study found junior developers using AI tools completed coding tasks marginally faster but demonstrated significantly worse comprehension of the code they created - The research methodology has limitations including short time constraints, unfamiliar environments, and small sample sizes that may not reflect real-world AI usage patterns - Different approaches to using AI showed varying impacts on learning, with delegation patterns showing poor comprehension while conceptual inquiry approaches maintained better understanding ## 📰 Newsletter **AI Coding Study Sparks Developer Skill Debate** New research from Anthropic reveals a concerning trade-off: while AI tools help developers work faster, they may be hurting our understanding of the code we create. Junior developers using AI scored 17 percentage points lower on comprehension tests, particularly struggling with debugging skills. The study has limitations - short timeframes, unfamiliar tools, small sample sizes - that may not reflect how experienced developers actually use AI in practice. Industry data shows consistent 30% productivity gains for AI-using developers over time. The key insight? Context matters enormously. AI that helps beginners get early wins and stay motivated could be valuable, while AI that replaces learning entirely is counterproductive. The goal should be augmentation, not replacement. **Practical Takeaway**: Consider implementing AI code review tools like Code Rabbit to catch issues before human review, and create learning-focused AI guidelines for junior team members. Resources: [Learn WordPress](https://learn.wordpress.org/), [Developer Docs](https://developer.wordpress.org/), [WP-CLI](https://wp-cli.org/) ## ✅ Action Items #### DevRel - Developer education: Create AI usage guidelines that promote learning over delegation - Developer experience: Review how AI tools impact onboarding and skill development - Community management: Facilitate discussions on balanced AI adoption practices #### OSPO - Developer Support: Provide training on effective AI tool usage for different skill levels - Strategic Alignment: Monitor how AI adoption affects codebase quality and maintainability #### Community - Contributors: Develop onboarding programs that balance AI assistance with skill building - Maintainers: Consider AI review tools for code quality while preserving learning opportunities #### WordPress - Core: Evaluate AI tools for code review and quality assurance processes - Training: Develop lesson plans addressing AI tool usage in development workflows - Community: Create guidelines for AI usage in educational contexts and skill development - Documentation: Update developer resources with AI tool recommendations and best practices ## 🧪 Hypotheses - AI tools provide greater productivity benefits for experienced developers than beginners due to existing mental models and debugging skills - Short-term comprehension gaps from AI usage may resolve as developers gain more experience with both AI tools and programming concepts - Different AI interaction patterns (delegation vs. inquiry) create different learning outcomes that persist over time - Early positive experiences with AI-assisted development increase long-term persistence in programming careers ### 📊 Methodology - Longitudinal study comparing skill development over 6-12 months between AI-assisted and traditional learning groups - Analysis of code quality metrics and bug rates for AI vs. non-AI generated code across experience levels - Controlled experiments measuring debugging skill retention after different types of AI assistance ### ❓ Questions - How do comprehension gaps from AI usage evolve as developers gain more experience with both programming and AI tools? - What specific AI interaction patterns maximize learning while maintaining productivity benefits? - Do early positive experiences with AI tools increase or decrease long-term programming skill development? ### 🔮 Future Research - Long-term longitudinal studies tracking skill development over multiple years with different AI usage patterns - Analysis of AI impact on different types of coding skills (debugging, architecture, testing, etc.) - Investigation of optimal AI usage guidelines for different experience levels and learning contexts ### 💡 Implications - Practice: Need for balanced AI usage that maintains skill development while leveraging productivity benefits - Policy: Educational institutions should develop guidelines for AI tool usage in programming curricula - Education: Training programs must address both AI tool proficiency and fundamental programming skills - Funding: Investment needed in research on optimal AI-human collaboration patterns for skill development ### 🧠 Learning Styles Assessment Reading level estimate: Grade 10-12 #### 👀 Visual learners - Present: Code examples, performance comparisons, interaction pattern diagrams - Improve: Flowcharts showing different AI usage approaches, visual timelines of skill development #### 🔊 Aural learners - Present: Video format with clear explanations and examples - Improve: Podcast version focusing on key insights and implications #### 🗣️ Verbal learners - Present: Clear analogies (skateboarding), defined terminology, contextual explanations - Improve: Glossary of technical terms, more real-world examples #### 🧩 Kinesthetic learners - Present: Practical recommendations, actionable takeaways - Improve: Hands-on exercises comparing different AI interaction patterns #### 🧠 Logical learners - Present: Statistical analysis, methodology critique, cause-effect relationships - Improve: Decision trees for AI usage, more quantitative comparisons #### 🧑‍🤝‍🧑 Social learners - Present: Community implications, team collaboration aspects - Improve: Discussion prompts for team meetings, collaborative exercises #### 🧘‍♀️ Solitary learners - Present: Individual reflection opportunities, personal development focus - Improve: Self-assessment rubrics for AI usage, personal learning plans #### 🌿 Naturalistic learners - Present: Ecosystem thinking about developer communities - Improve: More connections to natural learning processes and environments ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Clear heading structure, descriptive video content - Improve: More detailed alt text for any visual elements, transcript availability #### Keyboard navigation - Present: Video player supports keyboard controls - Improve: Ensure all interactive elements are keyboard accessible #### Contrast and readability - Present: High contrast video content, clear audio - Improve: Closed captions for better comprehension #### Media and captions - Present: Audio content with clear narration - Improve: Professional captions, downloadable transcript ### 🌍 DEIB Assessment #### Cultural breadth - Present: Technology-focused examples relevant globally - Improve: More diverse geographic examples of AI adoption #### Inclusive language - Present: Generally neutral technical language - Improve: Avoid assumptions about educational background or experience #### Neurodiversity support - Present: Clear section breaks, consistent structure - Improve: Content warnings for sensitive topics (animal research) #### Representation - Present: Mix of junior and senior developer perspectives - Improve: More diverse voices and experiences in examples ### ✅ Logical Validity #### Strong reasoning - Deductive strength: Logical critique of study methodology and scope limitations - Inductive strength: Drawing patterns from personal experience with AI tools - Causal reasoning: Connecting AI usage patterns to learning outcomes - Analogical strength: Skateboarding analogy effectively illustrates motivation and persistence #### Fallacies - Appeal to personal experience: Overreliance on individual anecdotes about AI productivity, but acknowledged as such - Hasty generalization: Study sample size critique valid but could be more precise about statistical requirements #### Tone of voice - Overall tone: Passionate, opinionated but substantive, mixes technical analysis with personal perspective ### 🛡️ DARVO Analysis - Deny: No evidence of denial patterns - Attack: Strong criticism of study methodology but focused on technical limitations rather than personal attacks - Reverse victim/offender: No evidence of victim/offender reversal - Emotional cues and rhetoric: Passionate delivery but grounded in technical arguments and data ## 🎯 Conclusion The Anthropic study reveals important tensions between AI-assisted productivity and skill development, particularly for junior developers. While the research has methodological limitations that restrict its broader applicability, it raises valid concerns about cognitive offloading and the need for balanced AI adoption strategies. The key insight is contextual: AI tools can accelerate experienced developers while potentially hindering skill acquisition for beginners. The solution isn't avoiding AI but using it thoughtfully --- ## Full Transcript [00:00] I think we can finally put the debate [00:01] about how useful AI is for coding to [00:03] rest. It is. We don't know to what level [00:06] yet, but we know it can make you way [00:07] faster as a developer. What we don't [00:09] know yet is the impact on us. How does [00:12] this affect ourselves and our skills and [00:13] the things that we're capable of [00:15] building? Are we running to a future [00:16] where everyone is too dumb about code [00:18] stuff to make changes and actually fix [00:20] things because we just get used to the [00:21] agents doing it? These are actual [00:23] interesting and kind of scary questions. [00:25] And Anthropic is now taking the time to [00:27] research it themselves. They just [00:28] published a new paper. how AI impacts [00:30] skill formation. And it's really, really [00:32] interesting. They took multiple groups [00:34] of developers and had them build the [00:36] same project, some with AI, some [00:38] without, and then checked to see how [00:41] much they understood what they built and [00:42] how well they could answer questions [00:44] about what they built afterwards. And [00:46] the results are fascinating. On average, [00:48] participants in the AI group finished [00:49] about 2 minutes faster, although the [00:51] difference was not statistically [00:53] significant. There was, however, [00:55] significant differences in test scores. [00:57] The AI group averaged 50% on the quiz [00:59] compared to 67% in the handcoded group, [01:02] or the equivalent of nearly two letter [01:04] grades. The largest gap in scores [01:06] between the two groups was on debugging [01:08] questions, suggesting that the ability [01:09] to understand when code is incorrect and [01:11] why it fails may be a particular area of [01:14] concern if AI impedes coding [01:16] development. That is scary. We all might [01:19] be getting much dumber really fast. [01:22] There's a lot to dig into here, and I [01:24] obviously have a lot of feelings about [01:25] it. Some of those feelings are bad, but [01:27] some of those feelings are good. Do you [01:29] know what else I feel good about? [01:30] Today's sponsor. Today's sponsor is a [01:31] product that I use every single day. I [01:33] can only imagine how many bugs I would [01:35] have shipped without it. It's Code [01:36] Rabbit, the AI code review that my team [01:38] chose for T3 chat, ping, and all of the [01:40] other crazy things that we build. We've [01:41] tried all of them, and this is what the [01:43] team settled on. And honestly, I [01:44] understand why. The quality of the [01:46] reviews is great. The integrations are [01:47] great. Its understanding of us is great. [01:50] You tell it you want to do something [01:51] different, and it remembers and does [01:52] things differently just like a human [01:54] would. It's actually better than humans [01:56] at code review because it'll do much [01:57] more meticulous stuff. But I'm not going [01:59] to sit here and say you should replace [02:00] your engineers with an AI for code [02:02] review. You shouldn't do that. You [02:03] should use these tools to augment your [02:05] teams. How many times have you read a [02:06] PR, found obvious mistakes, and then to [02:08] wait a day or two for somebody to go fix [02:10] it? Flip that. Now imagine if instead of [02:12] waiting days for a review, you're [02:13] waiting minutes. You get the feedback, [02:15] you go make the changes, and you don't [02:17] bother a human until you know the code [02:18] is good. This can get down to a much [02:20] more granular level. Why even push until [02:22] you've had the review done? That's why [02:24] they now have a CLI and IDE plugin. [02:26] Whether you're in VS Code, Cursor, or in [02:28] your terminal, you have tools now to use [02:31] Code Rabbit to review your changes [02:32] before you even ship. And if you do try [02:34] out the CLI reviews, you can include [02:36] this in the loop for your codegen. Use [02:38] something like Cloud Code, Codeex, or [02:39] Open Code, add in the reviewer CLI, and [02:42] now when it makes a mistake, it will [02:44] catch it and fix it before you even get [02:45] bothered. It's so weirdly powerful to [02:47] add code review to your loop when you're [02:49] building with agents because the quality [02:50] of the output code just goes up a ton as [02:53] a result. There's a reason everyone's [02:54] using Code Rabbit, even the Linux [02:56] Foundation. Yes, that Linux Foundation. [02:59] Also, Bun, Clerk, Cheg, Ashb, and so [03:02] many other companies. There's a reason [03:04] we all pick Code Rabbit. And it's not [03:05] just because it's free for open source, [03:06] which by the way, free for open source. [03:08] Figure it out now at [03:09] soyf.link/codrabbit. [03:11] Before we dig too deep into the numbers, [03:12] I think it's important to better [03:14] understand how the study worked and was [03:16] structured. It's very important to get [03:18] those things before you just go share [03:19] numbers around because I've already seen [03:21] people saying, "Look, this is proof AI [03:23] makes you 15 to 20% dumber." Not that [03:26] simple. We recruited 52 mostly junior [03:28] software engineers, each of whom had [03:29] been using Python at least once a week [03:31] for over a year. We made sure they were [03:33] at least somewhat familiar with AI [03:35] coding assistance and were unfamiliar [03:37] with Trio, the Python library on which [03:39] our tasks were based. We split the study [03:41] into three parts. A warm-up, the main [03:43] task consisting of coding two different [03:45] features using Trio, which requires [03:47] understanding concepts related to [03:48] asynchronous programming, a skill often [03:50] learned in a professional setting, and a [03:52] quiz. We told participants that the quiz [03:54] would follow the task, but encourage [03:56] them to work as quickly as possible. We [03:58] designed the coding task to mimic how [04:00] someone might learn a new tool through a [04:02] self-guided tutorial. Each participant [04:04] was given a problem description, starter [04:06] code, and a brief explanation of the [04:08] trio concepts needed to solve it. We use [04:10] an online coding platform with an AI [04:11] assistant in the sidebar which had [04:13] access to participants code and could at [04:15] any time produce the correct code if [04:17] asked. Interesting. They're using a web [04:19] interface for this. There's a lot of [04:21] layers of things that are interesting [04:23] here from the mostly junior to using a [04:26] library they were unfamiliar with to [04:28] specifically being told and encouraged [04:30] to work as quickly as possible, but also [04:32] that a quiz would occur later on. So to [04:35] see just like what the flow of this [04:36] looks like, you have two groups, [04:38] treatment and control. Control has no AI [04:40] throughout. Treatment has no AI for the [04:42] first task, which is a warm-up coding [04:43] task. They do have AI for the second one [04:46] which is the core task of creating this [04:48] project or solving these problems they [04:50] were given. Then they have posttask quiz [04:52] which they 25 minutes to do and then a [04:54] survey at the end about things they did, [04:56] what they thought etc. In our evaluation [04:58] design, we drew on research in computer [05:00] science education to identify four types [05:02] of questions commonly used to assess [05:04] mastery of coding skills. Debugging, [05:06] code reading, code writing, and general [05:08] concepts. Our assessment focused most [05:10] heavily on debugging, code reading, and [05:12] conceptual problems as we considered [05:13] these to be the most important for [05:15] providing oversight of what is [05:16] increasingly likely to be AI generated [05:18] code. And again, the results, as [05:21] mentioned before, on average, [05:22] participants in the AI group finished [05:24] slightly faster, but also perform [05:26] meaningfully worse on the test. There [05:28] are important pieces to measure here, [05:30] and this is where I'm going to start my [05:31] push back. I want to talk about the [05:33] twominut thing. When you're first [05:35] learning a new environment of any form, [05:38] it takes some amount of time to ramp up [05:40] on it to figure out how it works, what [05:42] goes where, what key bindings function. [05:44] If I have to use a computer or an [05:45] environment that isn't mine, I have to [05:47] spend the first 5 10 minutes just [05:49] adjusting. So, I have a bunch of weird [05:50] key bindings on my computer. I hate that [05:52] I do this. I try to be as stock as [05:53] possible on my machines, but there's a [05:55] few I just can't give up. I use [05:57] Carabiner heavily. It's a way to rebind [05:59] your keys on Mac, and I have a handful [06:01] that I I live on. For example, I have [06:03] caps lock bound as hyper and then other [06:05] number keys for switching between apps. [06:07] So one is helium, two as my terminal, [06:10] three is my editor or as notion. And [06:12] being able to switch between my apps [06:13] with one key press like this is very [06:15] very useful when I'm doing my day-to-day [06:16] work. I also have caps lock and U and O [06:19] as page up page down and JKL for my [06:21] D-pad. So I don't have to move my hand [06:23] all the way down to the arrow keys to [06:25] move around in my editor. So, like if I [06:27] want to just go up to, I can do it [06:28] without having to move my hand over to [06:30] the arrow keys or the trackpad. Just [06:32] little things like that that I've built [06:33] up over the years that I I can't really [06:36] use a computer without. And I've had [06:38] this problem when I was doing job [06:40] interviews, by the way, where I would go [06:42] in and they would give me a computer to [06:43] use during the interview and I could [06:45] barely use it cuz I could barely use [06:47] their keyboard cuz I just it's not mine. [06:48] I'm not used to it's not how I'm set up. [06:50] If you give all of these people a brand [06:52] new environment, at least half of the [06:54] time is going to be spent figuring out [06:56] the environment, especially in a [06:57] 30-minute window, it would take me more [06:58] than 30 minutes to figure out some weird [07:00] AI vibe coding tool in the browser. So, [07:02] if half the time is setting up, some of [07:05] the time is reading and figuring out the [07:07] code. The only piece that's left for [07:09] typing is like 5 to 10 minutes of the [07:11] actual time, obviously, the AI is not [07:13] going to be that helpful there. But, if [07:14] you're using environments you're already [07:16] familiar with, the time savings are [07:18] crazy. I have had many a time where [07:21] legitimate difficult work that would [07:23] have taken me hours if not days took an [07:25] agent 10 to 20 minutes. And I've had [07:27] times where work that would have taken [07:28] me weeks took an agent 3 to four hours [07:31] of an overnight run. Like you cannot [07:34] measure this in a study so simply when [07:37] it's in a 35minut window and half the [07:39] time is spent figuring out the tool. The [07:41] number measurement for time here is we [07:43] should just throw it out. It is not [07:45] valuable at all. Just let's be real. The [07:47] percentage gap in their understanding, [07:49] however, that is somewhat real and we [07:51] definitely want to dig more into that. [07:52] One of the pieces they do dig into here [07:54] that I think is really important to [07:55] learn from is the AI interaction modes. [07:58] This isn't like a button they hit in the [07:59] settings. This is how they actually used [08:01] the AI. We were particularly interested [08:03] in understanding how participants went [08:04] about completing the tasks that we [08:06] designed. In our qualitative analysis, [08:08] we manually annotated screen recordings [08:09] to identify how much time participants [08:11] spent composing queries, what types of [08:13] questions they asked, the types of [08:15] errors they made, and how much time they [08:16] spent actively coding. One surprising [08:18] result was how much time participants [08:20] spent interacting with the AI assistant. [08:22] Several took up to 11 minutes, which is [08:24] 30% of the total time allotted, [08:25] composing up to 15 queries. This helped [08:28] explain why on average participants [08:29] using AI finished faster, although the [08:31] productivity improvement was not [08:32] statistically significant. This sent [08:34] should not even be in here. The fact [08:35] that they're trying to measure the [08:36] productivity win in a 30-minute task [08:39] means the people who built this study [08:40] and threw it at a bunch of juniors [08:42] fundamentally do not understand how we [08:43] operate with these tools. And I want to [08:45] really emphasize this point from my [08:47] understanding of the study that I have [08:49] read separately. My understanding is [08:51] that this is interesting in the limited [08:53] scope of how students and really really [08:56] really early developers when first [08:59] introduced to an AI tool similar to like [09:01] the first time they open their IDE. How [09:03] does using it with AI compare to using [09:04] it without in like the limited one to [09:07] two hour starting window? I do not like [09:09] the fact that people are trying to [09:11] project the study beyond that, [09:12] especially on the speed side. I've [09:14] already seen people say, "Look, there's [09:16] proof that AI doesn't actually speed you [09:18] up as a dev." Recent studies are showing [09:19] a pretty consistent 30% improvement in [09:22] code output year-over-year for [09:24] developers who are using AI in real [09:26] companies. So, no, this is [ __ ] [09:29] There is parts here that we should dig [09:30] into though, so I'm not trying to [09:31] discount the whole thing. I'm just [09:32] trying to limit the scope of how much we [09:34] use this to prescribe what's going on. [09:37] They do call us out a little bit. We [09:38] expect AI would be more likely to [09:40] significantly increase productivity when [09:42] used on repetitive or familiar tasks. It [09:44] goes further. It improves a lot more too [09:46] when you're also familiar with the AI [09:48] tools you're using in the development [09:50] environment that you like. Once you get [09:52] yourself in the harness in the tool set [09:54] that you feel good about and get good at [09:56] it, you start shipping at speeds that [09:58] don't make sense. I wish I had my other [10:01] laptop plugged in right now so I could [10:02] show you guys. I was last night working [10:03] on four projects at the same time with [10:05] codecs. It's insane. I'm building like a [10:08] new app every day right now to just test [10:10] out ideas. Obviously, I'm not shipping [10:11] all of these things, but I'm putting [10:13] time in to experiment with different [10:14] ideas to see what is and isn't possible. [10:16] And I've also seen some of the best [10:17] engineers I know, people building crazy [10:19] libraries, having AI agents go out they [10:23] have for a feature and being not just [10:25] surprised that it could do it, but by [10:27] decisions it made that informed them on [10:28] how they would go write it by hand [10:30] later. There's a story that's going to [10:31] come out soon about this from a really, [10:33] really talented full stack front-end [10:35] framework person that y'all already know [10:37] and love. A lot of the AI haters know [10:39] and love, and I cannot wait for him to [10:40] share the story because it is crazy. [10:42] Back to the study, though. [10:43] Unsurprisingly, participants in the no [10:45] AI group encountered more errors. These [10:47] included errors in syntax and in trio [10:49] concepts, the latter of which mapped [10:51] directly to topics tested on the [10:52] evaluation. Reminder, this is a Python [10:56] test. These people are making syntax [10:59] errors in Python. I think I've said all [11:01] I have to about the quality of the [11:02] people being tested. Our hypothesis is [11:04] that the participants who encountered [11:06] more trio errors, namely the control [11:07] group, likely improved their debugging [11:09] skills through resolving these errors [11:11] independently. Sure. Just to emphasize [11:14] the like weirdness of how people were [11:16] using these tools in the study and the [11:18] just to be frank the junioress of the [11:21] people who were in this study. The [11:23] researchers noted that half of the AI [11:25] using cohort spent most of their time [11:26] literally retyping the AI generated code [11:29] into their solution instead of copy [11:31] pasting or manual coding. They wrote [11:33] their code from scratch with light AI [11:34] guidance. If you ignore the people who [11:36] spent most of their time retyping, the [11:38] AI users were 25% faster. I confess this [11:41] baffles me. What kind of person manually [11:42] retypes AI generated code? Did they not [11:45] know how to copy paste? Unlikely since [11:47] the study was mostly composed of [11:48] professional or hobby developers. It [11:50] certainly didn't help them on the quiz [11:51] score. The retypers got the same low [11:53] score as the pure copy pasters. In any [11:55] case, if you know how to copy paste or [11:57] use an AI agent, I wouldn't use this [11:58] paper as evidence that AI will not be [12:00] able to speed you up. Thank you, Sean, [12:01] for the observations here. Great [12:03] article. If you want to read other [12:04] opinions on this, it'll be linked in the [12:05] description. But back to my thoughts, [12:07] cuz I have so much to say. They try to [12:10] break down the different patterns in [12:11] which people interact with the AI here [12:12] to give you an idea of what the results [12:15] were from how they used it. The patterns [12:17] that result in the lowest scores were [12:18] ones where the users the test subjects [12:20] would rely heavily on the AI either [12:23] through code generation or debugging. [12:24] The average quiz score in that group was [12:26] less than a 40%. They showed less [12:28] independent thinking and more cognitive [12:29] offloading. We further separated that [12:31] into a few different groups. AI [12:33] delegation which was participants in the [12:35] group who relied on AI to write the code [12:37] and complete the task. They completed [12:38] the task the fastest and encountered few [12:39] or no errors in the process. Progressive [12:41] AI reliance participants in the group [12:43] started by asking one or two questions [12:45] but eventually delegated all codew [12:47] writing to the AI assistant. They scored [12:49] poorly on the quiz largely due to not [12:50] mastering the concepts in the second [12:52] task. And then the iterative AI [12:53] debugging group. Participants in this [12:55] group relied on AI to debug or verify [12:57] their code. They asked more questions [12:58] but relied on the assistant to solve the [13:00] problem rather than clarify their own [13:02] understanding. They scored poorly as a [13:03] result and were also slower at [13:05] completing the two tasks. High-coring [13:07] interaction patterns. We considered [13:09] high-coring quiz patterns to be [13:11] behaviors where the average quiz score [13:12] was 65% or higher. Participants in these [13:14] clusters used AI both for codegen and [13:17] conceptual queries. Generation then [13:19] comprehension. Participants in this [13:20] group first generated code and then [13:22] manually copied or pasted the code into [13:24] their work. After their code was [13:26] generated, they asked the AI assistant [13:27] follow-up questions to improve their [13:29] understanding. These participants were [13:30] not particularly fast when using AI, but [13:33] they showed a higher level of [13:34] understanding on the quiz. [13:35] Interestingly, this approach looked [13:37] nearly the same as that of the AI [13:38] delegation group, except they used the [13:40] AI to check their own understanding. One [13:42] other thing of note here, n equals 2. [13:45] The other three ones before, n equals 4. [13:49] Two people did this in the study. This [13:51] is not some crazy comprehensive group of [13:53] people testing all sorts of theories. [13:55] This is like 50 people. And out of the [13:58] 50 people, 25 got to use AI and two of [14:01] them happen to use it in this way. Two [14:03] out of 25 is bad. That's less than 10%. [14:05] But do you know why so few people use it [14:07] to clarify their understanding? Remember [14:09] earlier we encourage them to work as [14:11] quickly as possible. How much effort do [14:13] you expect the AI developers who are [14:15] still insecure about their own [14:17] abilities, who are still learning how [14:18] code works when told to write code fast, [14:22] do you actually honestly expect them to [14:24] use those same tools to better their [14:25] understanding? And to be clear, I'm not [14:27] sitting here pretending that every [14:29] developer uses AI to teach themselves [14:31] things, but I like to think it'd be a [14:33] lot higher than 2% if you told them to [14:34] be quizzed after. Like, what the [ __ ] [14:37] It's two people. It's not a real useful [14:39] thing. I like they're being so [14:40] transparent about the numbers here, but [14:41] that's two people. You can't drive home [14:44] any real learnings from this. This is [14:46] not a statistically significant study. [14:48] Hey, Claude, can you contact some people [14:49] and get a paper together? Yeah, actually [14:53] though again there are interesting [14:54] things to talk about here, but like we [14:57] need to be realistic. There were also [14:58] the hybrid code explanation people. [15:01] There were three of them. Participants [15:02] in this group composed hybrid queries [15:04] which they asked for code generation [15:05] along with explanations of the generated [15:07] code. Reading and understanding the [15:09] explanations they asked for took more [15:10] time but helped in their comprehension. [15:12] And then there was the conceptual [15:14] inquiry. Participants in this group only [15:16] asked conceptual questions and relied on [15:18] their improved understanding to complete [15:20] the task. Although this group [15:21] encountered many errors, they also [15:23] independently resolved them. On average, [15:25] this mode was the fastest among [15:27] highscoring patterns and the second [15:28] fastest overall after AI delegation. You [15:31] get the idea. They did their best to [15:33] break these people into groups, but [15:35] there just weren't enough people to get [15:37] meaningful insights. When they created a [15:39] group out of behavior of two people, it [15:41] says all that you need to. A follow-up [15:43] study I would love to see for something [15:45] like this is, okay, there's a lot of [15:47] things I would want to see. I would want [15:49] more experienced developers. I would [15:50] want a wider variety of experienced [15:52] developers. I'd want a much larger [15:53] number of developers. And I would want [15:54] much harder tasks, ideally without the [15:57] same time cap put on them. It would be [15:59] so cool to see an idea for a general [16:02] project. Like I'm working on a Frame.io [16:04] alternative because I'm tired of dealing [16:05] with it for managing my team and I want [16:07] to fully exit Adobe. It's the only piece [16:08] of Adobe software we're still using. If [16:10] you're not familiar, Frame is like code [16:11] review plus Dropbox for video and it's [16:14] really cool. It also sucks to use more [16:16] and more every day because the site's [16:17] just slow as [ __ ] and like every time I [16:20] go to it, it signs me out. I sign back [16:21] in. I have to wait through three loading [16:23] screens. I have to go through a bunch of [16:24] really slow, miserable [ __ ] It's not [16:26] great. I built a fully functioning [16:27] alternative in under an hour. And I [16:29] spent half of that time evaluating [16:31] different storage providers to see which [16:33] ones would provide the best speeds on [16:35] good internet connections because I have [16:37] gigabit, my editor is 5 gig. I want to [16:39] make sure we can saturate our [16:40] connections with this. It'd be really [16:42] cool to see a study that takes a project [16:43] like that, has a group with no AI and a [16:46] group with AI, and see how long it takes [16:48] both to complete it, see how good the [16:51] products they complete function, how [16:52] solid the tech decisions they make are, [16:54] and how good their understanding of the [16:56] code is. I would bet for something like [16:57] that, the results would vary wildly. I [17:00] am promising you as somebody who is [17:02] known specifically for building [17:03] applications fast and building a whole [17:05] tech stack around making application [17:07] development safer and faster. That's why [17:09] I built T3 stack. The goal was to make [17:11] it easier to ship fast without running [17:13] into errors in your final outputs. I [17:16] would literally guarantee that at most [17:18] skill levels, the AI generated versions [17:20] would not just come out faster, but [17:22] would come out more stable and with less [17:24] bugs and probably be better experiences [17:25] overall. And for the junior devs, they'd [17:27] have way better tech decisions, too. The [17:29] best devs I know are telling me about [17:31] how the AI is making better decisions [17:32] than them. We are past that point. And [17:35] this notice nothing in here, not a [17:37] single thing in here has mentioned the [17:39] quality of the code that was output. The [17:41] word quality doesn't even come up. They [17:43] also had one framing here that I have [17:45] feelings on. The agent could at any time [17:48] produce the correct code if asked. The [17:52] correct code. It is possible the person [17:54] who wrote this does not code much if at [17:57] all and doesn't understand how we work [17:59] as developers. If you've written code [18:01] for any amount of time, you know that [18:02] there is no such thing as the correct [18:04] code. There are very few problems in [18:06] software that can't be solved dozens if [18:07] not hundreds of different ways. If there [18:09] is one correct code answer to the [18:12] problem they're asking, then we should [18:14] honestly throw this entire [ __ ] study [18:16] out cuz it's useless. Happy you guys get [18:18] this chat. There can be multiple correct [18:19] codes. That's interesting. Correct code. [18:21] There are hundreds of ways to solve most [18:23] problems. Yeah, I have feelings on this. [18:25] I don't think this study was done in a [18:28] way that maps well enough to real world [18:31] use for me to care about it beyond the [18:34] potential in slight differences in use [18:36] characteristics resulting in different [18:38] behaviors and different learnings as a [18:39] result. I do want to be careful when I'm [18:41] talking about things that affect like [18:42] the junior developers and people who are [18:44] learning how to code right now because [18:45] it's very different from how I use AI [18:47] and how I talk about it. I have been [18:49] writing code for a very long time. I [18:51] know a lot of things that can go right [18:52] and wrong. And I've had even in [18:54] languages and ecosystems I don't [18:55] understand. I'm able to get a rough idea [18:57] of where things can go wrong. I'm deep [18:59] in building multiple native apps with [19:01] AppKit and Swift, which is a stack I've [19:03] never used before. And I still have a [19:05] good enough intuition of where a thing [19:06] might be going wrong to steer the agent [19:08] the right way, even without reading the [19:09] code. That is not what you should be [19:11] doing if you're still learning how to [19:12] code. That is certainly not what you [19:13] should be doing if you've never been [19:14] paid to code yet. You still have to [19:16] build up the skills. And to be clear, [19:18] this does not mean you shouldn't use AI. [19:19] You shouldn't use it to complete your [19:20] homework for you, but you should use it [19:22] to better your own understanding of [19:24] things and also at times reach beyond [19:26] your own capabilities to experiment with [19:27] ideas. I have a whole video about junior [19:29] dev specifically that didn't perform too [19:31] great. So, I will not be talking about [19:33] y'all anymore because you don't actually [19:34] care to get better. You just complain [19:36] when I don't give you enough credit. Not [19:37] every junior, just the ones who are in [19:38] my comment section. There's a lot of [19:40] y'all in the comments section. You're [19:42] certainly writing a lot more comments [19:43] and probably not writing that much code. [19:44] But, I do want to touch on one point [19:46] from this video because I think it's [19:47] really important. The htmx.org org [19:49] account might seem like a weird thing to [19:51] site here, but the HTMXorg creator is a [19:54] professor who teaches computer science [19:56] in a real college. He's one of the best [19:58] computer science professors I've ever [20:00] met, talked to, or seen the work of. [20:02] He's genuinely great. I love Carson. One [20:05] of the cool things he did is give his [20:06] students a custom agent MD file for [20:08] steering their agents in a way that will [20:10] help them learn instead of cheat. AI [20:12] agent guidelines. This file provides [20:14] instructions for AI coding assistants [20:15] working with students in this course. [20:17] Primary role teaching assistant not code [20:20] generator. AI agents should function as [20:22] teaching aids that help students learn [20:24] through explanation, guidance, and [20:25] feedback, not by solving problems for [20:27] them. What agents should do, explain [20:29] concepts, point students to relevant [20:31] lecture materials, review code that [20:32] students have written. You get the idea. [20:34] Agents shouldn't write entire functions [20:36] or complete implementations, generate [20:37] full solutions for assignments, complete [20:39] to-do sections in assignment code, [20:40] refactor large portions of student code, [20:42] provide solutions to quiz or exam [20:44] questions, etc. very much steering the [20:47] models away from cheating and helping [20:49] more towards teaching and learning. This [20:51] is great. I want to see more of this [20:54] because the student case, the junior [20:56] case, and the experienced developer [20:58] case/team/enterprise [21:00] case are so far separated that the [21:03] attempts to talk about them like they're [21:04] the same are almost insulting. Anthropic [21:06] even calls this out in their conclusion [21:08] section here. Prior studies have found [21:10] mixed results on whether AI helps or [21:12] hinders code productivity. Our own [21:14] research found that AI can reduce the [21:15] time it takes to complete some work [21:17] tasks by 80%. A result that may seem [21:20] intention with the findings presented [21:21] here. Yeah, no [ __ ] But the two studies [21:24] ask different questions and use [21:25] different methods. Our earlier [21:26] observational work measured productivity [21:28] on tasks where participants already had [21:30] the relevant skills also were using the [21:32] tools that they are familiar with. While [21:34] this study examined what happened when [21:36] people are learning something new, it's [21:38] possible that AI both accelerates [21:40] productivity on well-developed skills [21:41] and hinders the acquisition of new ones, [21:44] though more research is needed to [21:45] understand this relationship. There's [21:47] also one more key piece that this study [21:49] doesn't touch on at all that I think is [21:51] really important is motivation. The [21:54] harsh reality of building things and [21:56] doing hard stuff is that it's hard to [21:59] stay through. I'm going to do my [22:00] favorite analogy that you guys who have [22:02] been here for a while are very familiar [22:04] with my skateboard. A lot of people have [22:07] tried to skate. It's fun. It's cool. [22:09] It's rad. Skateboarding is also really, [22:12] really hard. Skateboarding is why I had [22:14] to have my whole left hand [22:15] reconstructed. And to this day, I have [22:17] this brutal scar for my entire left [22:20] thumb having to be reattached to my hand [22:22] just a few months ago. Yet, I still do [22:24] it. I love my skateboard. It is a thing [22:28] that is very important to me. It is one [22:29] of the coolest, best, most rewarding [22:31] skills I have taken the time to teach [22:33] myself. But there's a thing that sucks [22:35] about skateboarding. It hurts. [22:38] Literally, every time you try a trick, [22:41] there's a risk. It physically hurts you. [22:43] At the very least, it will physically [22:45] exhaust you some amount. The amount of [22:46] energy a skateboard takes from you is [22:48] unbelievable. It also hurts you mentally [22:51] a bit, too, because every time you fail, [22:53] you feel the failure. You see people do [22:56] the tricks so elegantly, so easily, and [22:58] then you go try it yourself and you can [22:59] barely get the board off the ground. 95% [23:02] of skaters will never even ollie. [23:03] They'll never get the board off the [23:04] ground the first time because even that [23:07] can take a year to learn and more than [23:09] that to learn. Well, even just jumping [23:11] on a skateboard is so difficult. But [23:14] when you can get over that, you'll do [23:16] incredible things. Skateboarding has a [23:18] very strange problem where the interest [23:21] in skateboarding can grow, but [23:24] skateboarding itself does not. Over co a [23:27] lot of people were looking for new [23:28] hobbies and skateboard sales went up in [23:31] a way we've never seen before. It was [23:33] literally five times more sales than [23:35] we've ever seen in the history of [23:36] skateboarding. It was crazy. The issue [23:38] was that all the people who bought a [23:39] skateboard weren't motivated to learn [23:41] how to skate. They were motivated to get [23:43] something to do during co skateboard [23:45] sales rely a lot on buying new boards. [23:48] The wooden part, the deck on a [23:49] skateboard can be broken relatively [23:51] easily when you start skating harder. I [23:53] can break one in a few hours after [23:54] setting it up. Usually it takes me about [23:56] a week or two on certain brands from [23:58] certain things they do like carbon fiber [23:59] and [ __ ] inside. They can last a couple [24:01] months, but I at my peak probably went [24:04] through a hundred decks in a year. That [24:06] is a big part of how skateboarding gets [24:08] funded is people like me having to buy [24:10] replacements for the parts that we [24:11] break. And then that money is used to [24:14] continue to pay skateboarders to do [24:16] harder things to encourage more people [24:17] to skate and get the flywheel. It was [24:19] assumed when this huge surge in [24:21] skateboard sales happened that there [24:23] would be some continued interest. But [24:25] 99% of the people plus who picked up [24:28] skateboarding during the COVID break [24:30] never got good enough to break a board. [24:33] That huge surge in board sales resulted [24:35] in no long-term sales. And a ton of [24:38] skateboard companies that I've loved my [24:40] whole life went out of business. I've [24:42] rode the exact same board for 15 years. [24:44] And you can no longer buy them because [24:46] the company foreclosed because they [24:48] produce too much stock thinking that the [24:50] sales would continue and then they [24:51] didn't and now they're dead. So why the [24:52] [ __ ] am I so deep in talking about [24:54] skateboards? It's why I do hard things. [24:57] the fact that when I was young, I [24:58] learned that to do really cool, [25:00] rewarding [ __ ] that makes you feel this [25:02] special way, you had to be willing to [25:04] get hurt. You had to be willing to feel [25:06] stupid. And my skateboard taught me that [25:08] feeling dumb is actually good because it [25:11] means you'll feel smart soon. And if you [25:13] don't feel dumb, you're not pushing [25:14] yourself. But there are limits to this. [25:17] If it takes you a month to learn how to [25:19] oie, you're much more likely to continue [25:21] skating than somebody who it takes 3 [25:23] months for. because the person who it [25:24] takes three months for is three times [25:26] more likely to give up before they get [25:27] there. This is a thing I think about a [25:29] lot. On one hand, I love things that [25:32] have these hard filters where it's hard [25:34] enough to get into that everyone doing [25:36] it is really good. I love the fact that [25:38] this existed for so long in programming. [25:40] Anybody with real open source work got [25:42] there because they cared too much and [25:45] got over the moments where they felt [25:46] dumb to get there. Anyone who can do a [25:49] trick down a stair set or a handrail got [25:51] there because they pushed through the [25:52] feeling of this is too hard. I'm going [25:54] to get hurt. This isn't worth it. And [25:55] they pushed through it until they could [25:57] do that thing. On one hand, this means [25:59] the people who make it through are [26:00] exceptional. On the other hand, there [26:02] are lots of people who could be [26:04] exceptional that give up because they [26:06] don't have a good feeling early enough. [26:08] And I do think for a lot of things, [26:10] especially places where we don't have [26:11] enough people doing the thing, making it [26:13] easier to have that first, oh [ __ ] I [26:16] might actually be able to do this moment [26:18] is valuable. If it was easier to land [26:21] your first oie and harder to land your [26:23] first kick flip, there would be way more [26:25] skaters. Because if we have the timeline [26:27] of how long it takes to feel good about [26:29] skateboards, it's probably going to be [26:31] the first oie. And if we say the first [26:33] oie happens in 3 months and we say the [26:36] first kick flip happens in I don't know [26:38] 6 months. Think about all of the people [26:41] who fail before they get here. By the [26:43] time you get to the first oie, you've [26:45] already lost 70% of the people, but the [26:47] first kick flip, a lot more survive. I [26:50] personally don't know any person who can [26:52] oie but can't kick flip. I know a few [26:55] that are like they've done a few ollies [26:56] that are sketchy but work and they got [26:58] close to a few kick flips, but I don't [27:00] know a whole lot of people that made it [27:01] to this point that didn't make it to [27:04] this point. So imagine this hypothetical [27:06] world where getting the board off the [27:07] ground was easier than it actually is. [27:09] Let's say the first oi didn't take 3 [27:11] months. Let's say it took 2 weeks. [27:14] Suddenly this gets moved way way way way [27:17] further to the front. And if that drop [27:20] off rate was consistent, I would suspect [27:22] the result there is that you get a much [27:25] higher percentage making it all the way [27:27] through. And even if you move the [27:28] kickflip way further along, let's say [27:30] you move this to 9 months, the fact that [27:32] you had that first, oh [ __ ] maybe I can [27:34] do it moment so early means that that [27:37] huge drop off curve gets smoothed out [27:40] and a lot more people make it to the [27:41] end. Now, let's replace oie and kickflip [27:44] with hello world and usable app. Do you [27:47] understand? Now, if AI tools allow a [27:50] developer to not feel stupid for as long [27:53] initially, the likelihood that they make [27:56] it through goes up exponentially. I can [27:59] tell you from experience, knowing a lot [28:01] of people who were really into tech, [28:02] that like built their own computers that [28:03] loved that stuff, that tried like [28:05] installing Linux on the computer or like [28:07] building a small app for a game they [28:09] played that gave up because the initial [28:12] learnings was just too much and they [28:14] felt too dumb for too long. If you can [28:16] give them a taste of the reward earlier, [28:18] they will survive longer. Trigger [28:20] warning for those who are sensitive to [28:22] animals being hurt. We're about to talk [28:24] about a rat study that involves rats [28:26] dying, but I think it's actually really [28:27] useful for showing biologically that we [28:30] are built to use hope. The experiment [28:34] was done in the 1950s where domesticated [28:36] rats were placed in a bucket of water to [28:38] see how long they would swim before they [28:39] drowned. On average, they would give up [28:41] and drown within 15 minutes. But they [28:43] did a separate trial where just before [28:45] the rats were about to drown, the [28:46] researchers rescued them, dried them [28:48] off, let them rest for a while, and then [28:49] placed them back in the water. When [28:51] those rats were pulled out once and put [28:53] back in, they would swim for more than [28:55] 60 hours instead of the 15 minutes [28:58] before they gave up before. Once the [28:59] rats were saved, they developed a sense [29:01] of hope. And I want to make sure this is [29:04] as grim as it is that this is the point [29:05] that I emphasize here. If you get a [29:07] little bit of hope early on where you [29:09] were about to give up, the likelihood [29:11] that you keep going is exponentially [29:14] higher. It just is. That's a fact. And [29:17] if AI tools are the equivalent of being [29:19] pulled out of the water before you drown [29:21] the first time, that is dark. But that [29:23] is the world. The world's a dark place, [29:25] and you need hope and motivation to push [29:28] through it. If the AI keeps you from [29:29] developing motivation, then it's not [29:31] useful. But if the AI keeps you from [29:33] losing motivation, it is incredible. And [29:35] quick shout out to the lab mice that [29:37] have made the world as much better as it [29:38] is. As we were saying, if you compare [29:41] two people who built an app, one who [29:43] vibe coded it and one who wrote it by [29:45] hand, obviously the one who didn't vibe [29:46] code is going to be more likely to [29:48] succeed in the field. But if you take [29:49] two people who are starting programming, [29:51] one used AI to get something useful [29:54] early and the other didn't, the one who [29:57] didn't is more likely to give up. And I [29:58] want to make sure we find a balance when [30:00] we talk about these things. Like on one [30:02] hand, you shouldn't use AI tools to [30:03] avoid learning, but on the other, you [30:05] shouldn't spend so much time in that [30:06] space where you feel dumb that you burn [30:08] out and quit. If AI can unblock you, if [30:10] AI can accelerate you a bit, if AI can [30:12] get you excited about code again, it's [30:15] probably worth doing. That doesn't mean [30:16] you should replace everything with it. [30:18] It doesn't mean you should outsource [30:18] your brain. It means you should reach a [30:20] little further than you might be capable [30:22] of so that you can see things that you [30:24] might not have been able to and feel [30:26] things that you might have given up [30:27] before you could. Everyone needs to feel [30:29] good about the work they're doing [30:30] sometimes. And if AI helps you feel [30:32] better faster, use it in moderation. [30:34] Thank you for once again letting me rant [30:36] about skateboards a bit. It's my [30:37] favorite thing. I'm always happy when [30:38] you do it. Hope this was helpful for [30:40] somebody. And until next time, peace [30:42] nerds. ## Highlights - skill formation. And it's really, really interesting. They took multiple groups of developers and had them build the same project, some with AI, some without, and then checked to see how much they understood what they built and how well they could answer questions about what they built afterwards. And the results are fascinating. On average, participants in the AI group finished about 2 minutes faster, although the difference was not statistically significant. There was, however, significant differences in test scores. The AI group averaged 50% on the quiz compared to 67% in the handcoded group, ([View Highlight](https://read.readwise.io/read/01kgfwzbvqpv0fz71z2xvvcpx6)) - Tags: 👻 ai highlighted - to work as quickly as possible, but also that a quiz would occur later on. So to see just like what the flow of this looks like, you have two groups, treatment and control. Control has no AI throughout. Treatment has no AI for the first task, which is a warm-up coding task. They do have AI for the second one which is the core task of creating this project or solving these problems they were given. Then they have posttask quiz which they 25 minutes to do and then a survey at the end about things they did, what they thought etc. In our evaluation design, we drew on research in computer ([View Highlight](https://read.readwise.io/read/01kgfwzbwkcwwejk8nv608n0qd)) - Tags: 👻 ai highlighted - the AI. We were particularly interested in understanding how participants went about completing the tasks that we designed. In our qualitative analysis, we manually annotated screen recordings to identify how much time participants spent composing queries, what types of questions they asked, the types of errors they made, and how much time they spent actively coding. One surprising result was how much time participants spent interacting with the AI assistant. Several took up to 11 minutes, which is 30% of the total time allotted, composing up to 15 queries. This helped explain why on average participants using AI finished faster, although the ([View Highlight](https://read.readwise.io/read/01kgfwzby8et5yq8dmc7v6mfb8)) - Tags: 👻 ai highlighted - the first time they open their IDE. How does using it with AI compare to using it without in like the limited one to two hour starting window? I do not like the fact that people are trying to project the study beyond that, especially on the speed side. I've already seen people say, "Look, there's proof that AI doesn't actually speed you up as a dev." Recent studies are showing a pretty consistent 30% improvement in code output year-over-year for developers who are using AI in real companies. So, no, this is [ __ ] There is parts here that we should dig ([View Highlight](https://read.readwise.io/read/01kgfwzc04nmq86ztn3fqmwr5q)) - Tags: 👻 ai highlighted - them happen to use it in this way. Two out of 25 is bad. That's less than 10%. But do you know why so few people use it to clarify their understanding? Remember earlier we encourage them to work as quickly as possible. How much effort do you expect the AI developers who are still insecure about their own abilities, who are still learning how code works when told to write code fast, do you actually honestly expect them to use those same tools to better their understanding? And to be clear, I'm not sitting here pretending that every developer uses AI to teach themselves ([View Highlight](https://read.readwise.io/read/01kgfwzc3xh8h2m5jh9xr91xg4)) - Tags: 👻 ai highlighted - the relevant skills also were using the tools that they are familiar with. While this study examined what happened when people are learning something new, it's possible that AI both accelerates productivity on well-developed skills and hinders the acquisition of new ones, though more research is needed to understand this relationship. There's also one more key piece that this study doesn't touch on at all that I think is really important is motivation. The harsh reality of building things and doing hard stuff is that it's hard to stay through. I'm going to do my ([View Highlight](https://read.readwise.io/read/01kgfwzcark4nb9fq56rq5c723)) - Tags: 👻 ai highlighted