Ep. 370: Deep Work in the Age of AI  ## Metadata - Author: [[Deep Questions with Cal Newport]] - Full Title: Ep. 370: Deep Work in the Age of AI - Category: #podcasts - URL: https://share.snipd.com/episode/c8494292-06b4-431f-aac9-0a4f8509a7a2 ## Document Notes ## Highlights - **AI Can Slow Experienced Developers** - A METR study found experienced open-source developers were ~20% slower when allowed to use generative AI. - This surprising result challenges the assumption that AI always boosts programmer productivity. Transcript: Cal Newport This study came out in July of 2025. It was produced by a nonprofit called METR. That's often pronounced meter. It's a nonprofit that does evaluation of AI and its capabilities. They do high-quality studies. They don't have any particular bias. AI companies often cite their results when they find them to be useful or impressive. So this is sort of a neutral body that does these reports. Here's the title of their July report that caused a bit of a surprise. ([Time 0:01:36](https://share.snipd.com/snip/2a1e46e8-d988-42b5-8cd6-1cc002cc2ff6)) - **Programming Tasks Are Deep Work** - The tasks studied required deep work: focused, cognitively demanding effort to create or fix code. - Deep work is the primary source of valuable output in knowledge work, not shallow tasks. Transcript: Cal Newport We got to get specific here. Now, I'm going to use a term that most of my audience is familiar with. Computer programming, handling these tasks, which are a combination I looked into more deeply, but it's a combination of creating original code or fixing code. They require what I call deep work. Deep work, as you know, is tasks that require you to focus without distraction on something that's cognitively demanding. So deep work is when you give your full attention to something that is demanding and you try to keep your attention on it as intensely as possible. I sort of wrote the book and it's sort of, I wrote the book on it. I coined the term. ([Time 0:06:10](https://share.snipd.com/snip/254c1d91-37fb-4361-8b70-f6bb82caaea3)) - **Cybernetic Collaboration Alters Work Rhythm** - Developers used interactive, back-and-forth prompting with AI and spent more time reviewing outputs than actively coding. - That review-and-wait loop replaced sustained coding focus with fragmented attention. Transcript: Cal Newport According to the study authors, the programmers were working interactively with these tools. So what they were doing was a lot of, can you produce me this code? I'll look at this code. Hey, can you fix this about it? Or this is not working. So kind of back and forth like that, or here's some code. You see any mistakes. Great. Then the programmer would run it and be like, it's still not quite working right. Can you try to fix it here? So it was a very interactive loop. They were going back and forth, having code produced, checking the code, having the AI check the code, looking at what it sent back. So think of it as like a back and forth interaction. Here is how, let me see here, here is how the paper itself describes this. Let's see here. Okay, here we go. Here text from the actual paper about this collaboration. When allowed to use AI, developers spend a smaller proportion of their time actively coding and reading, searching for information. Instead, they spend time reviewing AI outputs, prompting AI systems, and waiting for AI generations. Interestingly, they also spend a somewhat higher proportion of their time idle where their screen recording doesn't show any activity. ([Time 0:08:28](https://share.snipd.com/snip/5119b20f-62fd-418d-96ea-4f1dec69b9b7)) - **Whiteboard Effect Boosts Focus** - Collaborative deep work succeeds by increasing focus intensity and duration via social pressure and shared attention. - The 'whiteboard effect' accelerates concentration, producing better and faster cognitive outputs. Transcript: Cal Newport Well, here's the thing. If you talk to any professional thinker who does this, they'll say the same. I guess I should say the way you make collaboration help with deep work is that you use the presence of other people to increase the intensity and duration of your focus. So the underlying formula, focus produces results, deeper focus produces better results, that formula reigns supreme. So when I would sit down with my frequent collaborators to work on some sort of theorem or some sort of paper we're working on, the whole game was trying to get even more intense focus. ([Time 0:11:58](https://share.snipd.com/snip/43d20919-5ba5-4937-9fdc-daaa1a8b4164)) - **Pleasant Breaks Reduce Cognitive Gear** - Cybernetic collaboration with AI tends to decrease intensity and duration of human focus, making work slower despite feeling easier. - Pleasant breaks from AI interaction can reduce peak cognitive gear and productivity. Transcript: Cal Newport That's not what these programmers are doing. They are using collaboration with AI to reduce the amount of intensity of focus they have to experience. To get those breaks, you produce it, I'll look at it. It's easier for me to try to debug what you did that was broken than it is for me to have to produce it from scratch. It's easier for me to have these nice breaks while I'm waiting for the AI model to generate the code I'm going to look at, which can take a little while. We know this, Jesse, because we get this question all the time now. What should I do while waiting for the AI model to produce my code? We only put it on the show like once, but we get this question a lot. So that gives you a break. So it makes the experience more pleasant. There's actually the authors of this say this somewhere in the study that it was a more pleasant experience for the programmers because you get all these breaks. You don't have to be locked in when you're just looking at the bank blank coding page. No word, no piece of code is going to get there until you come up with it and write it. It's really hard. There's no real break until you tell yourself to take a break because you're trying to get all this code together. Compiling for small programs is fast, too. You don't get much of a break from it. And so it's much more pleasant. Cybernet collaboration means much less intensity of focus, much less duration of focus. It takes less energy. It feels nicer. But that's why they're slower. Because intensity of focus is what tells you how fast you're going to go. Intensity of focus tells you how good the stuff you're going to produce is when it comes to deep work. So the whiteboard effect says, yeah, come work with me so I can focus harder. Good stuff gets produced. Cybernet collaboration, by contrast, says I want a computer to offload some of the work so I get a lot more breaks. ([Time 0:14:56](https://share.snipd.com/snip/67007e28-2acf-4331-8444-dc5682a3c865)) - **Prioritize Intensity For Deep Work** - Protect and prioritize intense, sustained focus for tasks that require deep work. - Avoid adding tools that lower focus intensity unless they truly automate non-deep tasks. Transcript: Cal Newport Deep work rewards intensity of focus. And if you add anything into your workflow that's going to reduce this intensity, you'll probably get less productive. This seems to be the trap that a lot of knowledge workers experimenting with AI right now are falling into. Focus is hard. It doesn't feel pleasant. It's tempting to try to make it go away, but that doesn't make your work better. Now, I'm absolutely convinced that there will be upcoming applications of AI that will help our productivity. I think they will focus more on automating shallow tasks or speeding up things that don't require you to focus. But cybernetic collaboration, that is not the key to the AI future. Focus remains absolutely essential to doing deep work. ([Time 0:19:35](https://share.snipd.com/snip/abbc09cc-e3b6-4098-bcbf-b9f63e4f00c4)) - **Use Notes, Not Ongoing Chat Threads** - Be cautious using ChatGPT as an ongoing project notebook because it can encourage cybernetic collaboration habits. - Leave short narrative notes inside the work document to restore context and enable high-focus sessions. Transcript: Jesse Miller All right. First question is from Sam. I found that using ChatGPT questions can act as project diaries. I now keep multiple chats for projects I'm working on and enables a shutdown and a place marker for where I need to restart work. Can chat GPT chats act as single purpose notebooks for projects? I mean, sure. A couple of questions here. Cal Newport What is a project notebook, like this idea of like, I need a notebook to keep track of the progress on my project. Was that something you're doing before? I guess it's useful if you don't know where you left off and you want to have external notes on it. But here's my concern. My concern is that you're doing something like cybernetic collaboration with ChatGPT. We talked about this in the deep dive. When you have this running list of notes and conversations with chat GPT, I'm concerned what you're doing is transforming your work into something where you go back and forth with the Chat bot, you hand off more, you know, have them do some stuff and you look at what they did and you ask a questions, you go back and forth. This is more pleasant. This feels nicer. It makes work seem better. But just as the meter study found with programmers, if it's reducing your peak intensity of focus or the sustained duration of your focus and you're doing anything that's non-trivial, It also could be slowing you down and hurting the quality of your results. The key with any type of effort that requires deep work is not reducing the difficulty of the deep work. It's not reducing the intensity of focus. It's setting things up so that you can reach high intensity of focus. The focus mind is what produces value. So that's my only issue. I would be wary if you're trying to make work in this Cybernet collaboration that might not be making you as productive as you think, even if it feels nicer. Jesse Miller Last week, you talked about leaving a narrative for personal projects. And originally, I think this question was in relation to that. But now that you talk about the cybernet collaboration, it kind of makes sense. But if you could touch on the narrative part too. Yeah. ([Time 0:26:16](https://share.snipd.com/snip/29a07b9f-02c1-409f-a005-be58ab060342)) - **Refresh Memory Within Two Weeks** - Revisit newly learned but unused material within about two weeks to avoid forgetting details. - Use the material in projects or teach it to lock knowledge into long-term memory. Transcript: Jesse Miller When using active recall to study a large volume of material, how soon and how often should I revisit material I've already mastered? Right now, I find that when I return to something a few weeks after learning it, I've forgotten many of the details. Cal Newport Well, I mean, it depends what you're trying to do here. Like, what are you learning and why are you trying to hold on to it? If you learn something that you then use, it will stay of active in your memory and eventually will really dig in. If you're memorizing some sort of information that you just basically don't come back to again, you're going to need to retouch on that within a two-week window. Like if you let about two weeks or more go, your mileage will vary depending on the information in your brain. But if you let more than two weeks go without using information at all, it's going to start to lose its location. That is just like informal based on my experience as a student professor. So if it's like this is arbitrary information I want to remember, every week or so you want to keep going back and doing that active recall. Eventually, if you do active recall enough, it will lock in. But again, be sure that you need to, if you're not using this stuff, be sure you need to remember it. It's the best way to submit something in your memory is to use it, to do it. I, you know, you teach the concept a couple of times, you really remember it. You explain it to someone else because you're using it for a project, you know, you really remember it. Active recall, of course, simulates that. By recalling the information from scratch, you are firing up all the circuits you would use if you were actually applying the technique. So it's using your own brain's own memory apparatus. But yeah, I would find if I memorize something using active recall and just never touched it again, within about two weeks, I might start to forget it. ([Time 0:29:36](https://share.snipd.com/snip/744ca2c9-bd41-4f6b-96c1-321bd687e037)) - **Future AI Will Favor Small, Local Models** - Large frontier models (ChatGPT-style) consume far more compute per query than cached search like Google. - Cal predicts AI will shift toward small, task-specific models that run locally and reduce environmental and cost concerns. Transcript: Cal Newport Tools like google search all right i mean i do have a hot take here um if you want like the actual answers i don't know i don't have the numbers in front of me but something like a google search Is sufficiently uh significantly more energy efficient than like a chat gpt query i mean a chat gpt query you have to you have to have this frontier model loaded up in memory now this thing Could have hundreds of billions if not a trillion parameters defining it right so you're not going to fit into the memory of a single gpu chip you're gonna have to shard this thing over Like three four five maybe six gpus that are then going to have to be running all out just to generate tokens for your answers. I compare this to a Google search where they use commodity Intel chips. Most of this stuff is cached. They can be dynamic of like, oh, there's a little downtime on this chip. Hey, can you like go look something up in a hyper-efficient sort of cached search index? And the amount of computation they've got, it's down, you know, minuscule. Probably the answer you're looking for is in some CDN server that's like 10 miles away. So, you know, it's a lot of computational power. But here's my hot take. I might get yelled at for this one, Jesse. You ready for it? Yeah. I think right now the focus on the environmental impact of AI is in part a way for people who are critical of technology or big tech to get in on like, I don't like AI in a territory that they're Much more comfortable with than the technology. So if I'm just like, I don't know, like a typical like left-leaning critique of the tech industry and AI, I'm super comfortable for talking environmental stuff. Like you don't care about the environment. I do. I saw the Al Gore documentary. You don't know science. The environment's important. Like I feel like I'm smart there. When it's me talking to other people, like that's where I'm smart. Like I know more about this than you. You don't know science. Global warming is a real thing. So that's like a very secure place of critique. A less secure place of critique is, you know, the type of stuff I've been doing in my writing recently, like coming in and be like, here, I don't think that reinforcement based learning Post-training refinements is giving you the sort of lost reductions that you would have expected on a more elongated power lock curve. Like people getting into the technical details is very like, it's hard. And you're like, I don't really know this details well, but I, I don't like this and I don't like the people involved. And Sam Altman is kind of creepy and Mark Zuckerberg still talks like a robot whose emotion circuit board is shorted out. We've covered this on the show. And this is a territory that I'm much more comfortable in. But just like also really leaning into like the specific AI safety concerns over the particular words that a particular chatbot will or will not say, right? Well, I'm comfortable there. This is inappropriate speech. But I think if we're really thinking about AI, these are kind of transient concerns because I don't think a future of these massive frontier models being queried for everything is a Future that makes any economic sense anyways. Here's the right way to think about it. If querying one of these models is bad for the environment, that means the amount of computation it's using can't possibly be profitable. So they can't be bad for the environment too long because it correlates also with expense and people aren't willing to pay for the massive computational expense. I mean, stuff, it takes money to create heat because that requires electricity and electricity costs money. So like my thought about this is, um, yes, I would be concerned if we, we had chat GPT used all the time, but I just don't think it's profitable. I think the future of AI is going to be, it has to be, systems that have much smaller models, machine native models if possible, meaning like this thing is running on my iPad, not in the Cloud somewhere on a bunch of GPUs. Small specialized language models combined with other specialized components like policy networks for evaluating options, future simulators for trying to understand which actions To take next, control logic that's specific to the task at hand, put together into a program that you can get your arms around that can run on existing hardware and does really good at Being intelligent about a very specific thing. That I'm convinced is the future of AI. It doesn't have an environmental footprint that's different than other things that we're doing right now with our computer. So there probably is. It does use too much electricity, but I just don't think these massive frontier models are going to, it just doesn't make sense. ([Time 0:31:40](https://share.snipd.com/snip/49d3d1cd-60ae-4fb3-9131-ad1acc5dce74)) - **Writing And Podcasting Would Continue** - Cal would continue writing and podcasting if he left academia and expects it to occupy much of his time. - He already combines research, public writing, and teaching within his academic role. Transcript: Cal Newport I would continue to write and I would continue to podcast. I don't think I would become antsy. So I've kind of two reactions to this. No, I wouldn't become antsy. Writing and podcasting actually takes up a lot of time. And it's not like I'm hurting for other things to do. You know how many teams I'm coaching at my kids, like schools and sports leagues right now? Let me guess. Three, six. It's three. I'm coaching three teams right now. I don't have a lot of time doing them. I have a lot of things to do. That's fine. But here's the here's the here's the bigger point, though, is writing and podcasting is like a lot of what I do as an academic right now. So I don't know how different it would be. So, you know, for those who know my trajectory, I've been a writer my entire adult life. I started writing in college. I trained as a theoretical computer scientist focusing on the theory of distributed systems. So I studied at Nancy Lynch's theory of distributed systems group at MIT and did my postdoc under Hari Balakrishnan and his network and mobile systems group. So I had a computer science specialty that was on the math behind distributed systems. Went to Georgetown. This is where like my NSF funding was all about this. My papers were all about this. My grad students. We did a bunch of, I was pretty good distributed theoretician, published a bunch of papers around a bunch of steering committees. Somewhere around the time I was, you know, I got tenure, then I got full professor. Around the time I was getting full professor, Georgetown, my, where I work, they began making a real move for like, we want to be one of the places grappling with technology and its impacts. They call the field digital ethics. Like we want to be at the core of this because we're in Washington, DC. We have a big ethics background here. We were the university that like figured out bioethics in the 20th century, the Kennedy Center for Ethics. We want to do the same thing for tech ethics. We're in the, we're here in DC. We have all these policy centers. We have one of the biggest tech law faculties at our law school. Like it makes a lot of sense. And I was like, this is kind of what I'm doing already with my writing. I write about technology and how it impacts us. I want it. And so like I've really been focusing on that. So I was one of the founding faculty members of Georgetown Center for Digital Ethics. I'm the inaugural director of our computer science, ethics and society academic program. It's the first major in the country to integrate computer science and ethics in a combined major. There's places where there's majors where like you're a computer science major and you throw on a little bit of ethics. First integrated one in the country. So like, this is actually what I'm doing largely right now as an academic is technology, how it impacts us, what to do about it. And I do it in a lot of different forms. And I think public outreach is important. I think the podcast is very pragmatic, but I reach a lot of people this way. My writing for The New Yorker, now we're talking to a little bit more of like a rarefied crowd. This gets me in front of policymakers. This gets me in front of like the Senate and on NPR or whatever. But it's a way, again, to work on tech and its impact. I do some academic papers on tech and its impact, which is more for like an academic crowd. And then my books fall somewhere in between. So I don't really know what my life wouldn't be that much different if I left academia. The right and podcast right in podcasting is what I'm doing in academia. ([Time 0:38:50](https://share.snipd.com/snip/d6cedd4c-a09f-4ba2-94e4-b95cbb03c7c8)) - **From Farm To Tech: A Lifestyle Recalibration** - Vin moved from web design to working on an organic farm and later as a nature guide, but family and commute made it unsustainable. - He used evidence-based planning to revive web development skills, doubling his income and reclaiming time for nature. Transcript: Cal Newport I hold a bachelor's degree in interface design and in my 20s first worked as a web developer and later as a UX designer for several digital agencies. I enjoyed my work, but I felt stuck in big cities and kept dreaming about a life closer to nature. So in my mid-20s, I did something that you would surely advise against. Uh-oh. I threw my career into the trash and moved to Norway to live and work on an organic farm. I had a great time there and enjoyed every aspect of it. However, I didn't have a sustainable long-term plan. I realized I needed and wanted to move back to my home country where my girlfriend, now wife, lived. Back home, I decided that my next big move would be to find a job that allowed me to work in nature. So I signed up for a nature guide program and actually managed to land a job as a guide at the very place where I had completed that course. For the last six years, I worked there running workshops and similar programs. Sounds like a happy ending, right? Not quite. Life changed. I became a father of two children and we moved closer to my parents so we could get support from the grandparents with family life. This meant I was now living almost two hours away from my workplace, constantly torn between family and professional life. On top of that, since most workshops take place on weekends, I was away from my wife and our two little ones at the very time they needed me most. It turned out that the dream of working in nature definitely had its flaws. I'm going to stop there for a second. Let's take stock of where we are. In the story so far, we're seeing a common issue when people think about the deep life. So a common issue that people have is that you fixate on a single change that you begin to believe will make everything better. It's simpler to think about a single change. We can sort of inhabit that change. The idea of doing something radical itself makes us feel excited in anticipation. And we can sort of take the imagined feeling of like the best parts of that change and sort of expand that in our mind, like my life is going to be better. So Savin did this with nature. Like I like nature. So I'm just going to focus like a laser on if I could just be in nature all the time, I would be really happy. The problem, as we often say, when we talk about lifestyle centric planning, is that your daily subjective mood is not the result of a single decision or change, but on all of the relevant Aspects of your life, call this your lifestyle, all the different things that are relevant during the day add up to give you your subjective experience of the day. So you might like the part where I'm giving a workshop and it's in nature and that's nice, but you have other parts of your life too, like the fact that you had to drive two hours to get there And that you're kind of stressed out about what's going on with your parents. And like the kids are – you're not around enough to help and it's creating tension with your wife. All of your houses may be nowhere near nature and you're actually spending most of your time in a car. All of these other aspects of your lifestyle matter too. And this is how you can make one radical change that you're excited about and yet end up less happy than you were before. Because by making this thing better, you accidentally maybe made other things worse. So lifestyle-centric planning says you got to construct your ideal vision for an entire lifestyle, all the parts of your life. How would I reorient my life so that all of the parts were something that resonated? And then how do I make progress towards that? You're going to have, you're going to enjoy your life more if you explicitly construct it so that all of the things that contribute to your enjoyment are being pushed in ways that are Better. Let's return this fin because he recognized that and let's see how he made some changes. Around this time, I came across your books, Deep Work, So Good They Can't Ignore You and Slow Productivity. They gave me a framework to better understand my professional situation. I had to take a hard look at myself and realize that for the last five to six years, I hadn't built transferable career capital that could help me in a different role. After some reflection, I decided to revive my previously built career capital as a web developer. However, I hadn't coding seriously for years and looking at job descriptions, I realized how much had changed in the web development space and I had a lot of catching up to do. All right, so he's doing something here called evidence-based planning. So he makes this decision, he's looking at his life and is like, actually, I need a job that is going to whatever the criteria here, but probably like more money, more autonomy. I don't have to commute so much, whatever. Web development is like I have career capital there. Let me start there. But he does evidence-based planning. He actually looked at real job listings like, oh, skill A, B, and C I don't have. So I'm going to summarize a little bit here. But basically, he uses like techniques from deep work and slow productivity, as well as from Scott Young's excellent book, Ultra Learning, to begin a education process of learning The specific skills that his evidence-based planning said were important, not what he wants to be true, but like I need to know A, B, and C. These employers want exactly those skills. And how do I get there as effectively as possible? I'll jump back to regular speed here. Thanks to your techniques and tools, it took me about a year to get out of my difficult job situation and back into my old career. The side effect, I now earn more than twice as much as before. My next move won't be as bold as the ones I made in the past. I want to become a valuable asset for my company, get so good they can't ignore me, and then use my career capital to reduce my working hours so I can have Fridays off to spend in nature without Having to work there. So see what happened there is when he did lifestyle centric planning, he's like, I like nature, but by reorienting my life around my career is takes place in nature. A lot of other things got worse. And actually being in nature is not as great as you think when you're also like working. They realize like, oh, if I could take my programming expertise, make it up to date, make myself really valuable. ([Time 0:43:46](https://share.snipd.com/snip/ec65a6af-3b78-4437-9c82-67d417a7a0cb)) - **Plan Entire Lifestyle, Not One Change** - Lifestyle-centric planning considers all daily factors, not one appealing change, to improve overall well-being. - Small trade-offs in other life areas can negate gains from a single radical decision. Transcript: Cal Newport In the story so far, we're seeing a common issue when people think about the deep life. So a common issue that people have is that you fixate on a single change that you begin to believe will make everything better. It's simpler to think about a single change. We can sort of inhabit that change. The idea of doing something radical itself makes us feel excited in anticipation. And we can sort of take the imagined feeling of like the best parts of that change and sort of expand that in our mind, like my life is going to be better. So Savin did this with nature. Like I like nature. So I'm just going to focus like a laser on if I could just be in nature all the time, I would be really happy. The problem, as we often say, when we talk about lifestyle centric planning, is that your daily subjective mood is not the result of a single decision or change, but on all of the relevant Aspects of your life, call this your lifestyle, all the different things that are relevant during the day add up to give you your subjective experience of the day. So you might like the part where I'm giving a workshop and it's in nature and that's nice, but you have other parts of your life too, like the fact that you had to drive two hours to get there And that you're kind of stressed out about what's going on with your parents. And like the kids are – you're not around enough to help and it's creating tension with your wife. All of your houses may be nowhere near nature and you're actually spending most of your time in a car. All of these other aspects of your lifestyle matter too. And this is how you can make one radical change that you're excited about and yet end up less happy than you were before. Because by making this thing better, you accidentally maybe made other things worse. So lifestyle-centric planning says you got to construct your ideal vision for an entire lifestyle, all the parts of your life. How would I reorient my life so that all of the parts were something that resonated? And then how do I make progress towards that? You're going to have, you're going to enjoy your life more if you explicitly construct it so that all of the things that contribute to your enjoyment are being pushed in ways that are Better. ([Time 0:45:14](https://share.snipd.com/snip/44b2168d-5ed9-4145-aa7d-de6a4131dca4)) - **Don't Jump From Correlation To Causation** - An op-ed claimed Green Bank's lack of Wi‑Fi caused falling student scores, presenting an intuitive but thin causal story. - Cal emphasizes the need for time-series and controlled comparisons before attributing causation. Transcript: Cal Newport This article actually appeared recently in the Washington Post. It came to my attention, however, when Tyler Cowen wrote about it on his excellent blog, The Marginal Revolution. I'll pull it up here on the screen, the blog post, because the Washington Post is behind a paywall, but Marginal Revolution is not. All right, so here is Tyler's setup just summarizing this article. He says, Until recently, a nearby radio telescope meant the local school could not use Wi-Fi. And how did that go? All right. So this article that he's citing here is about Green Bank, West Virginia. Have you heard about this place, Jesse? No. It comes up a lot in technology circles because it's rural West Virginia and the world's largest steerable radio telescope is there. Oh, really? Yeah. And there's a town next to it. And radio interference could mess with the telescope. So there's no cell phones and there's no Wi-Fi. So it's like a town that's kind of like off the grid because of this telescope that's in the middle of the town. So what this Washington Post article that Tyler Cowen is citing, what it was about, it was actually an op-ed. It was written by someone who has a book he's working on about Green Bank. What it was arguing is the fact that there is no Wi-Fi in the town has been hurting the students in the school there. Green Bank has an elementary, middle school, combined school, about 200-something students. He's arguing without Wi-Fi, they can't use like the modern Chromebook, online curriculums, do online testing. And this is hurting the students. So let me read here. This is a Tyler Cowen citing The Washington Post. So what I'm about to read here is from The Washington Post article. While the rest of the country rushed to bring tech in the classrooms, Green Bank remained stuck in 1999. Without Wi-Fi, the school's 200 students couldn't use Chromebooks or digital textbooks or do research online. Teachers couldn't access individualized education programs online or use Google Docs for staff meetings. Even routine tasks such as state-mandated standardized testing became challenging with students rotating through a small hardware computer lab where they took the exams. Some of the teachers say this has been a problem. Here's a quote here. The ability to individualize learning with an iPad or a laptop, that's basically impossible. That was teacher Darla Huddle. Here's another teacher being quoted in the op-ed. Without the online component of our curriculum fully working, it's really detrimental to our instruction. That was Sarah Brown. So this is the issue here. They can't do this sort of modern ed tech stuff. And the argument of this article is like, that's held this school back. And how do we know this? Well, there isn't data on this in the article itself, like specific data, but here's what the author says. While these discussions dragged on, so discussions about like, is there some way we can get like low powered wifi or some other way to get wifi in the school? While these discussions dragged on, students fell further behind in math and reading with Greenbrink consistently posting the lowest test scores in the county. All right. This is an interesting argument here that without modern ed tech, students are falling behind. And that's, it's like a natural experiment. And look, this school has the lowest scores in the county. Now this is like a relevant time to be arguing this because there's so much discussion about phones in schools and technology in schools, and it's mainly negative. Like, technology in schools is often negative, but here's this op-ed in the WAPO that's being contrarian. And Tyler, who also has a real affinity for technology, is like, yeah, come on, let's be careful about it. Maybe some of this technology is really helping. So I want to look into this. You know, I'm not an expert data journalist, but I was like, this is interesting. And I want to look a little bit closer. So what did I find? Well, first, is it true that Green Bank consistently posts the lowest test scores in the county? This is turns out to be called Pocahontas County. It's kind of interesting. I'm like, West Virginia. Okay. So it's Pocahontas County, West Virginia. Yes, the Green Bank Elementary Middle School has the lowest scores in the county. But there's a problem here. It's a very small county. I mean, it's a lot of geographical area, but this is not Montgomery County here in Washington, D.C. ([Time 0:53:18](https://share.snipd.com/snip/9c596601-aab4-4d81-969b-d086cfa32b79)) - **County Data Undercut Simple Wi‑Fi Claim** - County-level test data show Pocahontas County didn't decline more than similar counties when Chromebooks spread in the 2010s. - This undermines a simple claim that lack of school Wi‑Fi caused the county's low scores. Transcript: Cal Newport It's hard to find time series data for the particular schools in Pocahontas County, but because Pocahontas County is so small, there's two middle schools. One of them is with Wi-Fi. One of them is not. There's three elementary schools, one with, one without, and the other two add up basically to the size of the one that didn't have Wi-Fi. It's such a small county. What we can do is compare Pocahontas County to other counties in West Virginia, because presumably if like half the students in your county didn't have Wi-Fi and that affected them, That should bring down the whole county around the time that Chromebooks, et cetera, became big in school. So there is data. I can get data on any AP, you know, these are state mandated test scores. I can get data on this county by county. So let's see what's going on here. So let me load up some charts. This is from education, education recovery scorecard, which aggregates a lot of this data. It's really interested in like what happened after COVID, but they have data that goes back to as far as 2009. So it's perfect for us. What I'm loading on the screen here for people who are watching instead of just listening is math performance grades three to eight. Because again, going past middle school, it's a shared high school. So like we don't learn much from that from 2009 to 2024. All right. So as we see here going from 2009, we see an increase. There's a bit of a gap in the data, but we see roughly actually math scores were getting better. Go Pocahontas County. Math scores were getting better from 2009 until about 2017. In 2017, oh, this is interesting, the score started going down. It went down, went down. Then we get the pandemic where we don't have data, but they were falling multiple years before the pandemic. And then we see after the pandemic, we begin to get a recovery, which we've seen nationwide as things fell so low during the pandemic that there's a recovery right after. So that's the chart we're seeing. If we look at reading scores, it's something kind of similar. Here, the peak is around 2015. Then it begins to go down after that as well. The timing here is compatible with the Wi-Fi hypothesis, right? That kind of makes sense. Chromebooks and all these other things that require internet begin spreading in the 2010s. It might make sense that where would you first start seeing the impact of not having this technology? Like midway through the 2010s as these things were gaining traction, that might be where you start seeing performance go down. So this data from within Pocahontas County itself is compatible, roughly speaking, is compatible with the Washington Post hypothesis of without Wi-Fi, things started to get worse. But if we want to do a controlled experiment, we have to compare this to similar counties. So what we need to do, because Pocahontas is half the non-Wi and we can't break that out. So let's find other counties in West Virginia that are similar in terms of size and demographics and socioeconomic status, like very similar other counties nearby in West Virginia Where there were no radio telescopes and everyone is allowed to use Wi-Fi. What we would expect to find if the Wi-Fi hypothesis was correct is that Pocahontas County should have a much more notable drop in performance starting in the mid-2010s than these other Counties that didn't have any Wi-Fi restrictions. We have that data. That's what's interesting. So let me scroll here a little bit. All right. So this is, we're going back to math performance here. We have three stack charts. So if you're, if you're listening, I'll explain to you what we see. Each of these charts has a downwards arrow and an upwards arrow, the downwards arrow, which is purple. I'll put this up here on the screen for Pocahontas County schools first. That is the decline in math scores during that period up to 2019. Let me get the exact dates here. 2019 to 2022, where we had the steepest sort of losses we saw in those curves before. And then the green arrow is the improvements that they've seen since the pandemic. So this chart is showing here for Pocahontas County Schools, it's just quantifying what we saw on that chart. In that downhill period from 2019 through 2022, there was a 0.6, negative 0.6 drop in their performance compared to, I think this is like state average. So yeah, they fell. And then we see the green, there was like this 0.36 increase. So don't worry about the numbers, but there was an increase. This is the recovery after the pandemic. All right. So I'm just quantifying what we saw on that chart. Here's what's interesting, though. They did the same thing for all of West Virginia counties. That's the next one. And for counties that had similar socioeconomic demographic and size. So similar population. So if you're wondering, this is Nicholas County, Hampshire County, Barber County, Tucker County, and Pendleton County. Here's what's interesting. The similar West Virginia districts that were the same in terms of demographics, but differed mainly and they didn't have the Wi-Fi restrictions. They saw a bigger drop from 2019 to 2022. And they saw a smaller recovery after the pandemic. So the similar counties that had Wi-Fi and never had that taken away did worse than the county in which like half the students are in this Wi-Fi free zone. ([Time 0:59:36](https://share.snipd.com/snip/1323070e-b0f4-46ef-be91-f9c34e033c18)) - **Demand Better Data Before Believing Narratives** - Be skeptical of intuitive tech narratives and demand proper data before concluding causation. - Probe for school-by-school time series or alternative explanations before accepting easy stories. Transcript: Cal Newport And it's a good reminder for myself or anyone else who talks about technology trends. When there's an answer that we like, it's often easy to jump to it, find any point of data that seems to imply that and expand it beyond what the data says. And I think that seems to be what was happening here. There's a nice subtle leap from the schools without Wi-Fi are worse to the schools without Wi-Fi are worse because they don't have Wi-Fi. It's an easy leap to make. But the picture gets much more murkier once you pull even a little bit on the data story. So anyways, I guess this is just a back to school note of we all have to be careful and take with grains of salt claims that are made that sound intuitive and there's a little bit of data to Support. Doesn't necessarily mean it's true. Like it's like when it comes to cell phones in schools, I have read exhaustively, not just the research, but the debates that researchers are having about the research and the complaints And then how the complaints are answered. There's an hour-long talk I give about the evolution of the research literature on harms from phones for kids. I feel like I know that data very well, and it makes me confident to say these are often a problem. The benefits aren't worth it. You shouldn't have phones before high school if you're a kid, right? ([Time 1:07:29](https://share.snipd.com/snip/95ca1b3e-d4d4-4470-904c-2af7e6df3f29))