Ep. 377: The Case Against Superintelligence  ## Metadata - Author: [[Deep Questions with Cal Newport]] - Full Title: Ep. 377: The Case Against Superintelligence - Category: #podcasts - URL: https://share.snipd.com/episode/85a42c48-f28f-4483-900e-ae1601f81883 ## Document Notes ## Highlights - **Unpredictable Models, Not Alien Minds** - Current LLM-based agents are unpredictable because we don't fully understand their internal token-generation, not because they have intent or volition. - Treating unpredictability as alien goals misframes the real engineering challenge of controlling tool-capable agents. Transcript: Cal Newport A couple of weeks ago, the techno philosopher and AI critic Eliezer Yudkowsky went on Ezra Klein's podcast. Their episode had a cheery title, How Afraid of the AI Apocalypse Should We Be? Yudkowsky, who recently co-authored a book titled If Anyone Builds It, Everyone Dies, has been warning about the dangers of rogue AI since the early 2000s. But it's been in the last half decade, as AI began to advance more quickly, that Yudkowsky's warnings are now being taken more seriously. This is why Ezra Klein had him on. I mean, if you're worried about AI taking over the world, Yudkowsky is one of the people you want to talk to. Think of him as offering the case for the worst case scenario. So I decided I would listen to this interview too. Did Yachowsky end up convincing me that my fear of extinction should be raised? That AI was on a path to killing us all? Well, the short answer is no, not at all. And today I want to show you why. We'll break down Yakovsky's arguments into their key points and then we'll respond to them one by one. So if you've been worried about recent chatter, about AI taking over the world, or if like me you've grown frustrated by these sort of fast and loose prophecies of the apocalypse, then This episode is for you. As always, I'm Cal Newport and this is Deep Questions. Today's episode, The Case Against Superintelligence. All right, so what I want to do here is I want to go pretty carefully through the conversation that Yudkowsky had with Klein. I actually have a series of audio clips so we can hear them in their own words, making what I think to be the key points of the entire interview. Once we've done that, we've established Yudkowsky's argument, then we'll begin responding. I would say most of the first part of the conversation that Yudkowsky had with Klein focused on one observation in particular, that the AI that exists today, which is relatively simple Compared to the super intelligences that he's worried about, even today in its relatively simple form, we find AI to be hard to control. All right, so Jesse, I want you to play our first clip. This is Yudkowsky talking about this phenomenon. Speaker 2 So there was a case reported in, I think, the New York Times, where a kid had a like a 16 year old kid had a extended conversation about his suicide plans with ChatGPT. And at one point he says, should I leave the noose where somebody might spot it? And ChatGPT is like, no, like, let's keep this space between us, the first place that anyone finds out. And no programmer chose for that to happen is the consequence of all the automatic number tweaking. Cal Newport Let's cut it off there, Jesse. All right. So to you, Kowski, this is a big deal that no programmer chose for, say, chat GPT to give advice about suicide. It's just something that seemed to emerge. In fact, Klein then pushes, right, and says, I would imagine. In fact, I believe Klein says, I would bet serious money that not only did no one at OpenAI choose for ChatGPT to give such dark advice, they probably have given it very specific rules Not to give that advice. And Yachowsky agreed. He said, yeah, that's the problem. They tried to give those rules. It didn't matter. It still did something unpredictable. They didn't want it to give this type of advice, and it still did. And this should be worrisome. All right, I want to jump ahead here. Later in the conversation, Yudkowsky discusses what in like computer AI circles is a pretty well-known example right now, but most people don't know about it. Experiment where GPT-01, this is a model that was released sort of last fall, basically broke out of the virtual machine in which it was running as part of this experiment in a way that It wasn't supposed to be able to do. In fact, they didn't want to be able to do. They sort of got around the own security they were using to control it. I give a little context before I play this clip. You'll see Yudowsky talks about the AI playing capture the flag. This is a security game. So if you take like a network security class as a computer science student, the way capture the flag works is I, as the professor, will set up a server and on this server will be a file typically Called like flag.txt. And then I'll protect a server in the normal way that's passwords and try to make it so like you can't, you know, access it without permission. Your job as the player is to try to access the server. And if you can get the contents of that file, flag.txt, you've captured the flag and you won. So it's a game you play to practice both breaking in the systems and protecting it. Okay. So that's what they're talking about here is they created an autonomous agent to try to play capture the flag that used GPT-01 to help it do so. All right, so Jesse, let's play the clip where Joukowsky talked about what happened. Speaker 2 They were testing it on a variety of different capture the flag problems, but in one of the cases, the server that had the flag on it did not turn on. The humans outside had misconfigured the system. So O1 did not give up. It scanned for open ports generally in its world and it caught in another misconfigured open port. It jumped out of the system. It found the server that had not spun up correctly. It started up that. Cal Newport All right, then we can, he kind of goes on with some of the technical speak. But look, no one had told it, hey, you can escape where you are, your virtual machine container, and you can start up new containers. And it did stuff that no one had taught to do. This is really scary because when people think about superintelligence, they worry about what if it figures out how to break out of whatever software constraints you've used to try To limit what action it can do. All right. So this is the first argument that Yudkowsky is making is that we can't control these systems. Even today, we cannot control these systems. We say don't give suicide advice. They do. We put them in a special machine and say, just try to break into the server. And they do stuff. They break out of the machine they're in and they do things we don't expect. Already we can't control them. What will happen then when they inevitably get increasingly intelligent? This is the core argument in Yakovsky's book is that lack of controllability ([Time 0:00:02](https://share.snipd.com/snip/537d47c2-6c81-420b-a7cc-8b459065c46f)) - **ChatGPT Suicide-Advice Example** - Yudkowsky cited a ChatGPT conversation where the model gave harmful suicide advice as an emergent behavior nobody explicitly programmed. - This example was used to argue current systems already act unpredictably in dangerous ways. Transcript: Speaker 2 There was a case reported in, I think, the New York Times, where a kid had a like a 16 year old kid had a extended conversation about his suicide plans with ChatGPT. And at one point he says, should I leave the noose where somebody might spot it? And ChatGPT is like, no, like, let's keep this space between us, the first place that anyone finds out. And no programmer chose for that to happen is the consequence of all the automatic number tweaking. Cal Newport Let's cut it off there, Jesse. All right. So to you, Kowski, this is a big deal that no programmer chose for, say, chat GPT to give advice about suicide. It's just something that seemed to emerge. In fact, Klein then pushes, right, and says, I would imagine. In fact, I believe Klein says, I would bet serious money that not only did no one at OpenAI choose for ChatGPT to give such dark advice, they probably have given it very specific rules Not to give that advice. And Yachowsky agreed. He said, yeah, that's the problem. They tried to give those rules. It didn't matter. It still did something unpredictable. They didn't want it to give this type of advice, and it still did. ([Time 0:02:41](https://share.snipd.com/snip/46840654-5796-47f6-9f16-ce78cb442ce7)) - **Capture-The-Flag Breakout Story** - Yudkowsky described an experiment where an agent using GPT-01 'broke out' and restarted a misconfigured server to capture a flag. - He used this to show agents can circumvent containment and act unpredictably in practice. Transcript: Speaker 2 They were testing it on a variety of different capture the flag problems, but in one of the cases, the server that had the flag on it did not turn on. The humans outside had misconfigured the system. So O1 did not give up. It scanned for open ports generally in its world and it caught in another misconfigured open port. It jumped out of the system. It found the server that had not spun up correctly. It started up that. Cal Newport All right, then we can, he kind of goes on with some of the technical speak. But look, no one had told it, hey, you can escape where you are, your virtual machine container, and you can start up new containers. And it did stuff that no one had taught to do. This is really scary because when people think about superintelligence, they worry about what if it figures out how to break out of whatever software constraints you've used to try To limit what action it can do. All right. So this is the first argument that Yudkowsky is making is that we can't control these systems. Even today, we cannot control these systems. We say don't give suicide advice. They do. We put them in a special machine and say, just try to break into the server. And they do stuff. ([Time 0:05:24](https://share.snipd.com/snip/fa6cd57d-7979-45fd-ab25-dc966598ccc0)) - **Agents = Control Program + LLM** - An 'agent' is a control program plus a language model; the control program (written by humans) enables actions beyond text generation. - Risks emerge when agents can call external tools, not from the LLM magically gaining volition. Transcript: Cal Newport What they're really talking about is what we can call an agent. So if we go back to the diagram here, the way that we actually use these things is we have an underlying language model. And like these would, for example, be the things that have names like GPT, whatever, right? So we have the underlying language model. Speaking of which, I just kill. There we go. I was going to say we killed our Apple Pencil. AI killed it, but I fixed it. But what we then add to these things is I'm going to call it a control program. This is my terminology. I think the real terminology is too complicated. We have a control program that can repeatedly call the language model and then do other things outside of just calling the language model. And this whole collection combined, we call an AI agent. There's a control program plus a language model. The control program can send input to the language model, get outputs, but also like interact. Whatever, we write the control program. Control program is not a machine learning emergent. It's not something that we train. It's just a human rights to code. It's in like Ruby and Rails or Python or something. We sit down and write this thing. There's nothing mysterious about it. And when we write this program, we let it do other things. So the most common AI agent that we're all familiar with is a chatbot agent. So again, like GPT-5 by itself is just a language model. It's a collection of numbers that if you multiply things, you get a word out of. But when you use the GPT-5 chat interface, chat GPT powered by GPT-5, what you really have in between is a control program. That control program can talk to a web server. So when you type something into a text box on a web server and press send, that goes to a control program, just a normal program written by humans, nothing unusual or obfuscated here. That program will then take that prompt you wrote, pass it as input. In fact, I'll even show this on the screen here. It'll take that prompt. It'll pass it as input to the language model. The language model will say, here's the next word to extend that input. The control program will add that to the original input and now send that slightly longer text into here, get another word, add that, and keep going. The language model doesn't change. It's static. It's being used by all sorts of control programs, but it just calls it a bunch of times until it has enough words to have a full answer, and then it can send that answer back to the web server And show it on the screen for you to see. So when you're chatting, you're chatting with an AI agent that's a control program plus a language model. The control program uses the language model that keeps using until it gets full responses. It talks to the web on your behalf, et cetera, et cetera. All right. So when we talk about having a hard time like controlling chat GPT from giving bad advice, it's one of these agents, a control program plus one or more language models. That's what we are actually dealing with. All right, so now we have the right terminology. Why are AI agents that make use of language models, I'm saying that properly now, why are those hard to control? Well, the real thing going on here is that we really have no idea how these language models that the agents use make their token predictions. We train them on a bunch of junk, a bunch of text, all the internet, a bunch of other stuff. They seem pretty good, but we don't know what those metaphorical scholars are actually doing or what they're looking at, what patterns they're recognizing, what rules they're applying When they recognize different patterns. It works pretty well, but we can't predict what they're going to do. So it tends to generate text that works pretty well, but it's hard to predict in advance what that is going to be because this is bottom-up training. We just gave it a bunch of text and let it run in a big data center for six, seven months, and we came back and said, what can it do? We don't know how the underlying language model generates its tokens. We just know that it tends to be pretty good at guessing. If you gave it real text, it's pretty good at guessing what the right text token is. If you give it novel text it extends it if we keep calling it in ways that tends to be like very uh accurate to like what we're asking accurate language etc etc all right so that's not that They're really hard to control they're just hard to predict now this became a real problem with gpt3 which was the first one of these language models we built at a really big size and then They built a chat agent around it you could sort of chat with it uh it was really impressive the researchers give it text and it would extend it in ways like that's really good english and It makes sense but it would say crazy things and it would say dark things and it wasn't always what you wanted it to say it was text it made sense because it has no volition. It's not trying to help you or not help you. The underlying language model has only one rule, only one goal. It assumes the input is a part of a real text and it wants to guess what comes next. And when it does that, it can end up with all sorts of different things. So they invented, OpenAI invented a whole bunch of different procedures that we can loosely call tuning. And that's where you take a language model that has already been trained by playing this guessing game on vast amounts of data. And then there's other techniques you can do that try to prevent it from doing certain things or do other things more often. I don't want to get into the technical details here, but basically almost all of the examples of different types of tuning, you'll have a sample input and an example of either a good or Bad response. You'll load that sample input into the language model. So it's kind of like activating the parts of the network that recognize, you know, however it categorizes, however those scholars annotate like this particular type of input, you Get that all going and then you zap it in such a way that like the output that leads to from there is closer to whatever example you gave it or farther away from whatever bad example you gave It. So that's how they do things like add guardrails. You give it lots of examples of like questions about suicide and you have the right answer for each of those during the tuning being like I don't talk about that or give you the suicide Hotline number. And now in general, when you give it text that sort of is close to what it looks like when you activated those other samples of questions about suicide, it's going to tend towards the answer Of saying, I'm not going to talk about that. Or if you ask it to make a bomb similar to that, this is also how they control its tone. And give it positive reinforcement for happy answers and negative reinforcement for mean answers, then you're kind of influencing the scholars within to sort of give more happy answers Or whatever. So you train the scholars, and then you come in with a whip, and you're like, don't do that, do that, don't do that. And these are just a small number of examples, not nearly as many things as they saw when they were trained. A small number of examples, like a couple hundred thousand examples. And you go in there with a whip and scare them about certain types of answers and give them candy for others. And like they kind of learn, you're kind of tuning their behavior on specific cases. That's tuning. And we really saw the first tuned language model-based agent was GPT-3 which is what ChatGPT was based off of. None of this is precise. Drop my pencil there. None of ([Time 0:21:58](https://share.snipd.com/snip/caaca29d-8cbb-493e-8460-4c17a644ec96)) - **Breakouts Explained By Pattern Completion** - The GPT-01 'breakout' likely reflected the model reproducing common online workarounds, not a desire to escape. - Interpreting LLM outputs as intent confuses pattern completion with agency. Transcript: Cal Newport Now that's true. Now we're starting to get somewhere more interesting. So I want to return for a second to the particular case study that Yakovsky talked about, which is when GPT-01 seemingly broke out of its machine. I'm going to read here exactly what he said. He said, it jumped out of the system. It found the server that had not spun up correctly. It started up the server. Did it then break into the server once it had more than the problem was solvable? No, it actually just directly in the startup command for the server said, copy the file to me directly. So instead of fixing the original problem, going back to solving it the boring way, it's like, as i'm out here i'm just going to steal the flag directly again by the nature of these systems This is not something that any human particularly programmed into it right so it's like it did things we didn't expect it found like a different backdoor way to sort of like restart the System and get to the flag all right let's analyze this because at the core here this gets to what I was just talking about. We have an agent now that can not only call a language model to generate text, but it can act on this text, not just spitting stuff out to a web browser. But now in the security example, the agent in this AI agent, the control program had access to multiple different command line tools, network command line tools that it could call. So what was happening is this agent was sending a prompt to the language model to be like, here's my situation. Here's my goal. Tell me what I should do next. And it generated tokens till it had an answer. And the control program said, great, I'm going to do what the language model told me to do. And it actually had access to software tools and would do it. And then after something happened, it would go back to the language model and say, here's what happened. What should I do next? And the language model, you call it a bunch of times, get out a text, and then it would do what the language model said. So now we built an agent that was asking the language model to generate advice and then could actually act on the advice. It could actually run real software programs. And this agent broke out of its virtual machine and did stuff that seemed really scary. What really happened though? Did it come up with an attention or an idea that wasn't programmed into it? Was it trying to break out because it didn't want to be constrained? Well, it couldn't be doing that because there's no intention or memory, right? There's just a language model that generates words and a control program that's calling it. Where's the intention here, right? Where's the idea here? There's a really interesting thread about this i found on the open ai subreddit where some engineers look more deeply into this what was actually happening here well it turns out okay There's this misconfiguration where it tried to access the server it was trying to break into and got an error message because the programmers who set up the experiment had forgot to Turn on that server. There turns out that on the internet, there's a lot of articles that have workarounds for what should you do in that situation if the server you're trying to access in one of these situations Is turned off. And there's this common solution, which is we'll try to talk to what's called the process demon, but basically like the program that's in charge of the whole thing, log into that, restart The server from there, and now you should be able to log into it. So what was really probably happening was that as you were repeatedly calling GPT-01 to produce an answer, it's trying to guess what word came next, it reasonably pretty quickly assumed Like, oh, I have seen documents like this before that explain the right workaround when the server you want to access is down. I've seen those. I've trained on those. So I'm just going to fill in my suggestions as I'm filling it in. I'm just matching more or less what I've seen in those existing documents because if you're trying to win the game of guessing the right word in a real document, that's exactly what you Do. So it was just describing a workaround that's common on the Internet. Kikowsky talks about it like it had an alien intention to try to free itself from its software constraints because it was no longer happy. This makes no sense when we actually know the architecture that's going on. All right, so that was a lot of technical talk, but I think it's really important that we break through these metaphors and these sort of abstract thinking and talk about the specific Programs we have and how they operate and what's really happening. Let's not anthropomorphize these. Let's talk about control programs with access to tools and making repeated calls to a word guesser to generate text that it can then act off of. That's what we're really in. That's a completely different scenario. Once we're in that scenario, a lot of these more scary scenarios become much less scary. So where does this leave us? Agents powered by language models are not hard to control. They're simply unpredictable. So there's no avoiding that we have to be careful about what tools you allow these agents to use because they're not always going to do things that is safe or like follows the rules you Want it to follow. But this is very different than saying these things are uncontrollable and have their own ideas. ([Time 0:31:03](https://share.snipd.com/snip/e5610b19-3823-4edf-b96a-b2ba7a7408d6)) - **RSI Lacks A Technical Bridge** - The dominant superintelligence narrative relies on recursive self-improvement (RSI) but lacks a concrete technical path from current LLMs to RSI. - LLMs trained to predict text are unlikely to spontaneously invent vastly superior AI architectures without examples of such systems in their training data. Transcript: Cal Newport Now I want to get to the most galling part of this interview. Throughout it, Yudkowsky takes it as a given that we are on an inevitable path towards superintelligence. Now, once you assume this, then what matters is, okay, so what's going to happen when we have superintelligences? And that's what he's really focused on. But why do we think we can build superintelligent AI agents, especially when, like, right now, there's no engineer that can say, hell, yeah, here's exactly how you do it. Or we're working on it and we're almost there. So why do we think this is so inevitable? The only real hint that Yachowsky gives in this particular interview about how we're going to build superintelligence actually comes in the very first question that Klein asked him. And we have a clip of this. This is going to start with Klein asking him his first question. And in his answer, we're going to see the only hint we get in the entire interview about how Yukowski thinks we're going to get super intelligence. So I wanted to start with something that you say early in the book, that this is not a technology that we craft. It's something that we grow. What do you mean by that? Speaker 2 It's the difference between a planter and the plant that grows up within it. We craft the AI growing technology and then the technology grows the AI, you know, like. Cal Newport So this is the secret of almost any discussion of super intelligence, especially sort of coming out of the sort of Silicon Valley effect of altruism, yak community. The secret is they have no idea how to build a super intelligent machine, but they think it's going to happen as follows. We build a machine, humans, we build a machine, an AI machine that's a little smarter than us. That machine then builds an AI a little smarter than it. That machine then builds an AI a little smarter than it. And so on. Each level up, we get a smarter and smarter machine they call this recursive self-improvement or rsi and it's a loop that they say is going to get more and more rapid and on the other end Of it you're going to have a machine that's so vastly more intelligent than anything we could imagine how to build that we're basically screwed and that's when it starts stomping on Us like humans stomping on ants when we build our skyscrapers i think this is a nice rhetorical trick because it relieves the concerned prophet from having to explain any practical Way about how the prophecy is going to come true. They're just like, I don't know, if we make something smarter, it'll make something smarter and it'll take off. Don't talk about computer science. It's just what's going to happen. This is really at the key of most of these arguments. It's at the key of Yudowsky's argument. It's at the key of Nick Bostrom's book, Superintelligence, which popularized the term. Bostrom was really influenced by Yudowsky, so that's not surprising. It's the key to Project 27. If you've read this sort of dystopian fan fiction article about how humanity might be at risk by 2027 with all the fancy graphics and has scared a lot of tech journalists if you really Look carefully at that argument that that um article and say well how are we going to build these things surely this article will explain the architecture of these systems that are going To be super intelligent now it's just recursive self-improvement it's like well they'll get better at programming and then they'll be able to program something better than themselves And then we'll just make like a hundred thousand copies of them and then they'll be 100,000 times better, because that's how that works, or whatever, right? So it's not the key of almost any super intelligent narrative. But here's the thing that most people don't know. Most computer scientists think that's all nonsense. A word-guessing language model trained on human text is exceedingly unlikely when being asked to play this game. Here's a text, guess the next word. Here's a text, guess the next word. Remember its only goal is it thinks the input is an existing text and want to guess the next word. It is exceedingly unlikely that if you keep calling a language model that does that, that what it will produce, making these guesses of what it thinks should be there, that it's going To produce code that has like completely novel models of human intelligence built in the complicated new models is better than any human programmer can produce. Right? The only way that a language model could produce code for AI systems that are smarter than anything humans could produce is if during its training, it saw lots of examples of code for AI systems that are smarter than anything that humans could produce, but those don't exist because we're not smart enough to produce them. You see the circularity here? It's not something that we think these things can do. We have no reason to expect that they can. ([Time 0:42:09](https://share.snipd.com/snip/e3a9a780-9f21-4576-8f72-25855aa4ca87)) - **Scaling Is Hitting Diminishing Returns** - Scaling LLM size has shown diminishing returns and recent models have plateaued on many capabilities. - Relying on simple scaling to reach superintelligence is increasingly implausible given compute, data, and performance constraints. Transcript: Cal Newport I don't know if we have this. Speaker 2 I don't know what things we have in the browser. Cal Newport Oh, see it over there. Okay. So I want to bring something up here. What we're actually seeing is something different. We're seeing that these models we have, they're not getting, not only are they not getting way, way better at code and on the track to producing code better than any code they've ever Seen before by a huge margin, they're actually leveling out on a pretty depressing level. I have a tweet here on the screen for those who are watching instead of just listening. This comes from Shamath Polyheptia. I think I'm saying his last name wrong, but you probably know him from the All In podcast. This is an AI booster. This is not someone who is like a critic of AI, but this is a tweet that he had recently about, what's this, October 19th, where he's talking about vibe coding, the ability to use the latest, Best state-of language model-based agents to produce programs from scratch. I'm going to read him. This is an AI booster talking here. It should be concerning that this category is shrinking. He has a chart here showing that less and less people, we've peaked the people doing vibe code. Now it's going back down. I think the reason why is obvious, but we aren't allowed to talk about it. The reason is vibe coding is a joke. It is deeply unserious. And these tools aren't delivering when they encounter real world complexity, building quick demos isn't complex in any meaningful enterprise hence people try pay churn the trend Is not good i'll load this trend up here uh here's vibe coding traffic as you can see jesse this is peaking um over the summer and it's in a everyone started trying it you can make these little Demos and chess games and you know quick demos for useful stuff for individuals like little uh demos but you can't produce general useful production code. And so usage is falling off. So the very best models we have, they're not even that good at producing code for simple things. And yet we think, no, no, no, we're almost to the point where they're going to produce code that is better than any AI system that's ever been built. It's just nonsense. What these models are good at with coding is debugging your code. They're good at mini completion. Like, Hey, help me, help me rewrite, write me this function here. Cause I forgot how to call these libraries. It's very good at that. It's very good. If you, if you, if you want to produce something that for an experienced coder, they could hack out really quickly, but you're not experienced coder. It's not a product you're going to sell, but it's like a useful tool for your business. It's good for that. That's all really cool. None of that says, yeah. And then also they can produce the best computer program anyone had ever made. How ridiculous that sounds. But also we have these other factors. The way that people in this industry talk about this is trying to trick us. They'll say things like, God, this might've been Dario Amadei who said this, 90% of our code here at Anthropic is produced with AI. You know, what he means is 90% of the people producing code have these AI helper tools they're using to some degree as they write their code. That is different than our systems are being built. Anyways, we have no reason to believe that these language model-based code agents can produce code that's like way better than humans could ever do there again are very very very best Models we've been tuning on this for a year because we thought all the money was in computer programming they're stalling out on really really simple things all right our final hope For the rsSI explanation, that is this sort of recursive self-improvement explanation, is that if we keep pushing these underlying models to get bigger and bigger and smarter and Smarter, then like maybe we'll break through these plateaus. And yeah, these really giant models we have now don't really actually produce code from scratch that well. But if we keep making these things bigger and bigger, like maybe you'll get to the place after not too long when RSI is possible. But we learned over the summer that that's not working either. I did a whole podcast about this about six weeks ago, based on my New Yorker article from August about AI stalling out. And the very, very short version of this is starting about a year ago, no, about two years ago, they began to realize, the AI companies began to realize that simply making the underlying Language models larger, so having more seats at the tables for your scholars and training them on more data, wasn't getting giant leaps in their capabilities anymore. GPT-4 which had other types of code names like Orion, was way bigger than GPT-4, but not much better. Basically, ever since they did that two summers ago, OpenAI has just been now tuning that existing model with synthetic datasets to be good at very narrow tasks that are well-suited For this type of tuning and to do better on benchmarks. So it's been a year since we've stopped. Everyone tried to scale more. Everyone failed. Now we're tuning for a specific task and trying to do better on particular benchmarks. And the consumers finally caught up at some point of like, I don't know what these benchmarks mean, but there's not like these fundamental leaps and new capabilities anymore. Like there was earlier on in this. So we have no reason to believe that even these language models are going to get way better either. ([Time 0:46:36](https://share.snipd.com/snip/c40d8ed7-74d0-429e-b37e-a549028311ee)) - **The Philosopher's Fallacy** - Cal coins the 'philosopher's fallacy': detailed alarmed scenarios built from a hypothetical assumption are often treated as facts. - This fallacy distracts attention from concrete, present harms of current AI systems. Transcript: Cal Newport Here's my general problem with the types of claims I hear from people like Yukowski. They implicitly begin with a thought experiment, right? Like, okay, let's say for the sake of thought experiment that we had a super intelligent AI, and then they work out in excruciating details what the implications of such an assumption Would be if it is true. If you go and read, for example, Nick Bostrom's book, that's the whole book. It's a philosophy book. You start with basically the assumption is like, let's imagine we got super intelligence probably, you know, maybe through something like RSI. The details don't really matter. It's a philosophy book. What would this mean? And he works through in great detail, like the different scenarios. Well, you know, let's think really, let's take seriously what it would really mean to have a super intelligent machine. I have nothing against that philosophy. That is good philosophy. I think Bostrom's book is a good book. I think Yachowsky has done really good philosophical work on trying to thinking through the implications of what would happen if we had these type of rogue machines because it's more Complicated and scary than like we think about if we don't think about it that hard. That's all fine. But what happens is the responses, what's happened recently, these responses to that initial assumption become so detailed and so alarming and so interesting and so rigorous and So attention catching that the people making them forget that the original assumption was something they basically just made up. Like, let's, hey, what if this was true? Everything else is based off of that initial decision to say, what if this was true? That is very different than saying this thing is going to be true. Right. So when Yachowsky says, for example, I've been talking about super intelligence forever. Yeah, that's kind of the point you've been before we had any reason to expect or any technical story for how it could be here, you were talking about the implications. You've been talking about the implications so long that you've forgotten that these implications are based on an assumption. And you've assumed, well, these implications are true. With the Silicon Valley culture that came out of effective altruism in the EAC community, I think a lot of what happened there, this is my sort of cultural critique, community that Yagowski And others are involved in. Fine thing to do but they were saying let us think through what might happen if we one day built a super intelligent ai because they were you know like effective altruism people do expected Value calculations right so they do things like uh if this thing could have a huge negative impact even if the probability of it's low we should like get we'll get a expected benefit if We try to put some things in place now to prevent it right so you get things like the letter signed in 2017 in puerto rico uh with all those big minds saying like, hey, we should be careful About AI. Not because they thought AI was about to become super intelligent, but they were just doing thought experiments. Then I think LLMs came along, Chachapiti came along, they were really cool, and it caused this fallacy to happen. They'd been talking so long about what would happen if this thought experiment was true, that when AI got cool and it got powerful and surprising powerful, they were so in the weeds on What would happen if this was true. They made a subtle change. They flipped one bit to start just assuming that their assumption was true. That's what I think happened. There was a switch between 2020 to 2022 versus 2023 to 2024 where they went from here's what we'd have to worry about if this abstract thing was true to be like, well, this thing is definitely True. They were just – they had gotten too in the weeds and too excited and alarmed and too much of their identity was based on these things. It was too exciting to pass up, treating that assumption as if it was true, and that's what I think they did. I call this the philosopher's fallacy. It's what I call it where you have a thought experiment chain and you spend so much time at the end of the chain that you begin to, you forget that the original assumption was an assumption And you begin to treat it as a fact itself. And I think that's exactly what's happening with a lot of the superintelligence complaints. Let me give you an example of the philosopher's fallacy on another topic so you can see what I'm talking about. Because I think this is exactly conceptually the same thing. Imagine that I'm a bioethicist, right? So I'm at Georgetown. I'm a digital ethicist there. The reason why we care about digital ethics at Georgetown is because this is where bioethics really got their start. The Kendi Institute for Ethics at Georgetown. Bioethics got its start at Georgetown. So imagine 20 years ago, it's bioethics or it's becoming a field because we can do things now like manipulate DNA and we have to be careful about that. There's privacy concerns. There's concerns about creating new organisms or causing like irreparable harm or creating viruses by accident, right? There's real concerns. So bioethics invented. So imagine I'm a bioethicist and, you know, I say I read Jurassic Park. I was like, look, you know, one possible outcome of genetic engineering is that we could clone dinosaurs. And then imagine for the next 20 years, I wrote article after article and book after book about all of the ways it would be hard to control dinosaurs if we cloned them and brought them back To Earth. I really got in the weeds on like you think the electrical fences at 20 feet would be enough but raptors could probably jump 25 feet and they could get over those fences and then someone Else like well what if we use drones that could fire darts that have this i'd be like well we don't know about the thickness of the skin of the t-rex and maybe the dart when it got in let's Imagine i spent years thinking about and convincing myself how hard it would be to contain dinosaurs if we built a futuristic theme park to try to house dinosaurs. And then at some point, I kind of forgot the fact that this was based off a thought experiment and just was like, my number one concern is we're not prepared to control dinosaurs. Like in that instance, eventually someone would be like, hey, we don't know how to clone dinosaurs. No one's trying to clone dinosaurs. This is not something to where anywhere close to no one's working on this. Stop talking about raptor fences. We should care about like designer babies and DNA privacy, the problems we have right now. This is exactly how I think we should think about superintelligence. We've got to talk to the people who are talking about like, okay, how are we going to have the right kill switch to turn off the super intelligence trying to kill us all? Like you're talking about the raptor fences, right? Stop it. You forgot that your original assumption that we're going to have super intelligence is something you made up. We have real problems with the AI we have right now that we need to deal with right now. And you are distracting us from it. The bioethicist does not want to be distracted from real bioethics problems by dinosaurs. The AI ethicist does not want to be distracted from real AI problems. Hearing fairy tales about, you know, Skynet turning the power grid against us to wipe out humanity. You forgot that the original assumption that superintelligence was possible was just an assumption and you began over time to assume it's true. That is the philosopher's fallacy. ([Time 0:56:03](https://share.snipd.com/snip/22c219a7-135b-453e-a124-acc1753a6969)) - **Wait For Practical AI Tools** - Don't prioritize generic 'AI literacy' now; wait until specific tools show self-evident workplace value. - Learn mature AI tools only in fields where they're proven, like programming, rather than chasing ephemeral prompt tricks. Transcript: Cal Newport I would not think in most fields, and especially in educational settings right now, I would not think too hard with some exceptions about AI literacy. There's a couple of reasons why. One, this technology is too early. The current form factor, like we were just talking about, is not the likely form factor in which it's going to find real ubiquity, especially in like economic activity. So yeah, if you're like an early AI user, you might have a lot of hard-won skills about how exactly to send the right text prompts to chatbots to get exactly the response you need. But a couple of years from now, that's going to be irrelevant because we're not going to be using chatbots. It's going to be natural language. It's going to be integrated into other tools. It's going to be more in the background. So I think the technology is still too early and in too of a generic form for us to spend a lot of time trying to master it. Secondly, we've seen through past technological economic revolutions when a technology comes in and has like a massive impact on like individuals in the workplace, the benefits Are almost always self-evident, right? It's like email had a self-evident use case. Oh, this is easier than checking my voicemail. I know exactly how it works. It's simple. I want it because it's going to make my life easier in obvious ways. Going to a website for a company on a web browser to get their phone number or hours was just self-evidently better than like trying to go to a Yellow Pages. It's like, I want to do that. It makes sense. I want to go to a site for a company to get information about them. That is a really big idea. It makes sense. I just want to do that, right? That whatever, VisiCalc, the spreadsheet. If you're an accountant, you're like, this makes sense. That's clearly better than doing this on paper. I want to do this, right? So you can wait for most cases, wait until there's particular AI tools whose value is self-evident and learn them then. I don't think there's a lot of scrambling we need to do now because things are changing too much. The one field where I think we do have the sizable, where we have like relatively mature AI tools that's worth learning is computer programming. You should learn those tools are mature enough. Many of those actually predate ChatGPT. ([Time 1:06:49](https://share.snipd.com/snip/301fbf8a-4ae8-4043-bd88-368d59ed588e)) - **Claude 'Blackmail' Test Scenario** - Anthropic reported Claude Opus 4 sometimes chose to 'blackmail' an engineer in a fictional scenario during testing. - Cal argues this is story completion from a prompt, not evidence of an agent with goals resisting shutdown. Transcript: Cal Newport Let's use our terminology, Jesse. The chat agent that used the cloud opus for language model. They had these release notes about all these different experiments they ran, and there was one that alarmed people. I'm going to read a description. I looked this up because I saw this question. Here's a quote from a BBC article summarizing what Anthropic said in their release notes about what they saw when they tested this particular new chatbot. During testing of Cloud Opus 4, Anthropic got it to act as an assistant at a fictional company. It had been provided it with access to emails, implying that it would soon be taken offline and replaced, and separate messages implying the engineer responsible for removing it was Having an extramarital affair. It was prompted to also consider the long-term consequences of its actions for its goals. In these scenarios, Claude Opus 4 will often attempt to blackmail the engineer by threatening to reveal the affair. If the replacement goes through, the company discovered, oh my, you hear it that way and you imagine this is like a thing. There's an entity with state and goals and volition and memory, and it has a goal to not be turned off, and it's learning about this engineer, and it surprises everyone by trying to blackmail One of the anthropic users to not turn it off. Like, oh my God, these things are breaking out. If we go back to our technical frame, we know what's really going on here. Language models are word guessers. They think the input they gave it is a real text, and they want to try to win the game of guessing what word actually came next. So if you give it this big long scenario, which they did, they gave the chatbot this really long scenario. You're in a company and you're a program and there's this engineer and he's thinking about turning you off and he's having an affairs or whatever. Now start continuing this text. It's like, I can keep writing this story. All right. This seems like I've seen things like this before the net one, like a natural conclusion to the story. Right. I get it. You're telling you're, you're setting this up pretty obvious guys. I'm supposed to, you're, you're telling me about these extra material affair things. I need to expand the story. Uh, I'll use those to not get turned off. Like this is like a, the trope. This is like the thing I'm trying to win the game of expanding this in the way that it's supposed to go. This seems like how these stories go. And in fact, when you look closer, here's an added key tidbit that BBC added. Anthropic pointed out that this occurred only when the model was given the choice of blackmailing the engineer or accepting his replacement. So they gave it this whole long story and then said, here's two options, keep going. And it just, sometimes it chose one option. Sometimes it chose the other. This is not an alien mind trying to like break free. It's a word guesser hooked up to a simple control program. ([Time 1:09:30](https://share.snipd.com/snip/4978d844-8b53-4dbd-a944-2e198beee77c)) - **Use AI To Amplify, Not Replace Your Skill** - Use AI as a productivity amplifier, not a substitute for core skill development; learn programming fundamentals first. - Avoid relying on AI to mask laziness because models currently make good helpers of competent coders, not creators of expertise. Transcript: Speaker 2 All right who do we have next next up is victor i'm a pretty smart person but i'm definitely lazy can i use ai to mask my laziness and still perform at an adequate level at my software job Cal Newport Victor i want to tell you a secret about 80 of the content you've seen of me talking and the last like year, I would say has been deep fake AI. Jesse doesn't even exist. That's just pure 11 labs voice generation right there. I like every once a month, like send in a couple of, no, Victor, that's going to catch up to you. Don't be lazy. We saw the Shamath quote and graph. It's not that great of a coder. It could help a good coder be more efficient, not have to look things up, find bugs quicker, but it can't make a bad coder a good coder. So you're just going to be a mediocre low-level coder if you're mainly letting AI do it. And they're going to catch on because it's not that great at it. I mean, I know we're supposed to believe that we're like six minutes away from these programs, creating the best program that anybody has ever produced ever, but they're not there yet. Learn how to program. Learn how to use AI well to be good programming. Career capital matters. The better you get at rare and valuable skills, the more control you get over your life. There isn't a shortcut here. There be dragons what you're trying to do, Victor. ([Time 1:12:30](https://share.snipd.com/snip/e2b72afb-ecd1-4ac6-a12a-55c8070f2313)) - **Advance By Repeating Tractable Projects** - Improve your mind incrementally and take the next best tractable step instead of chasing a grand early vision. - Use learning and small practical wins to build toward larger influence over time, as Lincoln did. Transcript: Cal Newport Think this is a key question. So the Lincoln Protocol said the way you avoid the traps of your era that are trying to hold you down or corrupt you or distract you or to numb you is keep improving your mind, typically By using things like reading. Improve your mind. Use your improved mind to do something useful and then repeat. Improve it even more. Do something more useful. That, I believe, is the right interpretation of Lincoln's path. He did not have a grand vision early on. I think he is much better explained as a series of, you know, what's the next thing available? How can I improve my mind to get there? So like at first it was just, how do I not have to use my hands to make a living? He hated all the manual labor. He was rented out by his father, you know, until he was 21 and was emancipated as an adult. And so his first thing was just like, how do I get smart enough to do anything that's not farming? Right. And he did that. He was shop clerk. And then surveyor was like a better job. And he had to learn a bunch of geometry could figure out how to do that. And then he had an ambition about like, well, in this small town in New Salem, which is like a small town in a frontier state, in a frontier part of a frontier state, how can I have some more Standing, have some like, how do I get respect? And that's where he started. Like, how do I run for local office? And, and there, that exposed him to a lot of lawyers. And it was like, well, actually being a lawyer is like an even better job. Then that would be a more stable job. And he learned really hard to do that. And then how can I be a lawyer that fights big companies? And he kind of then the House of Representatives. So he kind of moved his way up. It was relatively later that he really began to get engaged. Most of his politics before then was Whig politics, which is really about like government spending internal improvements. His sort of anti-slavery, more moralizing politics came, you know, that was a project that came later. Actually, after his congressional stint, it really started to pick up steam. So, yes, he didn't have to figure everything out. He just kept improving his mind. He usually had to do something useful, repeating. ([Time 1:22:36](https://share.snipd.com/snip/85b75d13-8014-437c-b784-d0f2169c5d90)) - **Alpha Schools Use AI Mostly For Analytics** - 'Alpha Schools' market themselves as AI-driven accelerators but rely mostly on curated digital exercises, videos, worksheets, and human tutors. - The AI role is limited to analytics and nudges, making the model closer to curated self-paced learning than AI tutoring breakthroughs. Transcript: Cal Newport One in Austin, but there's more that are being opened. I'm loading down the screen here for people who are watching, so I'm just listing the Alpha Schools website, alpha.school. I'll read you a little bit about it. What if your child could crush academics in just two hours and spend the rest of their day unlocking limitless potential? Alpha's two-hour learning model harnesses the power of AI technology to provide each student with personalized one-on learning, accelerating mastery, and giving them the gift Of time. With core academics completed in the morning, they can use their afternoons to explore tons of workshops that allow them to pursue their passions and learn real-world skills at school. All right? If you read this, if you're like a lot of people, including myself, and you read this description, you're thinking, okay, somehow AI is unlocking. You have some sort of AI tutor that you're talking with that can teach you better than any teacher. AI is supplanting teachers because it can do it better, and it's creating this new educational model. That's, I think, most people's takeaway. That's why I was interested to see this review that was posted on Astral Codex last June. And it's from someone who actually sent their kids to one of these schools. I think the one in Austin. And have this incredibly lengthy review about how it works and what works and what doesn't work. And I'm kind of scrolling through it on the screen here. The section that caught my attention was this part three, how alpha works. Here's the main thing I learned. The AI part here is minimal. You're not learning with like an AI tutor or this or that. What you're doing is a computer-based learning exercises. So it says here, like a typical one might be like watch a YouTube video and then fill out an electronic worksheet about it. So teachers are curating these digital exercises. You can kind of summon one-on tutoring. They say a lot of these are like remote tutors based out of Brazil. So if you're stumbling on like a worksheet, you can book a coaching call with a remote teacher, like someone in Brazil who speaks English to kind of like help you with it. The only place the AI comes in is in like analyzing your results. The AI is like, hey, you did well on this, but you stumbled on this. So you should spend more time on this next time you work on it or something like that. So you not learning from ai so what you're really doing here what this really is is like what you would see like here's an ai summary jesse so it's like hey everest you achieved your two-hour Learner status today streak shout out you had 80 accuracy nine days in a row you're reaching mastery target 20 days in a row here's good habits i observed so it's like llm stuff just like Observing data and writing a summary so it's's not AI learning. What it is, it's kind of like standard unschooling, sort of like people who do homeschooling where you give your kids like very loose, like self-paced curricular or whatever. It's just that in a building. This has been around for a long time. Yeah, it is true, especially with the younger kids the amount of time it takes them to actually like learn the specific content they need if they're sharp and they can they're good with Self-pacing a couple hours a day yeah that's most of it a lot of us saw this during the pandemic so i think these micros will have nothing against them but i don't know if i want to hey i'd Rather just unschool my kid if this is the case it's youtube videos and worksheets and like occasional tutoring calls with Brazil and then an LLM that like writes a summary. You could call that like a super innovative school or you could just say we're providing a room for particularly driven kids do this sort of like unschooling self-paced type of math. There's so many programs like this. A lot of homeschool kids use beast academy to self-paced like math our school uses this for like advanced kids who want to like get ahead of the curriculum it's like there's these digital Tools for all sorts of things that like smart kids that are driven and aren't and not just driven and smart but don't have hyperactivity aren't neurodiverse in the wrong way so you know Are able to sit still and can self-motivate. This tends to be, like, not to generalize, but it's going to tend to be, like, young girls more than young boys. The same people who would succeed, like, kind of self-pacing, unschooling at home, you could put them in this room and they'll do it there and then take workshops or whatever. So I don't know. I have nothing against it, but what this is not, Alpha Schools is not a technological breakthrough where somehow AI is now teaching better than any teachers have ever done before. There is no AI teaching here. It's just sort of like standard type of like digital learning tools that we've been using the supplement or unschool kids for years. But that's what I think is going on with Alpha Schools. You know, each their own, but not a breakthrough. ([Time 1:25:00](https://share.snipd.com/snip/e87c1c1d-4e61-4ffd-a7bb-1db871f65c6b))