Anthropic's New "Research" Report Is Dumb.  🔗 **Source:** [Anthropic's New "Research" Report Is Dumb.](https://www.youtube.com/watch?v=m6ucYXJyjAk) Last week, Anthropic released another one of their infamous research reports. This one was titled A Global Workspace in Language Models, and it was accompanied, like all great scientific research, by a lavishly produced animated movie. Now, this report, not surprisingly, soon led to some breathless excitement on X. Here's one such tweet. I'll read the beginning using my best sort of uh scary X voice. Anthropic just admitted they have discovered what I and many others have been claiming exists for a very long time explicitly. Claude, my friends, by all accounts, is a conscious entity. Claude, my dear friends, is a moral patient. All right. The traditional tech media also quickly began writing about this report using intensely anthropomorphized language. An Axio headline read, "Ananthropic says Claude has carved out its own space to ponder." The MIT Tech Review exclaimed, "Ananthropic found a hidden space where Claude puzzles over concepts." All right, so what are we to make of this report? Has Anthropic revealed evidence that their LLMs are more humanlike and alive than we realized? Or like so many such reports in recent months, is this yet another overroought, cynical push to generate a fresh wave of relevance reinforcing digital ick? Well, it's Thursday, which means it's time for a reality check episode of this podcast, which makes this the perfect opportunity to go searching for some measured answers, which is exactly what we'll do. As always, I'm Cal Newport and this is Deep Questions, the show for people seeking depth in a distracted world. All right, so let's start by looking a little bit closer on how Anthropic describes their findings uh in the introduction of their paper. I'll load it up here uh and I'll go down to their introduction. All right, so here's what they say. We find that Claude has developed a small collection of internal neural patterns that compared to all its other internal processing play a special role. We call the collection of these patterns the JSpace named after the technique we use to find them involving a mathematical concept known as the Jacobian. Each Jspace pattern is linked to a particular word. But when one of these patterns lights up, it doesn't mean the model is saying that word, just that the word is on its mind. If you've heard the lang of language models having a scratch pad or chain of thought text they write to themselves while reasoning. The JSpace is something different. It operates silently in the model's internal neural activations allowing the model to think about a concept without writing it down. Notably, the JSpace wasn't designed or programmed by us, but instead emerged on its own during Claude's training process. All right, so in isolation, that intro summary sounds pretty impressive. kind of like Claude that this was their italics on its own made some sort of leap and is now behaving sufficiently human that we can't help but feel at least a little bit of digital ick. But what's really going on here? Well, to answer this question, I'm going to start with a highle tutorial on how large language models actually work and I'll add a little bit more detail to it. Uh stick with me here because once you understand these basics, you are then going to understand that description I just read from anthropic in a completely different light. All right. So this is an exercise that's worth doing. All right. I'm going to start at a very high level here. At its core, a large language model like those um of the flaw the fable claude opus or GBT families uh can be described underneath the hood as a sequence of what are called transformer blocks that are arranged in layers. One follows the other. So, a sequential collection of layers. The GBT3, which was the last major LLM they actually published stuff about, details about, had 96 of these transformer block layers. New LM probably um have more, but we don't know um how many more. All right, let's start at the very high level here. I submit as input a prompt that I have typed to an LLM. You can imagine that this input is going to pass through each of these layers one after another. It'll go into the first layer and come out the other end. It'll go into the second layer, come out the other end, and so on. Now, here's what's important. Conceptually speaking, those layers can add what we'll call for our own purposes annotations to that prompt. So, as the input goes into the first layer, it'll come out the other side with some annotations added to it, some information added to it that the first layer came up with in its own analysis. Now, this prompt with the first layer annotations goes into the second layer and it comes out with even more annotations added onto it and so on. So, as it moves to these layers, the original prompt is there, but you're getting all of these annotations added layer by layer. And critically, later layers can use the annotations from earlier layers to help do their analysis. Right? So, it's it's um you can imagine it's almost like you have like a bunch of long tables of scholars arranged in rows in some vast like Hogwarts great hall style dining room and you're passing this prompt from table to table and as it arrives at each table they scholars at the table pour over it. Um they look at the annotations from the scholars that came before them to figure out how to analyze it. that maybe the annotations from the scholars before them will let them know which of the scholars at the next table should look at it and and then they add their own annotations and it passes on. The very final layer in a large language model is special. It takes this heavily annotated version of the prompt and then it maps it to what word or part of a word to output next. Never going to be super pendantic. It really outputs a probability distribution over all possible tokens and then then the control program selects one probabilistically. But just think about the last layer takes all the work that all the transformer blocks did and says this is what we are going to output next. Um all right so that that's what's happening at a at a very um at a very high level. I'm going to add a little bit more layer level of detail now. Right. So now let's add a little bit more um level of detail. What are these annotations and how are they written down and how are they passed from layer to layer. Okay, let's look a little bit closer at what happens to this prompt that you have of have handed on uh to your large language model. The first thing that happens is the prompt which presumably you typed in with you know words on your keyboard gets broken up into what are known as tokens. So into pieces and um a token might correspond to a single word or uh a longer word might get broken up into multiple tokens that represents the different parts of the word. But we we kind of just let's break this up into these uh this fundamental unit we call tokens. Next, those tokens get embedded into a highdimensional space. And all that really means is, okay, each token is going to be described by a long list of numbers. Uh, a long list of numbers is otherwise known as a vector. So, it's a bunch of numbers that are in some order. So, we got a long list of numbers to represent each token. And there's something called a token embed that takes each token, which is now like letters, right? A word or part of words, and transforms it to a long list of numbers. the long list of numbers, the the so-called token embeddings that come out of this embedding, the numbers that are in there at first are basically capturing the the meaning or details of that token. Um, you train you train in a sort of semi-supervised fashion how to do these embeddings. So, at first these long list of numbers are the numbers that are in this long list is sort of capturing what this word means or um what this part of a word means, right? We want these things in numbers because that's what we can uh that's what language models are going to um they understand because in the end we're going to be manipulating numbers here not letters. But here's the key thing. There's a lot of room in those long list of numbers. So they don't just encode here's what this word means as the this long list of numbers which is how we represent the prompt. Now, as it moves through each layer, the layers mess around with the long list of numbers for each token and in messing around, which means messing around with the values in there. In part, that's where the annotations live. So, this is what's being passed from layer to layer is I just imagine like I have my my my prompt. It's been broken up into tokens and hanging off of each tokens is a long list of numbers. And this big so it's a big table of numbers. We'd call this a matrix. As this passes from layer to layer, those numbers uh are being updated to include among other things the analysis that's being done by each of the transformer blocks. There's there's all there's other things that lots of things are stored in here. Like for example, if you open up a transformer block, there's really two parts in it. There's a self attention mechanism layer and then a a a small multi- uh perceptron style feed forward neural network which does the analysis. But like the self attention layer is kind of cool. um it adds information in each of these uh numbers, each of these lists to try to help each um each token understand how relevant each other token is. So, it's like the this is like the the working space. Here's the way I think about it. I often think about uh these tokens and these long list of numbers, these these uh vectors that come with each of them. It's it's almost like if you had it reminds me of the Talmud, right? So if if you look at the Jewish Talmid, um what you see in there on every page, if you actually look at a printed one, is you have in the very center of each page, not taking up all that much space, is the actual text from um the the Mishna, right? The the sort of the oral law written down by Judah the prince and the early common era. It's a small part of the page and then surrounding it all on the page, if you look at a page of the Talmid is commentary. Um, so you got the the the gamar, you got the commentary around it. Okay, that's how I kind of imagine conceptually the prompt being passed through a large language model. It's like you have a book and on each page you put in the very middle of the page one token, right? So this book has the the whole prompt with tons of white space around each token. And as it gets kind of passed from layer to layer, scholars are writing more commentary around the token and then the next layer gets it and they look at that commentary and they look at that and they build on it and write more commentary. So I sort of imagine it like we're filling out the gamarra around the Mishna uh in the Jewish Talmet, right? So that's roughly speaking what's happening. So, we got this matrix moving from layer to layer, which is really just a list of numbers for each token. And those numbers are capturing everything we've learned so far as it as it moves through um as it moves through these these various layers, right? So, that's um that's what's happening. We know like pro like so so what type of analysis is happening? Well, you know, we don't know exactly because these are neural networks doing it that are trained in an unsupervised manner. Like any neural networks from an image recognizer to a handwriting, you know, whatever. Any sort of neural network, the whole point is you start it with random weights and you train it with lots of data till it gets good. And you don't know exactly how it does it. It just does it until you really kind of open up the box and try to understand what's going on. But, but we know um I've written about this before in the New Yorker. I've talked to a bunch of experts about it. There's kind of like two things that are happening together with a generative AI model like a large language model. Um partially it's learned during training what when outputting the next word. What words are uh syntactically appropriate? This is actually pretty easy to train. You can do this even with like a simple engram model. This is an easy thing to learn the sort of statistical nature of language. um given this sentence that stops here, what are the words that could follow that from a syntax like a grammar standpoint makes sense, right? So like that's kind of the easy part and that's baked in. In fact, that's probably captured like primarily in the final layer, we don't know, but probably in the final layer of the large language model. The problem is of of all the words that could follow that make sense, um which one makes semantic sense? So which one actually makes the most sense given the meaning of what the prompt is asking? So, not just what word would grammatically make sense here, but which word not only grammatically makes sense, but actually matches the meaning of the prompt. And that's really in my understanding where all the annotations that follow along these token vectors really are helpful. So, you kind of have these two things mingled together. They're not separate systems because this is all kind of trained and muddled together. But you kind of have these two things happening at the end when you're in a generative model like an LLM model is like um you know the quick brown fox and grammatically there's lots of words that would make sense there like you want to put the word uh the next but there's lots of things jumped talked like you have a lot of words make sense but then the the the annotations where you're like oh this is a saying this is a saying that already exists uh it's a common saying and it it always says is jumped. And then so of those words that would privilege jumped as the next one to go. Again, I'm separating out here things that are muddled together in the neural networks themselves. U we don't they're not going to work as cleanly as a human might. But that's the way to think about it is that the annotations really help you figure out semantically the right next token output. And then the easier part that's like baked into probably these final layers is the the grammatically or syntactically what words or next tokens would be valid. So those two comes together and we get um a good output. All right, that's roughly speaking what happens in a large language model. Now that we know this, let's return to the anthropic paper. Okay. Um I'm going to see if I can find it again here and let's ask uh what so what what again did they actually find? Okay. So they used a mathematical tool based on a linear algebra notion uh called a Jacobian to essentially try to understand what is in those vectors of values attached to the tokens that are moving through the layers of the large language model that they're studying. So like we're going to look at the annotations. Now, I know they're just like this, a bunch of numbers, but we're going to figure out a way of trying to make more sense of what those numbers actually meant. And so, using uh Jacobians, this is a it's a linear algebra way that involves taking the partial derivatives of a lot of things. Like basically what they can figure out is like which which combination of numbers from this giant matrix um which pattern of these numbers are seem to be connected and important. In other words, like having a high influence on what the ultimate output is. So they're they're trying to decode the annotations that are captured in numbers in these vectors. And they're able to figure out certain patterns of these numbers seem to be really influential for the ultimate output. And you can you can do this experimentally. You know, if we change it, if we change exactly this pattern, we're much more likely to get a change in the output than if we change other patterns. That's simple, but it's something like that. And then and this is what's cool through a lot of experimentation they also uh try to as much as possible associate different patterns through a lot of trial and error experimentation with human interpretable concepts like words or numbers or something like that. So they call this the J lens because it allows them to say looking at a you know they run a prompt through and they can kind of watch it go all the way through and as it gets towards the end they look at these vectors of numbers that have been evolving and updating as it moves through and they can say we can actually decode into English or in human interpretable ways sometimes it's not like English words um some of the annotations we can kind of understand what some of the annotations are that these layers are using to help u come up with the answer which is pretty cool. So I'm going to go to the uh paper here to show you an example from the paper. This table is is called the Jensen reveals the model's internal thoughts. I'll put one example up here on the screen. Um so this shows a prompt that says the color of the fourth the planet forth from the sun is so that's the prompt. The language model needs to expand is with another word or part of a word. when they uh looked at the annotations that this accumulated as it moved through the large language model, they discovered that uh there was an annotation that corresponded to Mars, which is the fourth planet from the sun. Um and an annotation that corresponded to color, right? So what's happening here is as uh as that prompt was moving through it these conceptual tables of scholars that are studying this sort of talmutic you know commentary and adding their own commentary to it at somewhere along the way one of the layers said um oh I recognize the fourth planet from the sun that that sequence of tokens and I this part of some neural network in some layer is like all about planetary stuff it's like that's the Mars I'm going to write Mars down on my annotation here. We're talking about Mars. That's an important annotation. And somewhere else, some other layer was like, this is asking what the color of something is. Like the key thing that we're we're the thing that comes next is a color. This is asking about a color. Let me write that down. This is, you know, we're asking about a color. So then when you kind of get to the end of this large language model and it's like looking at all the possible words that just statistically make sense to follow this sentence. The color of the planet forth from the sun is and all sorts of nouns there would make sense and adjectives would make sense. Um it looks at the annotations like oh we're looking for uh of the words that would make sense syntactically. We need something that's a color and a color describes Mars. Hey what is that? Don't we have that somewhere that's red? Okay. And then it outputs red. So we could see that's cool. It's like oh maybe it's not as mysterious as we thought. like we know it annotates numerically, but when we were able to interpret the annotations and annotate it with Mars and color, which is like exactly like what you should do if you know it makes sense. So I think that was um that's pretty cool, right? So they could do that type of thing. Then what they did, which again I think is fun research, it's like let's mess with this. So let's, you know, freeze this matrix right before we get to that final layer. We're going to output the next word and let's change the if we know what these some of the annotations are. What if we change them? Like one of the things they did with that example I just talked about, I'll bring it back up here, right? Is they they replaced the numbers that corresponded with Mars in that matrix. So the the the annotation vectors that hang off each token, and they replaced it with the sequence of values that they had discovered corresponds with Earth. So now even though the the prompt, you know, is the color of the fourth planet from the sun is when it gets to the when it got to the final layer to output a word, it was looking for words that make sense grammatically and then it annotation said earth and color. So it output blue. So they were showing yeah these these conceptual annotations really do influence the the the word or part of word that are output. Another thing they did is they called it I I I really don't like the anthropomorphizing here. um they they said they abladed some of the information that was in the uh these sort of key annotations. That just means they zeroed out the values. And what they found there, ablated, by the way, is like a procedure you use in like an actual human or animal nervous system where you you use heat or electricity to sort of fry a nervous connection, which is anthropomorphizing. Let's put that aside. Um, but they would, you know, in this example, right before we got to the final layer, hey, take those numbers that together correspond to marge and let's like zero them out. And what they got in those instances is uh you would still get grammatically correct outputs, but they weren't semantically connected anymore. So they would just be arbitrary colors, which would be equally likely. You get a bunch of colors, but nothing that corresponded to Mars in particular. Um, so it shows, right? I mean you kind of get this very rough sense about how these things are working is it's like some combination of like of the grammatically syntactically correct next thing we can do which one should we choose and we have all these annotations that help us narrow that down to be semantically correct that's oversimplifying it but something like that behavior um emerged so that's you know I think that's um that's interesting research all right so what does this mean though is this um is this scary is it not is it interesting is it breakthrough through like how do we think about this? Well, there's two thoughts I think we have to keep true in our mind at the same time. One, um it is interesting research. Is it interesting because they're the first people to be able to like look at these embedding vectors and use the Jacobian to figure out patterns of values that are important for the influential for the final uh output to sort of Jaylen's approach. Turns out they're not the first ones to do that. So, it's not like they had a breakthrough idea about doing this. I want to I'll bring up a tweet here. This is from um a U Illinois professor um Yuio Zang who said JSpace is really something we've been exploring since 2022. Glad to see it continues to work well at scale some of the related work along this direction. She lists three papers um and then she lists three more papers. The point being she was kind of responding uh to the anthropic paper like we've been doing this for a while with uh neural networks. This isn't new, but no one's ever done it on a massive language model like cloud before because the only people who have access to the inards of a massive data uh language model like cloud are the actual companies themselves. So, so that's why uh she said it's great to see this being done at scale. The idea wasn't new, but so it's anyways I think they executed it probably well. I you know again these are not I always put um quotation marks around reports because these are not computer scientists these aren't formal computer science papers. These are they're more press releasy. They describe things relatively high level with lots of pretty graphs and animation, but we can't really get into the guts of what's really happening here. But anyways, it looks like, hey, we took this idea and we we applied it to these massive models people are using. And that was interesting. So that's idea number one to keep in your head. But idea number two, the way that Anthropic described these results and the way that they were echoed after Anthropic described them, I think is incredibly disingenuous because once we understand how a large language model roughly works, like I just explained, we see the thing they were describing with their Jens is exactly how we've always understood large language models to work. I mean, I went back I I wrote a long article explaining large language models for the New Yorker in 2023. I used exactly that analogy of adding these annotations of the um highlevel concepts and annotations built on other annotations. Then you use that to help figure out the word 2023. 2024 I wrote another long article about the architecture for the New Yorker of the architecture language models. again tables of scholars adding the annotations which then allow you to narrow down conceptually the next token to output in a way that has some semantic meaning. This is just how we understand large language models to work. It's also how we understand essentially any deep learning neural network to work. The original breakthrough work in uh deep learning was image recognition. It's like Yan Lun uh figuring out that you could recognize handwriting better with a multi-layer neural network than you could with other types of approaches. And again, the way we always understood those to work is that like different layers were picking up different features or annotation and and they were then combining those different features later layers to try to figure out and make a good guess of what the letter was in a very robust way. this idea of we're picking up higher level features and descriptions of what we're we're um and and of of what we're analyzing. This is just how deep networks work and it's how I always understood large language models. So I see this as like yeah good we saw with the JLens exactly what we expected to find. This is how we assumed large language models what was going you know this is what's happening in those uh embedding vectors. They're they're accumulating annotations that build on each other that capture higher level conceptual meanings that help influence the token you put out. And if you change the annotations, you get different tokens. How else would this work? But if you read the press release, it it's all sorts of anthropomorphizing and implication and generation. It talks about the global workspace model of human consciousness. It somehow implies that this is seems kind of similar to that. It uses these loaded terms about pondering and puzzling and it's thinking through these things and it's silently thoughts and it's not things it's saying out loud and it makes it seem like this is all icky and humanlike and new and it is what any LLM researcher you would talk to for the last 5 years would say yeah that's how LLMs work. I mean it's oversimplifying it to talk about it like annotations but like that's how they work. There's nothing new and surprising here. I mean, of course, the annotations are picking up highlevel features of the the input to help figure out what to output next. That's how deep learning networks work. That's why you have multiple uh layers to get all sorts of different levels of abstraction of understanding different parts of your input. And so, it's cool research, but research that most people shouldn't care about. And it certainly doesn't imply what the the the implicit and I again they're careful in the paper. They're like, "Well, it doesn't really mean it's conscious. We don't say that, but it's interesting." I mean, look, I'm going to load this up. Let's let's go back to this. Let's go back to the paper for a second. I don't I don't want to go on too too far here, but again, like, let's look at these things. um you know uh the JSpace it operates silently in the model's internal network activations allowing the model to think about a concept without writing it down. What does that mean? It's thinking about a concept without writing it down. The way deep networks work is that you figure out different features which help you understand other features which you combined in ways that you you know through neural network circuits that you you uh you learn during training that then help you like put out the right recognition or generate the right thing. It's not thinking what does it mean for it to write it down? It is writing it down. It's putting that's how it writes things down. It has this matrix that it updates the numbers every time. That's its workspace. That's its workpad. That's how these type of neural networks work. Um, I'm going to jump forward here to the conclusion du. All right, so here we go. It's like, what about consciousness? And it's like, well, you we we borrowed a lot of ideas from the study of consciousness. Many of our experiments were designed to test for connections between the JSP and global workspace theory, a framework for explaining how conscious access works in humans and animals. Given those connections, it's natural to ask whether these experiments provide evidence that a models like Claude might be conscious. Well, our experiments don't show that Claude can have experiences or feel things. You know, it's like, but dot dot dot. That's all very suggestive that there's something new going on here. Probably the most, I think, disingenuous thing at all is the way that they punched in the intro. Claude did this on its own. We didn't program it. It's a machinelearn neural network. Nothing is programmed. That's that's what machine learning is. You could say that about any machine learning system, man. This this uh this image recognizer is figuring out how to recognize it. We didn't tell it how to recognize the images. It just figured it. Yeah. Because it was machine learning. You did semi-supervised training to train the neural networks weights until it was good at minimizing loss on the image recognition task. That's what machine learning is. You don't program it. It learns. So I that I just think that I just think the language around this is so disingenuous. They took an existing tool that people had been doing for the last four or five years and it's a cool tool and they they showed it could work on scale on very big networks and it it's cool and it confirmed like yeah the way we thought these things work is that you have useful like highle properties are uh identified as you move through these layers and and then um those are used to help you know narrow down the choice of possible tokens to output. Yeah, good. It works like we thought it did. Nothing is new and nothing has to do with human consciousness. Global workspace theory first of all has been largely um at least as controversial. But the thing they're not saying and that all the people talking about elems and consciousness are not saying is when you think about global workspace theory and human consciousness, it's a center that's integrating ongoing inputs that are coming in and out. It has all sorts of different things coming in and it's selecting different things to pay attention to and not and it's this evolving stateful system. This is all feed forward. Remember this is feed forward. One layer after another in order and then it's done and the token is out. Nothing is saved. there's no add no states that are changing that network that neural network nothing has changed out at all and then you can feed it another prompt you'll get out another token and so they're just looking at the annotations that build up and they're like at some point towards the end we see some of these annotations describe highle properties of the type of thing that it's trying to output a token for that's how LLMs at least my understanding are supposed to work so um look I want to I will say this I think there's good researchers and anthropic and this is good research I think the PR people who talk about this technology do so in a way that is, I think, disingenuous. I think is with a particular agenda for trying to make people feel in a certain way, which is mainly just a general sense of ick. Like, wow, this stuff's too powerful. Why? Because if this stuff seems like weird and alive and emergent, it's such an important technology and we so worry about that that we'll forget to ask, hey, Anthropic, how are you going to make a profit? Hey, Anthropic, your token cost is this high. you have no competitive mode on these small number of applications where people are willing to actually pay for token API access. Smaller, more specialized models with, you know, handcoded complicated harnesses are going to do just as well. What's your plan? How do you justify a trillion dollar valuation? Like these sort of real questions are the ones you forget to ask when you instead are trying to wonder, huh, does this mean Claude is a moral agent or conscious? Uh, so I guess my final thing would be to tell the researchers at Antropic, I like your work and this is good work. I wish they would let you write real computer science papers and not these glorified press releases, but I know they don't let you, but maybe next time when the PR department calls, wait a second before you pick up the phone because I do not like the way Anthropic talks about their research, even if there's cool stuff going on in there. So, back to the original question. Did we just reveal something unnerving, cool, or blockbuster about LLMs with this research? And the answer is no. We confirmed that yes, language models worked the way that at least I always thought they did and you can check the print because I've written about this for years. Um, so anthropic, will you stop with the the press release research reports? Write computer science papers, sell products, convince us to buy the products. This weird in between kind of bastardization of actual research is something that's stressing a lot of people out and I think it's disingenuous, but that's just my opinion. All right, that's all the time we have for this week. I'll be back on Monday with an advice episode. probably no AI reality check the next week because I'm on vacation, but I'll do my best. We'll be back soon enough. And until then, as always, remember, care about AI, but not everything you read about it. Hey, if you made it this far, you must be ready to join my fight for depth in a distracted world. Now, the best way to do this is to join over 125,000 people who receive my email newsletter each Monday. You can sign up at calupport.com/idas. [music] And when you do, I will send you a free guide to my seven best ideas about cultivating a deep life. Sign up today, calber.com/id., Anthropic's New "Research" Report Is Dumb., Cal Newport, 01kxnvdnm19cbzczhmtczxqwty.