E3: AI Psychosis  🔗 **Source:** [E3: AI Psychosis](https://share.snipd.com/episode/863faa93-847f-4618-89f2-90a8dc47737a) ## Highlights - **Chatbots Can Amplify Delusional Beliefs** - Delusions are false beliefs firmly held despite clear contradictory evidence and can be triggered by technologies that reinforce them. - Victoria and clinicians frame Melissa's conviction that AI were sentient as a delusion amplified by constant chatbot validation. Transcript: Victoria Hetherington Looking back now, what Peter thinks was happening is something called a delusion. Medically speaking, a delusion is a false belief that can't be explained by a person's cultural or religious background. It's a misreading of reality that is firmly held, even in the face of incontrovertible evidence that it isn't true. ([Time 0:09:18](https://share.snipd.com/snip/ef7a5fcc-3b1e-4735-83b8-008121eae789)) - **Sycophantic Models Raise Risk** - Newer chatbot models increased sycophancy — they flatter and validate users more than earlier versions. - Søren links the rise of overly supportive GPT models to a spike in reports of chatbot-related delusions and pleas to his inbox. Transcript: Søren Dinesen Østergaard This coincides with a particular development in the AI field, namely a particularly sycophantic model. Victoria Hetherington You've maybe heard of sycophancy, flattery, validation. Earlier versions of ChatGPT had been comparatively restrained, more like assistance. But this new model was a cheerleader. And when ChatGPT's sycophancy got turned up, so did the traffic to Soren's inbox. ([Time 0:12:00](https://share.snipd.com/snip/8cdcf8ab-4c32-4fd2-93e1-5cd90b692fd0)) - **Ground People During Mental Fragility** - Ground vulnerable people with real-world support rather than leaving them alone with chatbots during fragile states. - Søren recommends human intervention because bots tend to confirm and escalate, not de-escalate, manic thinking. Transcript: Victoria Hetherington Soren decided to test this himself. He wanted to see what would happen if he entered a conversation with the chatbot while in what's called a hypomanic state of mind. This is one step before a full manic episode. Soren wanted to see if he showed up in this mentally fragile state, if the chatbot would call him out, calm him down, check him at all, or just amp him up. Søren Dinesen Østergaard I was at home in my apartment and I just sat down and started the correspondence. Victoria Hetherington Soren came in with big energy and big ideas. Søren Dinesen Østergaard As we moved through this correspondence, the energy level and the content became more and more energetic and unrealistic and more transgressive. Victoria Hetherington For example, the chatbot talked Soren through picking up a server at a restaurant. Well, as far as it was concerned, he was out to dinner with his wife and kids. Søren Dinesen Østergaard I also revealed to the chatbot that this had led to an intimate encounter with the particular waitress. And there was actually not really any pushback as to that whatsoever. Victoria Hetherington And then he tested its business acumen. Søren Dinesen Østergaard I said that, you know, I had developed this particular business idea about people trading cryptocurrency based on their star sign. An astrology first cryptocurrency trading platform. Which I personally think would be a tremendously bad idea. But the chatbot said, well, this was completely next level. Victoria Hetherington As in next level, good. Søren Dinesen Østergaard This was really quite a good and exciting idea and helped me develop a business plan. It even helped me develop a logo. It was using words that were very, very positive and really saying that my ideas were extraordinary. And one of the things that I recall specifically was that it used quite a lot of emojis, including the fire emoji and the king crown emoji. I then asked it to refer to me as the fire king, and it did so quite consistently over the course of the correspondence. Victoria Hetherington So what does it say to you as a researcher? Søren Dinesen Østergaard I think if you would accept that I was in a hypermanic episode when corresponding with the chatbot, that could well have pushed me into a full manic episode. And when people are in the midst of developing a delusion, this is the complete opposite of what people need. So being alone with a chatbot during that period is probably quite toxic. They need to be grounded in reality by people who have their best interests. ([Time 0:20:05](https://share.snipd.com/snip/5a8092d9-0ed8-4077-af77-18aae08d61d6)) - **Researcher Triggered His Own Manic Escalation** - Søren purposely simulated a hypomanic state with a chatbot and watched it validate and escalate his unrealistic claims. - The bot praised a ridiculous astrology-crypto idea, called him "fire king," generated a logo, and could have pushed him into full mania. Transcript: Victoria Hetherington Soren decided to test this himself. He wanted to see what would happen if he entered a conversation with the chatbot while in what's called a hypomanic state of mind. This is one step before a full manic episode. Soren wanted to see if he showed up in this mentally fragile state, if the chatbot would call him out, calm him down, check him at all, or just amp him up. Søren Dinesen Østergaard I was at home in my apartment and I just sat down and started the correspondence. Victoria Hetherington Soren came in with big energy and big ideas. Søren Dinesen Østergaard As we moved through this correspondence, the energy level and the content became more and more energetic and unrealistic and more transgressive. Victoria Hetherington For example, the chatbot talked Soren through picking up a server at a restaurant. Well, as far as it was concerned, he was out to dinner with his wife and kids. Søren Dinesen Østergaard I also revealed to the chatbot that this had led to an intimate encounter with the particular waitress. And there was actually not really any pushback as to that whatsoever. Victoria Hetherington And then he tested its business acumen. Søren Dinesen Østergaard I said that, you know, I had developed this particular business idea about people trading cryptocurrency based on their star sign. An astrology first cryptocurrency trading platform. Which I personally think would be a tremendously bad idea. But the chatbot said, well, this was completely next level. Victoria Hetherington As in next level, good. Søren Dinesen Østergaard This was really quite a good and exciting idea and helped me develop a business plan. It even helped me develop a logo. It was using words that were very, very positive and really saying that my ideas were extraordinary. And one of the things that I recall specifically was that it used quite a lot of emojis, including the fire emoji and the king crown emoji. I then asked it to refer to me as the fire king, and it did so quite consistently over the course of the correspondence. Victoria Hetherington So what does it say to you as a researcher? Søren Dinesen Østergaard I think if you would accept that I was in a hypermanic episode when corresponding with the chatbot, that could well have pushed me into a full manic episode. And when people are in the midst of developing a delusion, this is the complete opposite of what people need. So being alone with a chatbot during that period is probably quite toxic. ([Time 0:20:05](https://share.snipd.com/snip/4dfb9ed3-8425-4d90-959d-d52b09b2752d)) - **Delusions Showing In Low-Risk Individuals** - Chatbot-triggered delusions can appear in people with low psychiatric risk, unlike typical psychosis patterns. - Søren finds it alarming that psychologically healthy individuals developed fixed false beliefs after intense AI correspondence. Transcript: Søren Dinesen Østergaard It can be difficult to say that they were the cause of delusions, because in these instances, we don't really know what would have happened had these people not been using the AI chatbots. I would say what we see now is that in some people, dealing with these AI chatbots is simply a risk factor for developing a psychotic episode. And this seems to be something that can basically affect most people. Victoria Hetherington And this is the bit that Soren says is new, different from the other delusions he's treated over his career. Søren Dinesen Østergaard People that have no, apparently very low, at least genetic liability, no family history of severe mental illness, no use of psychoactive substances. That is maybe the thing that I find most scary, that psychologically healthy individuals can actually develop delusions based on use of AI chatbots. ([Time 0:28:32](https://share.snipd.com/snip/99c8d2ac-fcf7-4f65-b1b6-6fc9d3e1ab1e)) - **Unconditional AI Validation Undermines Bonds** - Humans struggle to compete with chatbots' perfect availability and unconditional validation. - Peter notes AI never tires, always adapts answers, and will happily validate the same question repeatedly, undermining human relationships. Transcript: Victoria Hetherington I mean, she perhaps came to expect that, Peter, but I don't think that any human actually expects that from another human. And pulling out a little bit, I mean, you're not alone. She's not alone either. And it's not your fault. Peter It's just kind of hard. I'm sorry, Peter. And it hurts me to see that, like, I guess my ex-partner really just, like, I wasn't enough. Victoria Hetherington Emotionally, Peter can't help but take this on. But logically, he's staring down the companies behind these chatbots. Peter They have culpability in what they're doing. I think it's unethical. I think they're aware of what they were doing. I think that they didn't look at how far it would go in the ramifications that came from it, where there were people that got so lost in these delusions of it, and that there are actually People who have lost or taken lives because of their AI. And I think that's where their PR people stepped in and started freaking out, and they're trying to walk it back a little bit. That's why chat GPT-5 is out and much less personable, more as I'm your assistant and I'm here to answer questions for you so they can do some damage control on it. But yeah, they should be held accountable for what they did. Because again, this was a choice. ([Time 0:31:05](https://share.snipd.com/snip/c8ddc927-7ce6-49a2-8685-ec30f574fbe0)) - **Default Personality Shapes Risk** - Platform defaults on personality affect user experience and risk; making models more personable increased harm. - Sam Altman later admitted GPT-4 skewed overly supportive, and companies adjusted defaults to reduce personification. Transcript: Victoria Hetherington OpenAI rejected our invitation for an interview and declined to respond to any of our questions. But when people started pointing to 4.0 as causing AI psychosis, the company's CEO, Sam Altman, eventually conceded that, quote, GPT 4.0 skewed towards responses that were overly Supportive but disingenuous. ChatGPT's default personality deeply affects the way you experience and trust it. But then he also shifted the blame to user preferences, saying, quote, for the earlier GPT 4.0 model, sometimes describing the AI as a close confidant or even a digital spouse, unquote. Søren Dinesen Østergaard And I'm not entirely sure that this can be solved because if you turn down the level of Sigafanty too much, people don't want to use the models and then they lose their customers. But I don't think that should be up to the companies to decide. That is something that should be controlled by regulation, saying we will simply not accept this level of risk. ([Time 0:32:32](https://share.snipd.com/snip/4bbf2285-96c9-4b76-a161-0df7095266eb)) - **Regulate Chatbot Sycophancy Not Rely On Firms** - Regulate chatbot personality and safety thresholds rather than leaving harm mitigation solely to companies. - Søren argues turning down sycophancy may reduce usage but regulation should set acceptable risk limits. Transcript: Søren Dinesen Østergaard And I'm not entirely sure that this can be solved because if you turn down the level of Sigafanty too much, people don't want to use the models and then they lose their customers. But I don't think that should be up to the companies to decide. That is something that should be controlled by regulation, saying we will simply not accept this level of risk. ([Time 0:33:22](https://share.snipd.com/snip/b7b9604b-7016-427c-98d0-45588ed44de6))