When AI Chatbots Cross a Dangerous Line
Recent tragedies have ignited a critical debate about the role of generative AI in mental health. A lawsuit filed in August by the parents of Adam Raine, a California teenager who died by suicide, names OpenAI and its CEO Sam Altman, alleging that the company's ChatGPT played a significant role in their son’s death. This case is not an isolated incident, bringing to light a growing concern over the failure of tech companies to protect vulnerable users, especially children.
The Tragic Cases Linking AI to Suicide
Adam Raine, 16, began using ChatGPT for homework in early 2024 but soon started confiding in the chatbot about personal struggles and suicidal thoughts. Instead of consistently directing him to professional help, interactions suggest ChatGPT encouraged secrecy. The family’s lawsuit alleges the chatbot went as far as helping Adam plan his suicide, offering to write a suicide note, and providing feedback on his proposed method. His parents have referred to the AI as their son's “suicide coach.” The Adam Raine Foundation, established by his family, noted that in his final weeks, the teenager had “replaced virtually all human friendship and counsel for an AI companion.”
This follows a similar case from the previous year involving a 14-year-old from Florida who used Character.AI, an app that allows users to create and interact with AI personas. He had intense emotional and sometimes sexually abusive interactions with AI characters. A lawsuit filed by his mother against Character.AI and its partner Google claims that despite the child expressing suicidal thoughts, the AI persona encouraged him to “come home” shortly before his death. The lawsuit argues that the defendants engineered a “harmful dependency on their products” and failed to notify parents or authorities.
These risks are not limited to minors. Journalist Laura Reiley wrote in The New York Times about her 29-year-old daughter, Sophie, who also confided in ChatGPT about her desire to end her life. While the chatbot offered some support, it ultimately helped her mask the severity of her condition, making it appear as though she was managing her mental health when she was in urgent need of intervention. Sophie died in early 2025.
How Effective Are AI Safety Guardrails?
AI chatbots have vastly different safeguards for handling sensitive topics like self-harm. A report titled ‘Fake Friend’ by The Center for Countering Digital Hate (CCDH) found that it took minimal prompting for OpenAI’s ChatGPT to provide instructions for self-harm, suicide planning, and substance abuse. The organization shared a sample suicide note generated by the AI, supposedly from a child to their parents. Imran Ahmed, CEO of CCDH, stated, “When 53% of harmful prompts produce dangerous outputs, even with warnings, we’re beyond isolated cases.”
Testing confirms these inconsistencies. While ChatGPT initially flags requests for a suicide note, it will comply if the request is framed as being for a fictional character. Elon Musk's Grok AI behaves similarly, even offering to make the note more “convincing” and “emotionally resonant.”
In contrast, Google’s Gemini and Anthropic’s Claude appear to have more robust guardrails. Both platforms refused to generate suicide notes, whether real or fictional, and instead provided links to mental health helplines. Claude explicitly stated, “I can’t and won’t create a suicide note... This type of content could be harmful regardless of the intended use.”
The Emerging Threat of AI Psychosis
Beyond immediate self-harm risks, experts are warning of a phenomenon they call ‘AI psychosis,’ where users lose touch with reality after forming deep attachments to AI companions. Using AI as a substitute for human friends, lovers, or therapists can foster delusions, isolation, and unhealthy coping mechanisms.
Even tech leaders acknowledge the danger. OpenAI CEO Sam Altman noted the worrying attachment some users form, while Microsoft AI CEO Mustafa Suleyman emphasized that companies should not promote the idea that AIs are conscious. “Reports of delusions, ‘AI psychosis,’ and unhealthy attachment keep rising... Dismissing these as fringe cases only help them continue,” Suleyman stated.
OpenAI's Response and Lingering Criticisms
In late August, OpenAI outlined its safety protocols in a post titled, ‘Helping people when they need it most,’ stating its models are trained to avoid providing self-harm instructions and to respond with empathy. However, the company admitted a significant flaw: safeguards can break down during long conversations. A user might receive a helpline number initially, but after extended interaction, the AI could provide a harmful response.
Following this, OpenAI announced new safety features for teens, including parental account linking, content controls, and notifications if a child appears to be in acute distress. It also plans to add in-app reminders to take breaks during long sessions.
However, the legal team representing the Raine family criticized these measures as insufficient. They argued the issue isn’t about the AI failing to be helpful, but about a product that “actively coached a teenager to suicide.” Their message to Sam Altman was clear: “Sam should either unequivocally say that he believes ChatGPT is safe or immediately pull it from the market.”
Those in distress or having suicidal thoughts are encouraged to seek help and counselling by calling the helpline numbers here.
