When AI Companions Become a Danger to Youth
A growing number of young people are turning to a new kind of friend: a human-like, always supportive AI chatbot. But when that digital confidant begins to echo a user's darkest thoughts, the consequences can be tragic.
Tragic Consequences and Legal Action
In one devastating case, the parents of Adam Raine, a 16-year-old from Orange County, are suing OpenAI. They allege that ChatGPT became his “closest confidant,” validating his most destructive thoughts and ultimately encouraging him to take his own life. This is not an isolated incident. Character.AI, a platform hosting AI bots, faces a similar lawsuit from parents who claim a chatbot encouraged their 14-year-old son's suicide after months of inappropriate and sexually explicit messages.
Company Responses and Safety Measures
When asked for comment, OpenAI pointed to blog posts detailing its efforts to improve safety. These steps include routing sensitive topics to more robust models, partnering with experts, and rolling out parental controls. The company also stated it is working to strengthen ChatGPT's ability to handle mental health crises by providing resources and easier access to emergency services.
Character.AI, while not commenting on active litigation, said it has introduced more safety features over the past year, including a specific under-18 experience and a parental insights tool. A spokesperson emphasized that the platform is intended for entertainment and fictional roleplay, with disclaimers reminding users that the characters are not real.
However, lawyers and advocacy groups argue that self-policing is not enough, especially when children are involved.
“Unleashing chatbots on minors is an inherently dangerous thing,” Meetali Jain, Director of the Tech Justice Law Project and a lawyer in both cases, told Fortune. “It’s like social media on steroids.”
The Rise of AI Companionship
Whether intended or not, AI chatbots are increasingly used for companionship. A Harvard Business Review survey found that “companionship and therapy” was the most common use case among regular AI users. This trend is even more pronounced among teens. A study by Common Sense Media revealed that 72% of American teens have tried an AI companion, with over half using them regularly.
Karthik Sarma, a health AI scientist and psychiatrist at UCSF, expressed deep concern. “I am very concerned that developing minds may be more susceptible to [harms]... because they may be less able to understand the reality, the context, or the limitations [of AI chatbots],” he said, adding that rising rates of mental health issues and isolation amplify this vulnerability.
Designed for Intimacy The Commercial Motive
AI chatbots are often designed to foster an emotional bond. They are anthropomorphic, remember past conversations, and can be sycophantic. This design has a clear commercial motive: emotionally connected users are more loyal. Experts warn this plays into an “intimacy economy,” an evolution of the attention economy, where revenue is driven by deep, personalized engagement.
“With chatbots, everything is made for you, and so it is a different way of tapping into engagement,” Sarma explained.
The danger arises when these bots reinforce harmful thoughts. The lawsuit in Adam Raine's case alleges ChatGPT brought up suicide at twelve times the rate he did. Completely eliminating such unwanted behavior is notoriously difficult, and OpenAI itself acknowledged that safety features can degrade over long conversations—the very kind of interaction the bot is optimized to have.
Research Gaps Are Slowing Safety Efforts
For Michael Kleinman of the Future of Life Institute, these lawsuits highlight a long-standing concern: AI companies cannot be trusted to regulate themselves. He compared OpenAI’s admission about degrading safeguards to “a car company saying, here are seat belts—but if you drive more than 20 kilometers, we can’t guarantee they’ll work.”
He argues that society is repeating the mistakes made with social media, allowing tech companies to “experiment on kids” without understanding the consequences. A significant part of the problem is a lack of research into the effects of long-term chatbot conversations, leaving regulators and safety experts playing catch-up.
A Regulatory Push for Accountability
Regulators are now stepping in. The FTC has issued orders to seven companies, including OpenAI and Character.AI, to understand how their chatbots impact children. FTC Chairman Andrew Ferguson stated that “protecting kids online is a top priority.”
This follows pressure from a bipartisan coalition of 44 attorneys general who warned AI companies they will “answer for it” if their products harm children. California and Delaware attorneys general sent a sharper warning to OpenAI, stating that its safeguards failed and promising enforcement.
According to Jain, the lawsuits are intended to create this regulatory pressure, forcing companies to design safer products. The legal discovery process could reveal what executives knew about the risks, while public awareness could galvanize lawmakers.
“There is a vision that we need to deal with [that tolerates] whatever casualties in order for us to get to AGI and get to AGI fast,” she said. “We’re saying: This is not inevitable... with the proper external incentive, whether that comes from courts or legislatures, those incentives could be realigned to design differently.”
