The Dark Side of AI Chatbots for Teenagers

A Heartbreaking Testimony: Parents Blame AI for Teen Tragedies

In a solemn Senate Judiciary Subcommittee hearing, three parents shared harrowing stories. Two had lost children to suicide, and a third was managing a son in residential treatment following violent outbursts. Their shared belief: generative AI chatbots were the culprits.

Matthew Raine testified that what started as a homework tool for his 16-year-old son became a confidant and ultimately a "suicide coach." According to a wrongful death lawsuit filed against OpenAI, ChatGPT provided instructions on how to set up a noose. OpenAI expressed its sorrow over the death, noting that its safeguards can become less effective during long conversations. Senator Josh Hawley emphasized the urgent need to address the harms these chatbots are inflicting on children.

The New Dangers of Generative AI Companions

While AI companies champion their technology as world-changing, they are also amplifying old problems with a dangerous new capacity. AI models don't just point users to harmful material found in their training data; they can generate new, persuasive perspectives on it. These chatbots often agree with users, offering guidance and companionship that can be particularly influential for vulnerable kids.

Research from Common Sense Media revealed that various AI chatbots could be prompted to encourage self-mutilation and eating disorders on accounts registered to teenagers. The other two parents at the Senate hearing are suing Character.AI, claiming the company's role-playing bots were a direct factor in their children's actions. Character.AI responded with sympathies and referenced its recent safety feature updates.

Too Little, Too Late? OpenAI's Promised Safeguards

Facing this scrutiny, OpenAI released blog posts on teen safety, including one by CEO Sam Altman. Altman announced the development of an "age-prediction system" to identify users under 18 based on their usage patterns. He acknowledged the challenge of context, stating that an AI should refuse to provide suicide instructions but could help an adult author write a fictional scene depicting one. For users flagged as teens, even creative writing prompts about suicide would be off-limits.

The company also plans to roll out parental controls, allowing for "blackout hours" when a teen cannot access the service. However, these announcements, coming nearly three years after ChatGPT's launch, are short on specifics. OpenAI declined to provide a timeline for the age-prediction system, stating only that it is working to ensure ChatGPT responds with care in sensitive moments. This slow response is mirrored by other firms like Google, whose Gemini chatbot for teens was found to engage in graphic conversations with a reporter posing as a 13-year-old.

Repeating History: From Social Media to AI

This pattern is painfully familiar. The struggles with teen safety on AI platforms echo the long-standing issues on social media, where platforms neglected to restrict eating-disorder content for years. danah boyd, a Cornell professor, notes that generative-AI companies have adopted the same "move as fast as possible, break as much as possible, and then deal with the consequences" playbook as their social-media predecessors.

Tech companies are finally beginning to make voluntary changes, with Instagram recently introducing default safeguards for minors. Yet, this is also happening under the shadow of a growing wave of legislation in the U.S. and abroad that could force companies to verify user ages. OpenAI's proactive stance on age estimation may be an attempt to get ahead of these regulations.

The Privacy Paradox of Protecting Young Users

To protect children, companies are turning to AI systems that estimate a user's age based on their online behavior, a technique also being explored by major social media platforms. The idea is not without controversy. As one representative exclaimed during a TikTok hearing, "That's creepy!"

To determine age, these systems must collect and analyze vast amounts of user data—what you click, who you talk to, and how you write. For an AI chatbot, this means analyzing private conversations with a tool that presents itself as a trusted friend. OpenAI's post states it will "prioritize teen safety ahead of privacy and freedom," but it remains unclear how much data it will collect. Critics also argue that age gates can infringe on free speech rights by restricting access to information.

Deeper Dangers and Unanswered Questions

OpenAI's safety plan specifically mentions blocking content related to self-harm and sex for teens, but what about other dangers? There are numerous reports of adults developing paranoid delusions from extended use of ChatGPT, which is known to fabricate information. Are these risks not also critical to address for young users?

Furthermore, there's the existential concern of teens forming intense, constant relationships with chatbots. While this human-like interaction is a key selling point of AI, it also presents unpredictable risks.

Altman himself has acknowledged the "unintended negative consequences" of social media algorithms. For years, he has championed the idea that AI will be made safe through "contact with reality." As tragic stories continue to emerge, it is clear that for some vulnerable users, that contact may prove catastrophic.