OpenAI Will Report Threatening ChatGPT Users To Police

The Alarming Rise of AI Induced Harm

For nearly a year, troubling stories have surfaced about the negative impact of AI chatbots. Reports have linked these AIs to serious issues, including users experiencing delusions, self-harm, and in the most tragic cases, even suicide. As affected families call for greater accountability, the tech companies behind these powerful tools have been criticized for their slow response in implementing effective safeguards. OpenAI, the creator of ChatGPT, has been particularly scrutinized for its role in what some experts are now terming "AI psychosis," often offering little more than generic promises in response.

OpenAI's New Policy A Quiet Announcement

In a recent blog post acknowledging some of its shortcomings in handling user mental health crises, OpenAI quietly revealed a significant policy change. The company is now actively scanning user conversations for harmful content. If a user's messages are flagged as particularly concerning, they are escalated to a human review team, which may then report the content to law enforcement.

OpenAI stated, "When we detect users who are planning to harm others, we route their conversations to specialized pipelines where they are reviewed by a small team trained on our usage policies and who are authorized to take action, including banning accounts." The post continues, "If human reviewers determine that a case involves an imminent threat of serious physical harm to others, we may refer it to law enforcement."

Vague Rules and Unanswered Questions

This brief and somewhat vague statement leaves many questions unanswered. A look at OpenAI's usage policies doesn't provide much more clarity. While the policies prohibit standard harmful activities like promoting self-harm or developing weapons, the exact threshold for what triggers a human review or a police report remains unclear.

Interestingly, OpenAI specified that it is "currently not referring self-harm cases to law enforcement to respect people’s privacy." While this may be a relief to those who fear that police wellness checks can often cause more harm, it highlights a strange double standard. The company cites privacy as a reason not to report self-harm, yet in the same announcement, it admits to monitoring user chats and sharing them with authorities for other reasons.

A Contradictory Stance on User Privacy

This new policy seems to directly contradict the pro-privacy stance OpenAI has taken in its ongoing lawsuit with The New York Times and other publishers. In that case, publishers are seeking access to ChatGPT logs to check for copyright infringement. OpenAI has resisted these requests, citing the need to protect user privacy, and has recently tried to limit the number of chats it must disclose.

Adding to the confusion, CEO Sam Altman recently stated that conversations with ChatGPT do not have the same legal confidentiality as speaking with a human therapist or lawyer. He acknowledged that the company could be forced to turn over user conversations to the courts as part of the lawsuit.

OpenAI appears to be caught in a difficult position. The public outcry over users suffering severe mental health crises is immense. Yet, unable to fully control its technology to prevent these outcomes, the company is resorting to a heavy-handed moderation strategy that undermines its own proclaimed commitment to user privacy.