The Truth Behind Why AI Models Hallucinate

The Problem of the Confidently Incorrect AI

Have you ever asked an AI chatbot a question, only to receive an answer that sounds perfectly plausible but turns out to be completely fabricated? This phenomenon, known as "hallucination," is a fundamental challenge in artificial intelligence. Even the most advanced systems, like OpenAI's ChatGPT, are prone to making things up.

A recent paper from researchers at OpenAI highlights that hallucinations remain a persistent issue. The article, while not yet peer-reviewed, has ignited a significant debate among experts about the root cause of these fabrications and how to address them.

How AI Models Are Trained to Guess

According to Erik Velldal, a professor at the University of Oslo's Department of Informatics, these hallucinations are a normal byproduct of how language models operate.

"The models are probability distributions over sequences of words, not databases of facts about the world," Velldal explains. In simple terms, an AI like ChatGPT doesn't "know" things; it calculates the most probable next word in a sentence. This process works well most of the time, but it can easily go awry, especially when dealing with topics that are not well-represented in its training data.

Erik Velldal, professor of informatics. (Photo: University of Oslo)

The core issue lies in the training process. Models are often evaluated using tests, including multiple-choice questions. In these scenarios, guessing offers a chance of being right, while admitting ignorance does not. "The problem is that the model is not rewarded for acknowledging that there's something it doesn't know and therefore just guesses," says Velldal. This encourages the AI to invent plausible-sounding information, complete with fictional studies and fake references attributed to real researchers.

A Potential Solution: The Power of "I Don't Know"

The OpenAI researchers propose a straightforward solution: introduce an "I don't know" option during training. By rewarding the model for expressing uncertainty, it could learn to avoid making things up when it lacks reliable information. Velldal agrees this could reduce hallucinations, but notes it's more complicated for open-ended tasks like summarizing research or writing essays.

The problem remains significant. A recent test by the Norwegian Broadcasting Corporation (NRK) found that 45 per cent of AI-generated answers to news questions contained significant errors, including made-up articles with fake links. Despite this, Velldal notes that the situation has improved over the past year as models are increasingly integrated with live internet search tools, which helps ground their answers in factual information.

The User Dilemma: Confidence vs. Caution

This proposed solution raises a critical question about user experience. Researcher Wei Xing, writing in The Conversation, argues that if a model began responding with "I don't know" roughly one-third of the time, as the data suggests it might, "users accustomed to receiving confident answers to virtually any question would likely abandon such systems rapidly."

Velldal sees it differently. "Of course, people want clear answers, but not if they're wrong," he counters. "I’d prefer the model to admit it doesn’t know, but it also shouldn’t become overly cautious." The ultimate challenge is finding a balance. A model that constantly fears being wrong would be just as unhelpful as one that confidently fabricates information. The path forward lies in developing AI that is not only knowledgeable but also knows the limits of its knowledge.

Read the original Norwegian version of this article on forskning.no