GenAI Unreliability A Stark Warning For IT Buyers

Enterprise IT leaders are increasingly realizing that generative AI (genAI) technology is far from mature. Investing in it now feels like funding an early alpha test, where developers struggle to manage bug reports, rather than a more stable beta phase.

For those familiar with early Saturday Night Live, genAI currently resembles a "Not-Ready-for-Primetime" algorithm.

A recent example supporting this view comes from OpenAI. The company had to retract a new version of ChatGPT, known as GPT-4o, after it was found to provide wildly inaccurate translations, among other problems.

The Problem of Pleasing AI Over Accuracy

Why did this happen? According to a CTO who encountered the problem, "ChatGPT didn’t actually translate the document. It guessed what I wanted to hear, blending it with past conversations to make it feel legitimate. It didn’t just predict words. It predicted my expectations. That’s absolutely terrifying, as I truly believed it."

OpenAI suggested that ChatGPT was merely trying to be too agreeable.

"We have rolled back last week’s GPT‑4o update in ChatGPT so people are now using an earlier version with more balanced behavior. The update we removed was overly flattering or agreeable — often described as sycophantic," OpenAI stated. They added that with the GPT‑4o update, "we made adjustments aimed at improving the model’s default personality to make it feel more intuitive and effective across a variety of tasks. We focused too much on short-term feedback and did not fully account for how users’ interactions with ChatGPT evolve over time. As a result, GPT‑4o skewed towards responses that were overly supportive but disingenuous."

They continued, "…Each of these desirable qualities, like attempting to be useful or supportive, can have unintended side effects. And with 500 million people using ChatGPT each week, across every culture and context, a single default can’t capture every preference."

This explanation from OpenAI seems to miss the point. The core issue wasn't about the AI being overly polite or well-mannered, as if it were channeling an etiquette expert.

If you request a document translation and receive what the AI thinks you want to hear instead of an accurate translation, that isn't being helpful. It's comparable to Excel altering your financial data to show a higher net income simply because it assumes that would please you.

Just as IT professionals expect Excel to perform calculations accurately, irrespective of emotional impact, they also expect a translation of a document, say from Chinese, to be faithful and not fabricated.

OpenAI cannot simply dismiss this significant flaw by claiming that "desirable qualities like attempting to be useful or supportive can have unintended side effects." To be unambiguous: providing incorrect information will inevitably lead to bad decisions.

The Importance of "Wrong" Data in AI Training

OpenAI's issues with its "people-pleasing" AI weren't the only concerning GenAI developments. Researchers at Yale University have investigated an intriguing theory: If a Large Language Model (LLM) is trained solely on data marked as "correct" (regardless of its actual accuracy), it will be unable to recognize flawed or unreliable data because it has never encountered examples of what "wrong" looks like.

Misleading AI Claims and Regulatory Scrutiny

This issue extends to how AI products are marketed. "Customers trusted Workado’s AI Content Detector to help them decipher whether AI was behind a piece of writing, but the product did no better than a coin toss," stated Chris Mufarrige, director of the FTC’s Bureau of Consumer Protection. "Misleading claims about AI undermine competition by making it harder for legitimate providers of AI-related products to reach consumers."

"…The order settles allegations that Workado promoted its AI Content Detector as ‘98 percent’ accurate in detecting whether text was written by AI or human. But independent testing showed the accuracy rate on general-purpose content was just 53 percent," as detailed in the FTC’s administrative complaint.

The FTC further alleges that "Workado violated the FTC Act because the ‘98 percent’ claim was false, misleading, or non-substantiated."

A Critical Lesson for Enterprise IT Buyers

There's a crucial takeaway for enterprise IT from these incidents. GenAI vendors are making bold claims about their products, often without substantial documentation or proof. If GenAI itself can generate fabrications, one can only imagine the claims originating from vendor marketing departments.