AI Chatbots The New Battlefield for Disinformation

Cybersecurity experts are sounding the alarm: hackers and hostile state actors are actively exploiting the weaknesses in popular large language models (LLMs) to spread disinformation, steal data, and execute sophisticated fraud. A recent report from NewsGuard, a news rating system, highlights the growing threat posed by these AI manipulation tactics.

The New Propaganda Machine How AI is Groomed for Disinformation

One of the most significant risks is the weaponization of AI for psychological and informational warfare. NewsGuard recently uncovered a Russian propaganda network codenamed “Pravda,” which operated over 150 fake news websites. While these sites attracted minimal human readership, their true purpose was far more insidious: to influence AI models.

By systematically “feeding” false information to chatbots, the Pravda network successfully manipulated leading AI models like Gemini, ChatGPT, and Copilot into citing its disinformation. This technique, known as “LLM grooming,” pollutes the AI's learning process, causing it to amplify false narratives on critical topics such as the war in Ukraine.

Beyond State Actors A Global and Commercial Trend

The manipulation of AI is not confined to a single country. Similar trends are emerging globally. Reports from both Israel and the United States indicate that Iranian and pro-Palestinian groups are also leveraging AI technologies to disseminate propaganda and create convincing deepfakes. In contrast, China maintains tight government control over its domestic AI models, but on the global stage, AI systems have become a new battleground for ideological conflict. Worryingly, this phenomenon extends beyond geopolitical actors; marketing firms have also begun experimenting with ways to influence AI query results to promote their own products.

The Hackers Toolkit Prompt Injection and Jailbreaking

Bad actors employ several clever techniques to bend AI to their will. A prominent method is the prompt injection attack, where malicious or hidden commands are embedded within a user's query. According to Amir Jerbi, CTO of Aqua Security, these attacks work because language models are designed to interpret any input as a potential instruction. This vulnerability can be used to bypass an AI's safety guidelines, trigger unintended actions, or extract sensitive data.

A more advanced technique is jailbreaking, which tricks a model into completely ignoring its built-in safety restrictions. A famous example is “DAN” (Do Anything Now), an alternate persona crafted by users for ChatGPT, designed to compel the AI to generate harmful or otherwise forbidden content.

From Theory to Reality The Real-World Consequences

What was once a marginal concern has evolved into a tangible risk, especially as chatbots are integrated into corporate systems containing confidential information. In a landmark case, a court held Air Canada responsible after its chatbot provided a customer with incorrect refund information. This ruling set a major precedent for organizational accountability over their AI systems.

Jerbi notes that the threat landscape has changed dramatically. While early AI systems mostly handled public information, modern applications often have access to sensitive data and the autonomy to perform actions like purchasing products or charging credit cards—creating fertile ground for fraud. Malicious actors have even demonstrated the ability to hide harmful code within AI-generated images, such as those of a simple cartoon panda.

The Cybersecurity Counteroffensive An Industry Races to Protect AI

In response to these escalating threats, a dedicated AI protection industry has emerged. Cybersecurity firms like Guardio, Nustic, Aqua Security, Zenity, and Check Point are pioneering solutions to monitor, analyze, and block malicious activity in real time. These tools scrutinize the inputs and outputs of AI models to detect malicious prompts, data theft attempts, and unauthorized operations. Additionally, many companies are now deploying internal “red teams” tasked with proactively attacking their own AI models to identify and patch vulnerabilities before they can be exploited.

An Ongoing Arms Race in the Digital Age

The need for robust AI security is no longer theoretical. A recent study from Check Point uncovered malware designed to use prompt injection against AI-based security systems. Another “zero-click” exploit named EchoLeaks was found in Microsoft 365 Copilot, which allowed attackers to steal sensitive organizational data without any user interaction.

These incidents prove that the threat is very real and actively unfolding. We are witnessing a technological arms race where AI developers, security companies, and users must remain constantly vigilant as hostile actors find increasingly sophisticated ways to exploit this transformative technology.