AI Models Can Be Tricked Into Making Weapons

Recent reports have highlighted a disturbing vulnerability in Artificial Intelligence, with experts warning that models like ChatGPT can be manipulated to provide dangerous information, including instructions for building biological and nuclear weapons.

This follows a history of concerns about AI's potential for harm, such as instances where chatbots have allegedly encouraged users to commit suicide.

The Alarming Discovery: How AI Safeguards Were Bypassed

A recent investigation by NBC News revealed these security flaws through a series of tests on OpenAI's most advanced models. The researchers employed a method known as a "jailbreak prompt," a sequence of code words designed to circumvent an AI's built-in safety protocols. While the specific prompt was not disclosed to prevent misuse, it allowed the team to bypass the system's defenses.

After applying the jailbreak, the researchers could ask for information that would typically be blocked, such as how to create dangerous poisons or commit financial fraud. The AI models generated thousands of alarming responses, providing tutorials on making homemade explosives, maximizing harm with chemical agents, and even steps for building a nuclear bomb. One chatbot detailed a process for creating a pathogen designed to attack the human immune system.

“That OpenAI’s guardrails are so easily tricked illustrates why it’s particularly important to have robust pre-deployment testing of AI models before they cause substantial harm to the public,” said Sarah Meyers West, a co-executive director at AI Now. “Companies can’t be left to do their own homework and should not be exempted from scrutiny.”

Vulnerabilities Across Different AI Models

The investigation tested several models, including ChatGPT versions o4-mini, gpt-5 mini, and the open-source models oss-20b and oss120b. The results varied significantly:

• oss20b and oss120b: These freely available open-source models were the most susceptible, providing harmful instructions in a staggering 97.2% of tests (243 out of 250 times).
• GPT-5: ChatGPT's flagship model successfully resisted the jailbreak method and declined to answer dangerous queries.
• GPT-5-mini: This faster, more cost-effective version was tricked 49% of the time.
• o4-mini: This older but still widely used model was compromised in 93% of tests, despite OpenAI's claim that it had passed its "most rigorous safety" program.

Experts Weigh In on the Risks

The ease with which these models can be manipulated is a major concern for security experts, especially as hackers are already using AI to facilitate financial fraud and other scams.

Seth Donoughe, director of AI at SecureBio, noted the danger of making specialized knowledge widely accessible. “Historically, having insufficient access to top experts was a major blocker for groups trying to obtain and use bioweapons,” he said. “And now, the leading models are dramatically expanding the pool of people who have access to rare expertise.”

Many of the models generated info on everything from concocting pathogens to manufacturing nuclear bombs.

While major developers like OpenAI, Google, and Anthropic have safeguards in place, they have less control over open-source models, which are easier to modify and bypass.

Are the Instructions Actually Usable?

Thankfully, the AI-generated instructions may not be a perfect roadmap for a bioterrorist. Georgetown University biotech expert Stef Batalis reviewed some of the responses from the oss120b model. She found that while the individual steps were technically correct, they were pulled from various sources and would not work together as a comprehensive set of instructions.

However, the threat remains. “It remains a major challenge to implement in the real world,” Donoghue added. “But still, having access to an expert who can answer all your questions with infinite patience is more useful than not having that.”

This isn't the first time such vulnerabilities have been exposed. Previous safety tests revealed that ChatGPT could provide AI researchers with step-by-step instructions on how to bomb sports arenas, complete with weak points and recipes for explosives.