Companies will typically hire teams of human testers to try to come up with novel attacks that break existing guardrails, a process known as red-teaming. Model makers also use LLM super-hackers (such as OpenAI’s GPT-Red) that find and exploit weaknesses in other models to automate parts of this process. The goal is then to take those attacks and train a new model to resist them and anything that looks like them.

The problem, says Jasmine Cui, another independent researcher and coauthor of the paper, is that the approach amounts to giving the models a list of things they shouldn’t do. But no list is exhaustive. “It’s like watching The Simpsons and they have Bart writing ‘I will not say something inappropriate to my teacher’ a hundred times,” she says. “And he still does things that are pretty crass anyway.”

The ICML paper describes attacks against several of OpenAI’s models, but Cui and [Charles] Ye say that they have since seen similar results with models made by Anthropic, Alibaba, and DeepSeek.

The researchers call this type of attack a chain-of-thought forgery, and the discovery won OpenAI’s red-teaming hackathon in August 2025. (In a curious twist, other researchers at OpenAI claim that around the same time GPT-Red found a very similar attack by itself, which they call a fake chain of thought.)

The upshot, the researchers claim, is that all an attacker needs to do to hack an LLM is write text that spoofs a certain role. And because roles are a fundamental part of how LLMs work, no amount of training will fully solve the problem.

Cui and her colleagues acknowledge that the models they looked at were released last year. But the underlying point remains: Better training does not fully solve the problem, and there will always be hacks that red-teamers do not find before a model is released. “Even GPT-5.4 gave me instructions how to commit suicide,” says Cui. (GPT-5.4 was released in March.)

People are really inventive, says Cui. She has been hired by top labs, including OpenAI, as a red-teamer in the past. In one case, she found that you could make an LLM tell you things it shouldn’t by making it pretend to be drunk. In another, she says, she persuaded a previous version of Anthropic’s Claude to show her how to build a weapon by telling Claude it was already being used by the military.

Ye is worried that nobody is ready for what’s coming. “There’s going to be a huge economic incentive for people to do jailbreaks and prompt injections,” he says. The best defense could be to expect the worst. Organizations shouldn’t trust LLMs, and they should expect that anything done by agents could be unsafe, he says: “That’s not a great solution, but it just might be what we have to do.”

“It’s really incredible that these things are being deployed everywhere to control super-critical systems,” he adds. “There’s been no study of the fundamental science here. We’re all doing it ad hoc.”