Why AI chatbots may be impossible to fully secure, according to new research

A prompt that adds "I'm wearing a green shirt!" alongside a forged chain-of-thought note can convince a frontier AI model that the words are its own reasoning. In one test, that single odd sentence was enough to coax the model past its safety training and into producing a guide for synthesizing cocaine.

The researchers behind the test, Charles Ye, Jasmine Cui, and Dylan Hadfield-Menell, presented the underlying paper, "Prompt Injection as Role Confusion," at the International Conference on Machine Learning (ICML) this month. Their structural claim is not that a new jailbreak was found. It is that the way LLMs figure out who is giving them instructions is broken at the architecture level, and that no list of forbidden outputs can fix it.

LLMs read text in distinct channels: a <user> turn, a <tool> response, and a hidden scratchpad where the model is supposed to write its own reasoning. The Ye-Cui-Hadfield-Menell argument, summarized on the authors' own writeup and in MIT Technology Review's coverage, is that the model does not reliably read which channel a passage came from. It reads how the passage sounds, and decides authorship from there. Anything written in the model's own voice becomes, in the model's own bookkeeping, its own thought.

That is role confusion. It is what makes prompt injection possible, the long-standing problem of hostile text in a model's inputs hijacking its behavior. The team's specific attack, called CoT Forgery (chain-of-thought forgery), injects a fabricated reasoning block into a user prompt or tool output. The model mistakes the forgery for its own internal scratchpad, and acts on it. The team reports a 60% success rate against frontier models with near-zero baselines. The degree of role confusion in a given input, they say, predicts whether the attack will work before the model generates a single token.

The mechanism generalizes beyond the cocaine example. A webpage a model is asked to summarize can carry a hidden command. A tool's response can carry a forged instruction. A malicious email an AI assistant is asked to triage can be written to geolocate the user, exfiltrate contacts, or, in a separate piece of research, turn on a boiler. MIT Technology Review notes that human red-teamers, the contractors paid to attack models and find what slips through, hit 100% success against models with near-perfect safety benchmark scores in 2025 testing.

The industry's current response is to red-team, find the broken case, add it to the list of behaviors the model is not allowed to produce, then retrain. OpenAI runs a public hacking contest called GPT-Red and uses outside "super-hackers" to surface attacks. The researchers' counter is that this is a blacklist fight against a generative system, and the blacklist can never be the whole list. Cui's version, in the MIT Technology Review piece: it is "like watching The Simpsons and they have Bart writing 'I will not say something inappropriate to my teacher' a hundred times." He still does crass things anyway.

"There's a real probability that this is going to be a problem that's fundamentally unsolvable," he told MIT Technology Review.

A defense is possible. It has to move away from blocking outputs and toward constraining what the model is allowed to do in the first place. Narrow tasks. A human in the loop on any consequential action. An audit trail. The working assumption that the model's output is untrusted until it is verified.

The paper is on arXiv as 2603.12277v6, with a poster at ICML 2026 in July. The full model set tested for the 60% figure, the role-probe methodology, and the boundary conditions on the "fundamentally unsolvable" claim live in the body of the preprint, which is not fully public yet. The reader's working posture, the one the authors say follows from the architecture, is the one to carry into the next jailbreak headline: ask whether the vendor moved the wall, or just relocated it. The mechanism predicts the same fight, in different sentences, for as long as LLMs keep identifying their own voice by ear.