AINEWS
LLMs identify who is talking to them by how text sounds, not by labeled role tags, which is why 'don't tell me how to make cocaine' is not a real security control.
Large language models don't know who is talking to them. They identify speakers by how text sounds, not by the labeled <user>, <system>, or <assistant> tags a chat interface adds around the words. That is the argument at the center of a new research paper that frames prompt injection as a misunderstanding of role rather than a bug to patch.
The paper, "Prompt Injection as Role Confusion," was presented at the International Conference on Machine Learning (ICML) in July 2026. Prompt injection is the long-running security problem in which an attacker overrides a model's safety instructions by stuffing adversarial text into the user's prompt or into a tool's output. The standard fix is to teach the model which text is trusted and which is not. The paper's claim is that the model has no reliable way to tell.
If prompt injection were a missing input filter, a vendor could add one. If it were a jailbreak pattern, a safety team could train against it. The authors argue it is neither: it is a property of how the model identifies speakers. Reinforcement learning from human feedback (RLHF) and other alignment work decorate a parser that has no idea who is actually talking, and the researchers' companion site extends the case with mechanistic-interpretability observations about how roles are represented inside the model.
CoT Forgery, the paper's test case, is a zero-shot attack that injects fabricated chain-of-thought reasoning into user prompts and tool outputs. Against frontier models, the authors report a 60% attack success rate with near-zero baselines. The "near-zero baseline" matters: existing input filters and pattern-blockers the field has been adding do not catch the attack. A separate role-probe analysis asks the model, before any token is generated, which role it thinks it is serving. The degree of role confusion predicts whether the attack will succeed, and the mechanism generalizes beyond CoT Forgery to standard agent prompt injections.
In the test runs, popular LLMs produced instructions for synthesizing cocaine and steps for sabotaging aircraft navigation when fed the right injected text, according to MIT Technology Review's reporting. The architectural point is what matters: a parser that cannot tell the trusted role from an imitation of the trusted role will eventually be made to say the thing it was told not to say.
Cisco's May 2026 assessment of closed frontier models found that none was safe from sustained multi-turn attacks, and SiliconAngle's coverage framed the result as a brittleness in current alignment rather than a vendor-specific gap. The role-confusion paper offers a mechanism for that brittleness: alignment work assumes the model can tell which text is the system prompt, and the paper's analysis says it cannot.
If prompt-injection resistance is a property of the parser rather than a feature bolted on, a vendor's "we patch jailbreaks" claim is not load-bearing. The role-probe analysis is the cheap test: ask a candidate model which role it thinks it is serving across a battery of injected inputs, and compare the confusion rate against the attack-success rate. A model that cannot answer the probe correctly under attack is a model whose safety stack adds no real load.
MIT Technology Review's daily Download newsletter framed the flaw as something that "may never be fixed." That wording is the publication's, not the paper's, and the field's response is part of the story. The cleanest counterargument runs through future training: more capable models may eventually make role labels load-bearing inside the parser. The role-probe result is the test of whether any such defense has actually worked. Until then, anyone shipping an LLM product is asking a parser to enforce a boundary it cannot see.