Catching the One Bad Field in an AI's Command

An AI assistant has been told to draft and send an email. The body of the email was written from untrusted web content and contains a hidden instruction. The recipient, the account, and the subject line were all fields the model itself filled in. The right question for the assistant is not whether the email is safe on average. It is whether any of the high-stakes fields (the recipient, the account number, the command) has been tampered with.

A new preprint from six researchers, Beyond Aggregate Risk: Role-Stratified Conformal Risk Control for LLM Tool Calls (arXiv 2607.24343), takes that question literally. Instead of certifying an AI agent's tool call as a single object with one risk score, the method breaks the call into its named argument slots (recipient, account, body, command line, credential) and gives each slot its own statistical threshold and risk budget. Roles that show up often enough in the calibration data are certified directly with a finite-sample guarantee. Rarer roles are pooled under a shared budget so the math still holds.

The motivation is a real failure mode. When risk is measured over the whole action, a clean low-stakes field can hide a dirty high-stakes one. An attacker who plants a prompt in an email body does not need the body to survive every safety filter; they need the recipient field to pass. Under aggregate-only certification, the safety budget for the action is the same regardless of which field carried the attack, and a single high-stakes failure can be diluted by clean fields around it. The paper formalizes that intuition: an aggregate-only certificate that wants to guarantee role-specific risk at level α has to use an effective budget of α·p_r, where p_r is the share of times the role appears in the data. For a role that fires 1% of the time, that is a budget a hundred times tighter than the headline number.

The empirical work is broad enough to take the claim seriously without overstating it. The team tested the method on two benchmarks built around exactly this attack surface, AgentDojo and InjecAgent, across six language models, and stressed it under model and attack transfer, detector noise, gradual drift, unseen tool suites, and adaptive attacks. The headline empirical finding is that the role-stratified method holds its per-role risk budget more consistently than aggregate certification across those conditions, and that the gap tracks the predicted "price of coarseness" of treating the whole action as one unit.

The paper is a preprint, not a peer-reviewed result, and the authors are careful about what they have and have not shown. The formal per-role guarantee holds under exchangeability, the assumption that calibration and deployment data are drawn from the same distribution, or after the system is recalibrated. Under frozen distribution shift, what holds is empirical compliance, not a formal certificate over arbitrary future drift. The method is also only as good as the per-field detectors underneath it. If a detector cannot tell a poisoned recipient from a clean one, role-stratified calibration will certify a bad estimate more finely rather than fixing it. Pooled certification for rarer roles is a partial answer, not a final one.

For readers, the practical takeaway is a question to ask the next time an AI agent claims a safety number. The number is only useful if you know the unit it is certifying, the whole command or each field separately, and which fields are sharing that budget. A system that certifies the whole action as one is making a claim about averages that an attacker does not have to respect. The preprint's argument is that the field, not the action, is the unit that has to be guaranteed, and the next step is watching whether the broader agent-safety literature treats it that way.