A paper from ACL 2026, a top AI conference, replaces Shannon entropy — a standard measure of output uncertainty — with a normalized version that flags the single outlier step in a long reasoning chain.
A long-horizon LLM agent, one asked to book a trip or orchestrate a multi-step API workflow, drifts in ways its own trainer cannot see. The reward arrives only at the end, so which intermediate step was wrong stays opaque. STAPO: Selective Trajectory-Aware Policy Optimization for LLM Agent Training, an ACL 2026 paper submitted to arxiv on 6 July 2026, targets that gap at the most local level. It replaces Shannon entropy with a normalized version that flags the single step where the agent has wandered from its own typical behavior, then corrects that step and only that step.
Its eleven authors, Qiuyi Qi, Tian Liang, Mutian Bao, Jinjian Zhang, Dongnan Liu, Wei Zhou, Linjian Mo, Ming Kong, Jie Liu, Feng Zhang, and Qiang Zhu, frame the failure mode as "trajectory neglect." On tasks with sparse delayed rewards, an agent that takes twenty coherent intermediate steps and one wrong one still ends up with a single terminal signal. Standard reinforcement learning treats every step the same and so cannot tell whether the failed outcome came from the eleventh step, the seventeenth, or none of them.
The normalized-entropy move measures an action's confidence against the agent's own mean behavior under the same state, replacing the standard Shannon-entropy score that treats each step in isolation. A step that looks normal in absolute terms but is unusually confident or unusually hedged for that agent gets flagged as an outlier. The paper calls the overall framework hierarchical group-based RL: a first pass locates outlier steps via normalized entropy, a second pass optimizes them through a joint mechanism that combines a trajectory-aware reward, or credit that follows the actual path the agent took, with a trajectory-independent penalty, a regularizer that does not depend on which trajectory succeeded.
GiGPO, the closest cousin in the lineage STAPO inherits, also targets step-level credit assignment but does so with group-in-group policy optimization across trajectories rather than by selecting individual outlier steps for treatment. STAPO's selection step, finding the step that drifted and treating only that step, is the architectural difference. The selection step is also the falsifier. In very short tasks where every step already looks like an outlier, the signal saturates; in domains where high entropy is the desired behavior, including creative writing and exploration, normalized entropy will wrongly flag correct steps as wrong.
STAPO is not the first attempt at trajectory-aware LLM agent training. The lineage it inherits starts with ReAct in 2022, which interleaved chain-of-thought reasoning with environment actions; Reflexion in 2023, which added verbal self-critique as a reinforcement signal; and Voyager in 2023, an open-ended embodied agent that maintained a skill library over time. Each added a layer of self-monitoring on top of the base policy. STAPO's contribution is narrower and more local. It does not give the agent a memory or a self-critic; it gives the trainer a sharper ruler.
Three caveats belong in the same paragraph. First, the paper is a same-day preprint. No third-party replication, no downstream adoption signals, and no peer reviews beyond the ACL 2026 acceptance header are visible yet. Second, whether STAPO ships with code, checkpoints, or evaluation harnesses is not visible from the arxiv metadata alone and should be confirmed before any "reproducible" or "open-source" phrasing is fair. Third, the exact baseline numbers, training compute, and model sizes used in the headline comparisons live in the paper body and appendix, not in the abstract; the arxiv preprint is the right place to read those claims before citing them.
The camera-ready version at ACL 2026 will carry the experimental table that today sits in the preprint's appendix. Whether normalized entropy generalizes beyond the settings its authors tested is what that table will show.