A Famous Reinforcement-Learning Architecture Gets Stronger for the Wrong Reason

A reinforcement-learning architecture that researchers have spent a decade adding components to does not get stronger because the new components are good. It gets stronger because they are unlikely to fail in the same place at the same time. That is the conclusion of a new arXiv preprint on option-critic, a long-studied way of breaking an agent's decision-making into named sub-policies called options. The agent can switch into and out of those options as it works through a task.

The architecture still gains from added options. The curve of task performance as more options are introduced is unchanged when the authors force a specific simplification on the architecture. What changes is the field's working theory of why the architecture gains from added pieces. It is not because each new option is better. It is because the chance that all of them fail in the same state drops sharply as their number grows.

The paper reports three named findings, each of which reframes a piece of the architecture.

The termination rule option-critic learns, a small decision procedure that decides when an agent should stop running the current option and pick a new one, is essentially vestigial. In a regime where the termination test and the option-picking policy read the same learned values, the test fires at every step. The learned rule collapses to always terminate, and that simpler policy reproduces the same option-count curve. Where the rule does get to explore, the authors bound its regret at Ω(T), a worst-case growth rate that scales linearly with the number of decisions, while always-terminating holds to O(log T), a much slower logarithmic rate. The more complex rule is the worse one. The authors write that the contribution of the learned termination test "is nothing."

The second finding is what the authors call policy necrosis. Inside an option, the policy, the rule mapping states to actions, barely explores. A state locks onto the first action that looked good and never updates again. The authors give a state-level test for this: if a state's value estimates are consistent with the same action across perturbations, it is necrotic. In a typical option in their experiments, three fifths of states are necrotic. Restoring exploration to those states repairs them, and a single option then solves the task. That is the falsifier: the redundancy dividend only exists as long as necrosis is real and prevalent.

The third finding is the redundancy dividend itself. Extra options do not improve any individual option. What falls is the chance that all of them fail in the same state, from 59% to 4%, as options are added. Performance follows that joint quantity, not the quality of any single option. The architecture's apparent gain is a coverage effect, not a competence effect.

The paper is a preprint, not peer-reviewed, and the mechanism claims rest on the authors' own theory and experiments. The redundancy claim is, however, falsifiable. A regime where intra-option exploration is repaired should collapse the dividend, and the paper reports that collapse. That is a sharper prediction than most architecture critiques make.

For practitioners who work with option-critic or related hierarchical reinforcement-learning systems, the paper offers two concrete things. The first is a state-level diagnostic: the necrosis test tells you which states have stopped learning. The second is a different optimization target. Instead of trying to make each option better, the right move may be to ensure that no two options share the same failure states. The field now has a name for a failure mode it had been measuring without a vocabulary, and a different thing to optimize.