AGTNEWS
A new preprint argues most AI strategy benchmarks can't separate reasoning from recitation. Models hold up when they reason out loud and break when they must infer rules from play.
When a language model is asked to reason step by step about a strategic game, it usually stays internally consistent. When the same model has to figure out a hidden rule by watching an opponent play, that consistency collapses. A new preprint argues this gap is not a bug in the models. It is what most existing "AI strategy" benchmarks fail to measure.
The paper, Level-k Distinguishable Mechanisms for Evaluating Bounded Rationality in LLMs by Binchi Zhang and Atrisha Sarkar, landed on arXiv on August 21, 2026. It isolates a credibility problem at the center of every public claim that a model "reasons strategically" or "plays like a person." Most of those claims are tested on well-known games, the kind whose rules and common solutions sit in textbooks, lecture notes, and online strategy guides. A model that scores well on the All-pay auction or the Nash demand game could be reasoning through the situation, or it could be reciting the most common answer it saw in training.
Zhang and Sarkar's fix is a stricter standard. Their Definition 1, which they call the level-K distinguishability condition, requires a benchmark to be sensitive to how a model is asked to think. A benchmark passes the bar only if the answers a model produces when it reasons out loud differ from the answers it would produce by default. If a benchmark cannot tell those two answers apart, it cannot tell reasoning from recitation. Most familiar games cannot.
To put the standard into practice, the authors built a new game from scratch: Ring 11-20. It is a five-player directed cycle in which each player wins a 10-point bonus for undercutting the player to their left by exactly 3, plus a 3-point bonus for beating the player to their right by exactly 7. The structure has 10 distinct payoff values before it cycles, which is what makes it possible to recover a "level-K" signal: a measurable read on how many layers of "I think that you think that I think" the model is actually using. The authors also reworked three familiar games (the 11-20 money request, the All-pay auction, and the Nash demand game) until each passed the same bar.
The team then ran four unnamed large language models through all four games across ten levels of iterated reasoning. They read the models on two registers. The first is recursive reasoning, where the model writes out, step by step, what it believes and why. The second is an inductive trace, where the model only sees the opponent's past moves and has to reverse-engineer the rule.
The split was clean. Under recursive reasoning, the models held accurate strategic depth at every level, and the chain of thought they wrote lined up with the action they took. Under the inductive trace, accuracy fell sharply and did not fall evenly. The pattern of error was also specific. When models got the wrong answer, they got the wrong number of reasoning steps, not the wrong move. A model asked to think four moves ahead but stuck at three would pick the move that fits three-deep reasoning and miss the one that fits four. It computed correctly at whatever depth it chose; it just chose the wrong depth.
Whether a public claim about an AI's strategic ability holds up depends on which regime the test exercises. A benchmark that lets the model think out loud will look generous. A benchmark that asks the model to infer a strategy from observed play will look unforgiving, and the gap between them is the part the field has not been measuring.
The honest caveat: this is a two-day-old preprint, not a peer-reviewed paper. The four evaluated models are not named in the abstract, and the recursive-versus-inductive contrast is paper-internal, drawn across four game structures rather than tested in real multi-agent deployment. The strongest falsifier is whether the split survives contact with negotiation, trading, and other settings where AI agents actually have to read each other. The paper's contribution is a framework for asking the question more sharply, not a verdict on whether AI can or cannot strategize.