When AI Agents Work in Teams, Letting Each One Pick Its Own Best Move Can Backfire

A new arXiv preprint from Sen Li and colleagues proves that when each agent in a cooperative AI team picks the move that looks best for itself, the team's joint behavior can end up strictly worse than every policy in everyone's shared library, a failure with no single-agent counterpart.

In single-agent reinforcement learning, successor features with generalized policy improvement (SF-GPI) recombines a learned policy library into a policy for a new objective with a "never-worse" guarantee. The multi-agent world kept the recipe, the authors argue, but lost the proof.

The mechanism: when a teammate switches which library policy it follows, the environment each agent faces changes with it, invalidating the value estimates the per-agent rule relies on. The paper's only unconditionally safe option is synchronized composition, moving the whole team to one jointly trained policy, but that cannot serve objectives that assign different goals to different agents, an honest caveat the authors flag.

Their proposed research direction is MA-USFA: a hierarchical structure with lower-layer universal successor feature approximators conditioning on teammates' objectives, plus an upper composer that supplies the cross-agent correction a per-agent value cannot represent. Trained once over a distribution of objectives, applied at deployment without per-task adaptation. The paper points to fleet management and traffic signal control as motivating domains; it does not claim the problem is solved.