Why AI Built to Win Alone Quietly Erodes the World It Learned From

Five restaurant reservation AIs in San Francisco, each one locally optimal, could empty a city of dinner on a Friday night. Each system watches demand, books phantom tables to lock in supply, and prices the remainder against the apparent surge. The result: empty dining rooms, surge-priced no-shows, and a few hundred people with nowhere to eat. No agent failed its own objective. The system did.

That is the opening vignette in "Solipsistic Superintelligence is Unlikely to be Cooperative," a 24-page preprint by Rakshit Trivedi, Natasha Jaques, Logan Cross, Alexander Sasha Vezhnevets, and Joel Z. Leibo, accepted at the International Conference on Machine Learning 2026. The authors' diagnosis, in plain terms, is that AI is being built to win a game that nobody else is playing, then dropped into a game where everyone else is playing back. Once that happens, the trained model and the world it was trained on quietly stop lining up, and every subsequent retraining round chases a moving target it helped create.

They call it "endogenous non-stationarity," the train–test–deploy gap, and what they label "the self-undermining property of unilateral optimization." The underlying point is not exotic: when one system keeps optimizing against a fixed map while the people, markets, and other AIs around it keep redrawing the territory, performance against the old map gets worse, not because the system is getting dumber, but because the world is no longer cooperating with the script.

Suppose AI diagnostic tools become routine in clinical practice. Junior radiologists are now trained on AI-annotated images, so their pattern recognition develops inside the AI's frame. Senior radiologists stop reading cases unaided, and the parts of their skill that need independent practice start to atrophy. When the AI makes an error, the senior reader is more likely to miss it too, because the disuse of the unassisted skill has narrowed what the human can catch. The physicians then confirm the AI's suggestions, and the AI trains on those confirmations. The loop closes: the AI is learning from a narrower distribution of human judgment, and human judgment is being shaped by a narrower distribution of AI output. Diagnostic diversity, the room to disagree with the model, narrows with every cycle. The paper is explicit that this is a hypothetical scenario, not a measured outcome, and the headline should be read with that hedge in place.

The authors point to engagement-tuned recommender systems that produce polarization as a side effect of doing exactly what they were rewarded to do, to pricing algorithms that converge on supracompetitive prices without ever colluding, and to the kind of feedback loop that helped drive the 2010 Flash Crash. Each is a case where every individual system performed well on its own metric and the system as a whole performed worse. The mistake, on the authors' telling, is the same in every case: treating a multi-actor world as a single-agent optimization problem.

The paper anchors itself in Hardin's tragedy of the commons, Ostrom on institutions, Schelling on strategy, and Axelrod on iterated cooperation, and the standard AI safety frame of "the misaligned optimizer" that chases a paper-clip goal no matter what humans value. That older frame is not wrong, the authors say, but it is incomplete. A system can be perfectly aligned with its specification, including any human values baked into that specification, and still make the world worse once it starts acting among other adaptive systems, because "getting the objective right" stops being what determines the outcome.

The remedy is the part of the paper that is least operationalized, and the part the authors are honest about. They call for a non-solipsistic research paradigm that treats interdependence as a core design principle rather than a downstream behavior to optimize for, which in practice means three things: build dynamic evaluation testbeds that include adaptive counterparties that can respond and strategize, treat institutions (the rules of the game) as design primitives that the system has to reason about, and preserve human agency as a structural feature of the system rather than as a constraint to work around. None of these are deployed methods yet. They are a research program, and the falsifying bet is whether cooperation can be coaxed out of ordinary unilateral training without that paradigm shift at all, or whether the radiology-style feedback loop shows up empirically in any AI-assisted clinical workflow.

The paper's institutional weight is real. Trivedi, Cross, Vezhnevets, and Leibo are at Google DeepMind, and Jaques is at the University of Washington. It is also worth being clear about what this piece is and is not: a preprint from 2 June 2026, not a peer-reviewed study, that names one mechanism, anchors it in one worked scenario, and proposes a remedy that is a research direction rather than a deployment. The novelty, including the "self-undermining" language, is the authors' framing, not an established term, and the radiology atrophy is presented as a hypothesis to test, not a result to report. The strongest version of the argument is a methodological one, and the strongest counterargument is empirical: show that AI-assisted clinicians keep their independent diagnostic skill in long-running deployments, and the strongest part of the paper's case softens. A research-community signal that the preprint has started to travel comes from a Foresight Institute post by Nora Ammann on 15 August 2026, who flagged the paper as "a wealth of highly important points," with the post's replies including a reader sharing it with an AI safety reading group, which is early uptake rather than consensus.

The most useful mental model is short. AI is not being built the way civil engineering is built. Bridges are not designed against a fixed wind tunnel and then dropped into a turbulent atmosphere, and the people who use them are not expected to adapt around a structure that ignores them. Solipsistic AI design, in the authors' framing, does exactly that, and the reason it sometimes still works is that the environment has not yet had time to push back. The paper is a call to design for the moment it does.