What a 'Move 37' in AI research would actually look like

Three researchers from major AI labs make a more disciplined bet against recursive self-improvement than the usual "AI isn't there yet" hedge. Their case runs like this: a real AI-improves-itself-into-AGI leap would have to look like Move 37 in research itself, a move so surprising it changed what the experts thought was even possible in the game. By that test, recent frontier gains do not read as RSI. The missing piece is not compute or data but a mechanism for the kind of move that would count.

The argument comes from a long Dwarkesh Podcast episode with John Schulman, Beren Millidge, and Charlie O'Neill, all currently at "openish" labs in the loose sense host Dwarkesh Patel uses for researchers whose work is published but whose employers vary. The conversation ranges from Chinese-lab progress to automated AI researchers to long-horizon reinforcement learning. The throughline is a single skeptical question: if AI is improving itself, where is the equivalent of Move 37?

Move 37 is the move AlphaGo played against Lee Sedol in game two of their 2016 match. A human watching the position would not have played it. Commentators thought the machine had blundered. Two-thirds of the way through the game it was clear AlphaGo had seen something the rest of the field had not, and the move reset what professional Go players thought was possible at the top level. The phrase now travels as shorthand for a qualitative leap inside a system already doing useful work.

That is the test the researchers want applied to claims about AI improving itself. "Move 37" in their usage asks whether a new result resets the field or simply flatters an existing benchmark. "Entropy collapse" is the inverse condition they name: when a model's outputs converge on the high-probability modes of its training distribution, the next round of self-improvement mostly polishes what is already there and does not open a new part of the board.

Reinforcement learning, which lets a model learn from trial-and-error feedback rather than from labeled examples, is finally delivering on tasks it struggled with for years, and the researchers treat that as the most interesting development in the current cycle. Long-horizon RL has a structural problem: the further out a goal sits, the harder it is to assign credit to the steps that mattered, and the easier it is for the optimizer to settle into a local maximum that scores well without changing the underlying picture. One commenter on the Hacker News thread framing the episode put it plainly: "we have recursive self improvement that can find local maxima, I'm not seeing really strong evidence of unguided RSI that finds the true optimal."

A second constraint sits in the sim-to-real gap, the distance between a system that performs inside a controlled training environment and one that performs in the open world where its training distribution thins out. The researchers treat the gap as load-bearing. Any RSI story has to clear it, because the surface a model is improving on is the surface it was trained on, and a system that is not yet generalizing cleanly across that surface is not, in their framing, in a position to improve itself across it either.

The strongest counterargument the researchers take seriously is the one they are least able to dismiss. RL is finally working on tasks that resisted it for a decade, and the gains keep showing up in places (code, math, multi-step tool use) that look qualitatively different from the next-token-prediction improvements that defined the previous cycle. If the qualitative difference is real, a "Move 37" reading of the next year is not crazy. The researchers' reply is that the gains are still inside the regime they have a working model of. The question is what happens when the next gain requires a move no one currently has a working model of, and the answer is that we will only know by watching for it.

A working test the reader can carry into the next breakthrough headline: does this claimed leap look like optimization polish, or does it look like a move that resets what experts in the field thought was even possible. The first is the regime the researchers can already see. The second is the one that would actually change the RSI story.