World Models That Predict the Screen Can Still Miss the Jump

World models that ace next-frame prediction can still fail at the first jump. That gap is the field's quiet problem, and Bai's postmortem points to the loss function as the reason: a predictive architecture is rewarded for being right about the next screen, while a planner is rewarded for ending in the right state. Those are not the same objective, and no amount of pixel fidelity closes the distance. Move the goal one screen further and the predictor stays calm; the agent falls in the pit.

Benjamin Bai's LeMario postmortem is the cleanest public cut at that seam. A from-scratch reproduction of Yann LeCun's Joint-Embedding Predictive Architecture — four stacked frames compressed to 192-dim latents, actions injected through AdaLN-Zero transformer blocks, trained on NES Super Mario Bros — nails nearby goals to within two to five pixels. Move the goal image one screen further and the same model cannot clear the first major obstacle. As Bai puts it on the project page, the model had learned to predict the game, but that did not mean it had learned how to make progress through it.

The portable read: pixel-prediction success is not gameplay progress, and the gap appears structural rather than a tuning problem. Until the loss function pays for the terminal condition, "the model learned the game" will keep meaning two different things.