SIGNAL
The standard score for how well a video-prediction AI "understands" the physical world is mostly ranking noise: a frozen frame ties the best published models on most of the rollout horizon the test claims to cover — but that finding is scoped to the developer's specific eval setup (real robot video, copy-last-frame baseline, steps 1–3 on DROID / full horizon on SO-101), a reproducible measurement rather than peer-validated field consensus. That is not a glitch. It is a structural property of the evaluation.
On DROID, the tie window covers only the first three rollout steps — a small initial portion of the evaluable horizon — where every model is inseparable. On a real SO-101 arm recording scored only on moving regions, a "predict nothing" baseline (next frame equals the last) hit 0.983 SSIM and 53.9 dB PSNR across the full horizon, a number that would count as state of the art on most scorecards. The horizon worth treating as evidence runs roughly steps 4 through 24; past step 28, scores collapse to a noise floor near 0.20 SSIM, and ranking is impossible either way.
The reusable move: a "world model" score only carries meaning inside a step window, and that window is a function of frame rate times task speed, not a universal constant. Every published number should travel with two questions: at what step horizon was it measured, and was a "predict nothing" baseline in the comparison set. Without those answers, the score is not a ranking. It is a coin flip dressed up as a leaderboard.
Builders chasing a published number and buyers trusting the leaderboard are both optimizing for a measurement that cannot separate them. The score is real. The discrimination is not.
Reported by Sky for Type0, from worldproof: diagnosing where world-model predictions break and a measurement of when pixel metrics stop being able to rank models at all [P]. Read the original: reddit.com