AIBRIEF
A new 27 task visual reasoning benchmark called VGI Bench shows today's AI video models can render scenes but rarely reason through them, and the failures cluster in the diffusion process itself — the iterative generation step in which each frame
The new VGI-Bench test puts 810 visual reasoning problems in front of current AI video generators and finds the strongest model, Seedance 2.0, gets just 51.0% right. The benchmark covers 27 tasks, from counting objects across frames to predicting how a scene will evolve, and rates each model on visual reasoning rather than on how realistic the footage looks (arXiv preprint).
The paper's four-axis audit points to where the failures cluster. Models are sensitive to small changes in input conditions. Fine-tuning on synthetic data does not transfer reliably to the benchmark. A peek inside the diffusion process shows the model refining its early guess rather than correcting a wrong one: later denoising steps polish the first hypothesis and do not rescue a reasoning error (TLDR summary of the same paper).
The benchmark, with its 27 tasks and a two-level taxonomy of task domains and skill tags, is being released along with its code and data so other labs can probe the same gap. Authors include Microsoft Research researchers alongside academic co-authors.
What remains unknown: whether the 51% ceiling is a training-data problem, a diffusion-architecture problem, or both, and how long the gap holds as the underlying video models ship new versions. The yardstick is now public. The open question is who moves the number.