When a Coffee Mug Slides 5 Centimeters, Vision-Based Robot Policies Lose Most of Their Grip

Move a coffee mug 5 centimeters to the left on a kitchen counter and most of today's best robot-manipulation policies can't pick it up. Under a controlled stress test on LIBERO-Spatial, a standard simulation benchmark for object-moving tasks, four vision-based manipulation policies fell to at most 3 successes out of 20 once the target object shifted that distance. A small language model trained only on relative coordinates held its own much better, keeping 7 of 10 successes when the coordinate frame was relabeled. The gap between those two numbers is the point. "Coordinate invariance," the property of being robust to which axis the robot calls x, is not the same thing as "physical-layout generalization," the property of being robust to where the object actually sits. In a new arXiv preprint, the authors' own attempt to bridge that gap with "geometry conditioning," the practice of letting the model see explicit object and goal coordinates, did not cross it.

The paper adapts a 0.8-billion-parameter hybrid language model, a small language model (or SLM) repackaged as the brain of a robot, for simulated manipulation. The SLM is "embodied," meaning it takes in the robot's physical state and outputs actions rather than chatting. In their reported numbers, only about 6.2 million of the model's parameters are actually trained. They test six variants of how that SLM is given geometric information, on three LIBERO-Spatial tasks across three seeds and 540 held-out rollouts.

A baseline with no explicit object or goal geometry hits 24.4% success. A version whose recurrent decay gates, the model's internal memory of what changed between time steps, are conditioned on the actual geometric increments between the gripper and the object reaches 28.9%. The same gates, retrained after the geometric increments were shuffled, hit 36.7%, the highest of the six. A token adapter fed the same increments scores 27.8%, and the differences between the two "real geometry" conditions vary across seeds in a way the authors call inconclusive. A "token clock" condition that recodes time itself bottoms out at 11.1%, with one of its three seeds never converged.

The pattern is uncomfortable for anyone who hoped feeding explicit coordinates to a small language model would buy geometric reasoning for free. Under this recipe, the model trained on the right geometric signal underperforms the model trained on a shuffled one. The authors, to their credit, do not paper over that. Their reported conclusion is that training-time geometric alignment, in this setup, "does not provide a reliable advantage." That is a null result.

The robustness stress test is where the paper's second claim lands. When the coordinate frame, the convention for which direction counts as right or forward, is relabeled, the state-only relative-coordinate policy still completes 7 of 10 tasks. When the target object is shifted 5 cm in the world, every one of the four tested vision-based policies falls to 3 of 20 or worse. Vision-based policies are sensitive to where the object physically sits. Coordinate-based policies are sensitive to which frame the robot uses to describe the world. Neither is robust to the other's failure mode. The paper puts both failures on the same page, in the same units, with the same evaluation protocol.

The paper is a preprint, not a peer-reviewed paper. The evaluation surface is narrow: three tasks, one architecture, 540 rollouts. The token-clock null rests on a single non-convergent seed, and the headline 5 cm figure depends on the choice of which policies and which perturbation step. None of this is a verdict on whether language models belong in robotics. The authors are running a controlled study, not arguing the field should pack up.

"Does this model know geometry?" usually gets one of two answers: yes, it was trained on coordinates, or no, it's just pixels. The paper draws a third line. Robustness to coordinate-frame conventions and robustness to physical layout are different properties, and a model that tests well on one is not automatically robust to the other. The 5 cm mug shift and the relabeled frame are different kinds of fragility, and the next round of "embodied" results will be more useful if they report both.

The authors have released their episode records, seed-level analyses, and figure-generation code, so other groups can re-run the same stress test on their own policies. The practical watch item is whether anyone else's 7B, 70B, or vision-language action model survives the same 5 cm shift, and whether a different "geometry conditioning" recipe closes the gap. Until then, the mug is still in charge.