A robot arm that measures its own progress, then lands on unseen targets without retraining

What does it look like when a robot is actually making progress toward a target, as opposed to just moving plausibly? Most vision-guided robotic arms answer that question with a hand-coded error metric: a camera measures the gap to the goal, the controller shrinks it, and a human defined what "gap" and "shrink" mean.

A new arXiv preprint called WM-VS: Progress-Aligned World Models for Closed-Loop Visual Servoing flips that setup. Instead of teaching a robot what to do, it teaches a world model, the system's internal simulator of how the camera view changes when the arm moves, to value getting closer, not just looking plausible. The training signal rewards actions whose predicted future state sits inside a smaller, signed version of the target coordinate, and the ablation shows what happens when that signal is removed: the final-frame success rate drops from 25 of 30 to 8 of 30.

The work sits inside "visual servoing," a long-standing robotics category in which a camera, not a pre-mapped coordinate system, drives the arm toward a target. Closed-loop versions react to what the camera sees each frame rather than following a fixed path, and "eye-to-hand" means the camera sits beside the arm looking at the workspace, not mounted on the gripper. The arm used here has 7 degrees of freedom, which means seven independently controllable joints, making it flexible enough to reach a small target from many angles.

WM-VS splits the system into two stages. In Stage 1, the researchers align the world model's action-conditioned latent transitions with a signed 4D servo coordinate covering translation, scale, and in-plane rotation. The coordinate is computed offline from target-region correspondences using DINOv2, a self-supervised vision model that finds matching points between images without labels. The world model learns to predict latent states that move in the right direction along that coordinate, not just latent states that look like the next frame.

Stage 2 freezes that world model and trains a reactive joint-velocity policy on top of it. The policy uses action imitation, consequence supervision, and short imagined rollouts in the world model that favor error contraction. Deployment is RGB-only and reactive, with no online trajectory optimization; the arm reads the camera each frame and reacts, the way a human would nudge a reaching motion when it is off-course.

The team evaluated the system on a real 7-DoF eye-to-hand arm. In all 30 trials, the corner root-mean-square error, a standard measure of how far the camera's view of the target is from where it should be, was no larger than 10% of its starting value. In 25 of those 30 trials (83.33%), the arm held that criterion through the final valid frame. (A full HTML version of the paper is also available.)

The ablation is the load-bearing number. Remove the future-error alignment from the world model, and the final-frame retention drops to 26.67% (8 of 30 trials). The rest of the framework, on its own, does not hold the result up.

A second check comes from a sensor the system never sees during training. The learned progress signal agrees with a corner error measured by AprilTag, a fiducial marker designed for precise camera-based pose estimation, at a mean Spearman correlation of 0.8778. Spearman's rho is a rank-correlation measure that captures how well two orderings agree, on a scale from -1 to 1. The system is not just optimizing an internal number; it tracks a real, external metric it was never told to track.

The transfer result is the second piece worth reading closely. On two 3D targets the system had never seen, translation errors fell by 86.48% and 90.27%, and rotation errors fell by 70.01% and 65.70%, with no retraining. The progress signal, learned against a specific coordinate, still pointed the policy in the right direction on objects outside the training set.

The system is one 7-DoF arm in a single lab setup. The targets are fiducial-marked, with AprilTag corners rather than free-form household objects, and the progress coordinate is defined relative to a target region the system has already identified offline. None of the numbers above imply a general-purpose household robot. The code and data, the authors say, will be released as open source, and the next test is whether the same training signal holds up on other arms, cameras, and marker-free targets.

The watch item is whether the signed 4D coordinate itself generalizes. Right now it is computed offline from a known target region. A version that learns the coordinate, or extends it to rotation in depth rather than only in-plane, would push the result from a focused lab advance to a candidate building block for vision-guided reaching more broadly.