ROBBRIEF
VLAff, a vision language model trained on 204,000 episodes of human object interaction, predicts where to touch, how to grasp, and how to move from a single image.
A research group has proposed a foundation model that learns three different robot-perception skills from the same egocentric video. The paper, VLAff (arXiv 2608.05215), unifies three classic questions in manipulation: where to touch an object, how to grasp it, and the trajectory to follow afterward. The authors describe it as the first peer-reviewed paper to fold all three into a single vision-language model with three output heads.
The model is trained on a new dataset, EgoAffordance: roughly 204,000 episodes of humans manipulating objects, hand-labeled with about 5.6 million "where to interact" regions and 11.6 million grasp and trajectory labels. To turn those labels into executable robot actions, the pipeline first reconstructs 3D structure from each video using Structure-from-Motion and recovers a hand mesh (the 3D shape of a person's hand) to anchor the grasp. The full method is on arXiv; the project page hosts code and demos.
The authors report the best visual-affordance-prediction scores on the paper's own benchmark and apply the model to zero-shot manipulation, meaning a robot running a task it was not explicitly trained on, and to affordance-guided learning on real hardware. The catch: those results are scoped to the paper's evaluation suite, so generalization beyond is unverified. The work is camera-ready for IROS 2026.