AINEWS
A new peer reviewed survey accepted to IJCAI 2026 (the International Joint Conference on Artificial Intelligence) argues the field's four approaches to teaching robots from human video are really four bets on the same missing translation layer
Robots could learn manipulation skills from the vast ocean of human how-to videos on YouTube — if researchers could only figure out how to translate human hand movements into robot control signals. A new survey maps four competing approaches to that translation problem, and argues they are all describing the same missing bridge.
The survey — From Human Videos to Robot Manipulation: A Survey on Scalable Vision-Language-Action Learning with Human-Centric Data — was accepted to the IJCAI 2026 Survey Track, authored by researchers from Tsinghua University, Hong Kong University of Science and Technology, and Microsoft Research Asia. First author Zhiyuan.
In language models, scale has been the defining variable: more text data reliably yields better capabilities. The same logic has driven progress in autonomous driving, where logged road hours accumulate into better models. In robotics, the bottleneck is harsher. Every training example requires a physical robot to perform an action, record the trajectory, and do it again at scale. The resulting datasets — including Open X-Embodiment and DROID — are large by robotics standards, but small by the measures that drove language model breakthroughs.
Human video offers a potential escape from this data constraint. People constantly perform manipulation tasks — grasping, placing, opening, closing, tool use — and cameras capture much of it. YouTube alone hosts millions of narrated instructional clips. The HowTo100M dataset contains roughly 136 million video clips sourced from approximately 1.2 million narrated instructional YouTube videos, according to the original dataset paper and the official ENS dataset page. The Ego4D dataset has accumulated thousands of hours of first-person video; the exact figure cited in the interview (more than 3,600 hours) should be verified against official Ego4D documentation before publication.
The appeal is clear, as the source described it: strap on smart glasses, walk around, and everyone becomes a data source, at very low cost. Human video could scale from tens of thousands of hours to millions.
But it cannot be fed directly to a robot.
The survey identifies three incompatibilities: video contains no robot-executable action labels; it lacks proprioceptive information such as joint angles and end-effector state; and human hand movements are kinematically incompatible with robot control interfaces. These three mismatches define the central problem the field must solve: once human video enters the training pipeline, what should it become?
The survey's core contribution is a unified classification framework. Over the past two years, different research groups have taken different approaches to making robots learn from human video — and they look on the surface like distinct lines of work. The survey's argument is that they are all solving the same underlying problem: choosing an intermediate representation layer between human video and robot action, and building a bridge at that layer. The four routes differ only in where they place the bridge's foundation.
Latent action compresses action information into a latent space, letting the model discover actions from video without explicit labels. LAPA, published at ICLR 2025, is the representative system, learning latent action representations from human video that are usable for robot control. This approach drew significant follow-on work through late 2024 and early 2025, with several subsequent VLA systems incorporating similar label-free learning ideas. Its limitation is interpretability: "Your supervision signal lives inside a latent space — at intermediate stages of training, you can only look at the training loss," as the source put it. The survey notes an open problem: whether highly compressed discrete tokens can adequately represent the rich structure of high-degree-of-freedom dexterous manipulation.
Explicit 2D extracts visible, interpretable action cues directly from video frames: hand keypoint coordinates, object bounding boxes, point trajectories, and 2D pose sequences. These can be extracted with existing vision models, without additional 3D sensors. ATM uses dense point trajectories as dense motion supervision; Magma renders these as visual prompts embedded in video frames; Gemini Robotics and related work explore 2D spatial information for action grounding. The limitation is that manipulation is inherently three-dimensional — 2D cues lack depth, and occlusion breaks tracking. In the survey's framing, 2D serves as an auxiliary supervision signal, not a standalone route.
Explicit 3D reconstructs complete hand motion in three-dimensional space from video, then uses that as a robot learning signal. The core enabling tool is the MANO model — a low-parameter representation of full hand pose that acts as a "standard language" between human video and robot control. EgoVLA, H-RDT, Being-H0, and VITRA all operate in this framework, extending data scale and coverage. The payoff is higher alignment quality: 3D directly provides the coordinate-space representations robots need. The cost is higher annotation expense, lower 3D reconstruction accuracy on in-the-wild video compared to 2D supervision, and the added difficulty of inferring 3D structure from 2D images. In the source's assessment, explicit 3D is currently "100% needed as auxiliary supervision" inside VLA training — not an independent main route, but an essential component for accurate spatial perception.
World models learn the underlying physical rules of the environment, predicting future states from current observations. This route remains open and active, with no consensus on how to ground specific actions within a learned world model.
The survey's framing deliberately de-emphasizes the routes-as-competition narrative. Instead, it positions the four approaches as four different placements of the same missing component: a representation bridge between human video and robot action. No approach has yet demonstrated that bridge in a way that transfers broadly across tasks and hardware. The paper is, in this sense, less a survey of four solutions than a diagnosis of one unsolved problem.
The paper argues it is the first to explicitly frame the four approaches as one problem rather than four — a taxonomic judgment that reframes the competitive narrative as a coordination problem.
The IJCAI 2026 Survey Track acceptance means the paper will appear alongside other accepted surveys; it does not itself present a working bridge. The field's next data point will be whether the framework influences how subsequent work is designed — or whether the four routes continue to develop in parallel, each implicitly betting on a different foundation.