A Robot Sees in 3D Without 3D Labels. The Trick Is in the 2D Stream.

A robot is set down in a room it has never seen and asked to find the chair, the cup on the table, and a partially open drawer. It has no catalog of chair-shaped objects, no 3D point clouds labeled by hand. The objects on screen still need to come out as discrete, persistent things rather than a soup of colored voxels. That is the problem a Seoul National University team takes on in CDIS, a framework presented at IROS 2025 and posted to arXiv in late July (arXiv:2607.17778).

"Zero-shot" in this setting means the system has never been trained on 3D labels for those object categories. "Instance segmentation" means assigning each pixel (and, by extension, each point in a 3D scan) a label that ties it to one specific object instance, not just "object-ish region." "Class-agnostic" means it does not need to know the object's name. CDIS threads all three: 3D instance labels for objects the system has never been trained to recognize, and without naming them.

CDIS takes an RGB-D stream (ordinary color video plus per-pixel depth) and tracks 2D instance masks across frames. It then links each tracked 2D region to a cluster of 3D points called a "superpoint," a small, geometry-coherent patch of 3D points the system treats as a unit. The trick is the loop. The 2D track tells the 3D side which superpoints belong to the same object, and the 3D superpoints push back when a tracked mask drifts because of depth noise, occlusion, or a reflective surface throwing off the depth sensor. The output is a 3D instance map that stays stable as the camera moves, instead of fragmenting into a stack of half-objects (paper PDF).

That is why the "no 3D training" line, which will almost certainly lead incoming coverage, leaves the mechanism out. The system is not learning nothing. It is leaning on a different supervision signal: temporal consistency in 2D. The trained part of the pipeline is the 2D mask tracker; the 3D side is geometry plus association rules. The contribution is the feedback loop that lets those two halves correct each other across the camera trajectory. Wire headlines will read "no 3D training needed." The honest read is "no 3D labels needed, but a continuous 2D track across frames is doing the heavy lifting" (arXiv HTML).

The paper positions CDIS against projection-based zero-shot baselines, including SAM3D-style pipelines that lift 2D masks from a foundation model into 3D and CropFormer-based mask merging. The reported claim is higher accuracy and instance consistency than those baselines on standard 3D instance segmentation benchmarks, while staying efficient enough for real robot use. The authors, Juno Kim, Hye-Jung Yoon, Yesol Park, and Byoung-Tak Zhang of Seoul National University's Interdisciplinary Program in AI (with the first three contributing equally), published the work in the IROS 2025 proceedings (pp. 4269–4274) before the arXiv posting on July 20, 2026 (IROS 2025 DOI).

A 2D-to-3D feedback loop depends on temporal continuity. If the camera moves too little, the track does not get enough views to vote out a bad segmentation; if surfaces are highly reflective, transparent, or heavily occluded, the 2D mask itself wobbles and the 3D side inherits that wobble. Calibration drift across long sessions is another boundary the paper flags. CDIS is a different engineering bet (temporal 2D consistency standing in for 3D supervision), not a replacement for cases where rich 3D ground truth actually exists.

The interesting question is not whether CDIS beats SAM3D-style baselines on the usual 3D instance benchmarks (the paper claims it does), but whether the 2D-anchored loop holds up outside curated indoor scans, on the cluttered, partially observed rooms a service robot would actually face. The authors have not released a public video or robot-system demo alongside the paper, and the Korean government funding line (IITP, NRF, KEIT) signals this is part of a broader Seoul National perception stack rather than a standalone product. Watch the next IROS and ICRA proceedings for replication, and for any independent robotics-perception group stress-testing the loop on mobile platforms in the wild.