MIT's three-agent AI builds the rooms where robots rehearse their jobs

Training a robot on a new task used to mean hand-building every practice room. Researchers at MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) and the Toyota Research Institute now say three cooperating AI agents can do that worldbuilding in a single pipeline.

The system, called SceneSmith, splits the work across a "designer" that drafts scene elements, a "critic" that judges realism, and an "orchestrator" that manages the workflow. Each agent calls a vision-language model; the team identifies it as GPT-5.2 for the spatial knowledge that tells the system what a restaurant, bedroom, or hotel lobby actually looks like.

The result is a prompt-driven generator that the lab says is more realistic and detailed than prior scene-synthesis systems, producing the varied indoor environments where household robots would be expected to operate.

Two limits stay on the surface. Every generated room is filtered through one closed third-party model, so the realism ceiling is set by a product the team does not control. And richer simulated practice has not yet been shown to translate into better physical-world performance, a gap the researchers frame as the next question to answer.

Russ Tedrake has called simulation the natural training ground for robots and identified scene-content diversity, not physics, as the remaining bottleneck. A project page with demos and an arXiv preprint accompany the release.