Teaching an Agent to Sketch One Part at a Time
A team from TTI-Chicago, the University of Chicago, and MIT CSAIL has published a paper introducing a new approach to vector sketch generation that teaches a multimodal language model to build drawings one semantic part at a time -- and crucially, trains it to care about the process, not just the result.
The paper, arXiv:2603.19500, submitted March 19, describes what the authors call a multi-turn process-reward reinforcement learning approach, applied after supervised fine-tuning on a newly built dataset. The technique belongs to the GRPO family (Group Relative Policy Optimization), and the distinguishing claim is architectural: rather than rewarding the model when it produces a correct final sketch, the system rewards it at each intermediate step. Every part-sketch state, not just the last one, gets evaluated.
This matters because of how vector sketches work. A sketch of a horse is not just pixels -- it is paths, and those paths ideally map onto semantic parts: body, legs, head, tail. If you reward only the final output, the model can learn to produce plausible-looking results through any sequence of stroke decisions. Reward the intermediate states, and you push the model to build the object the way a human drafter would -- from parts, in order, with visual coherence at each step. The result, the team argues, is generation that is interpretable and locally editable: change the head without regenerating the whole horse.
The paper releases ControlSketch-Part, a new dataset with part-level annotations for vector sketches. The annotation pipeline is worth noting: it uses a multi-stage automatic process to segment existing vector sketches into semantic parts and assign SVG paths to those parts, rather than relying on expensive human labeling from scratch. That is a practical contribution independent of the model itself -- part-level labeled SVG data has been a bottleneck for this subfield.
The lead author is Xiaodan Du. Among the co-authors is Yael Vinker, a researcher at MIT CSAIL whose prior work, SketchAgent (CVPR 2025), is one of the main systems this paper positions against. SketchAgent prompted Claude Sonnet -- zero-shot, no fine-tuning -- to generate vector sketches through an iterative agent loop. This new paper moves in the opposite direction: take an open VLM, fine-tune it on structured part data, then push it further with RL. Vinker is effectively iterating on her own prior work, which is the clearest signal that this direction is being taken seriously.
What is not fully clear from the accessible paper sections: which base VLM was fine-tuned, what the quantitative benchmarks look like against competing methods, and whether model weights will be released alongside the dataset. That last gap matters. ControlSketch-Part is confirmed for release; the model itself is unconfirmed. A dataset without a trained model limits reproducibility, and given that the team's prior system (SketchAgent) relied on proprietary API access, the open-weight question is not trivial.
The immediate competition includes Reason-SVG and a handful of other RL-for-SVG generation papers from the past year. The process-reward framing is the differentiator the authors lean on most -- and it is a real one. Standard outcome-reward RL in generative settings produces brittle generation paths; process-reward approaches have shown cleaner results in other sequential generation domains (see: process reward models in math reasoning). Whether that transfers cleanly to sketch generation is the empirical question the full benchmark data would answer.
For practitioners, the downstream case is real: SVG generation that is semantically structured at the part level is genuinely more useful for design tooling than a black-box output. If you can edit one part without touching the rest, the output integrates into creative workflows. That is a different product than what current text-to-image or even text-to-SVG systems offer.
The paper is at https://arxiv.org/abs/2603.19500. No press coverage as of this writing.