Teaching an Agent to Sketch One Part at a Time

Xiaodan Du ∗, Ruize Xu ∗, David Yunis ∗, Yael Vinker, Greg Shakhnarovich · 2026

Read Teaching an Agent to Sketch One Part at a Time by Xiaodan Du ∗, Ruize Xu ∗, David Yunis ∗, Yael Vinker, Greg Shakhnarovich (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-19500.txt.

Introduction

Du and co-authors teach an agent to sketch vector drawings one part at a time. They train a multimodal language model with multi-turn process-reward reinforcement learning after supervised fine-tuning. A new data set, ControlSketch-Part, gives rich part-level labels for sketches. An automatic pipeline segments sketches into semantic parts and assigns paths to parts. With part data and visual feedback, the agent's sketches stay interpretable and easy to edit. Parts can be changed locally without redrawing the whole.

Catalog record

CREATORXiaodan Du ∗, Ruize Xu ∗, David Yunis ∗, Yael Vinker, Greg Shakhnarovich
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52647
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.19500

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Teaching an Agent to Sketch One Part at a Time Xiaodan Du ∗, Ruize Xu ∗, David Yunis ∗, Yael Vinker, Greg Shakhnarovich Abstract We develop a method for producing vector sketches one part at a time. To do this, we train a multi-modal language model-based agent using a novel multi-turn process-reward reinforcement learning following supervised fine-tuning. Our approach is enabled by a new dataset we call ControlSketch-Part, containing rich part-level annotations for sketches, obtained using a novel, generic automatic annotation pipeline that segments vector sketches into semantic parts and assigns paths to parts with a structured multi-stage labeling process. Our results indicate that incorporating structured part-level data and providing agent with the visual feedback through the process enables interpretable, controllable, and locally editable text-to-vector sketch generation. Figure 1 : Progressive vector sketch generation using our VLM agent. Trained on our new dataset via SFT + RL training, our agent generates sketches part-by-part, conditioned on text instructions and the evolving canvas. It produces diverse, structurally plausible sketches and supports localized editing via arbitrary stroke removal and replacement. † † * Equal contribution. 1 Introduction Sketching provides a structured abstraction of visual content and enables rapid ideation and concept exploration in domains from industrial design to digital art. Sketches represented as vector graphics offer many advantages over rasterized canvasses, making them useful in creative workflows: infinite scalability, structured visual elements, support for precise and localized modifications, and more. Automatic text-to-vector sketch generation has been widely explored [ 13 , 7 , 8 , 9 , 42 , 10 , 39 , 6 , 25 , 36 ] . However, the majority of existing works generate the full sketch at once, overlooking the progressive, step-by-step nature of the sketching process. Having the strokes in a sketch grouped into meaningful parts makes the sketch more easily editable : parts could be removed, replaced or modified in isolation from the rest of the sketch more efficiently than by modifying individual strokes. It also makes the sketch more interpretable to a human user. Finally, one-shot generation from a long, compositional prompts may lead to failures that are localized but difficult to mitigate. In contrast, incorporating the notion of parts into the generation pipeline provides the designer fine-grained control : if a generated sketch of a part is not right, it can be replaced, and multiple choices can be explored at any intermediate stage before proceeding to other parts. Most prior work on text-to-vector sketch generation does not allow for part-by-part generation. The one exception is SketchAgent [ 36 ] which relies on close-sourced vision-language model (VLM) as backend (and thus is not easily adaptable to desired domain or style) and produces simplistic, icon-style outputs. This leaves a gap: existing methods are unable to achieve free-text guided part-by-part generation and highly detailed vector sketch with a unified model, struggling to support human-friendly workflows which include branching possibilities and creative exploration. In contrast, our method makes these possible, as illustrated in Fig. 1 . We believe that a necessary element for closing this gap is the right training data. Work like SketchAgent has demonstrated the potential of VLMs in iteratively generating sketches conditioned on text. However, large language models (LLMs) are known to be extremely data-hungry [ 33 , 16 ] and collecting a large amount of high-quality part-annotated datasets for vector sketch created by professionals can be costly and difficult to scale. To overcome this obstacle, we propose a scalable pipeline for annotating parts in vector sketches. Our pipeline relies on a multi-stage labeling process for part decomposition and path assignment. It includes proposal, critique, and revision stages. This pipeline is generic and can be applied to any vector sketch data. We apply the aforementioned data collection pipeline on the ControlSketch dataset [ 1 ] . We call the resulting part-annotated dataset (which we will release as one of our contributions) ControlSketch-Part , and use it to train a VLM on the text-guided part-by-part generation task. The training uses a two-stage supervised fine-tuning (SFT) and reinforcement learning (RL) framework, where the SFT stage teaches format and initializes the sketching policy for a single turn, and an innovative multi-turn process-reward GRPO RL training stage aligns multi-turn rollouts using intermediate-state rewards [ 30 , 29 ] . Our automatic data pipeline and the proposed training strategy enable free-text guided multi-turn interactive sketch generation. We show, quantitatively using automated metric and user studies, and qualitatively in the paper and the supplementary, that our results significantly improve on prior work. In summary, our contributions are: • A generic scalable pipeline for automated VLM-based part annotation of vector sketches, yielding a short overall caption, a set of semantic part descriptions, and a complete path-to-part assignment for any vector sketch. • A high-quality part-annotated sketch dataset, ControlSketch-Part, and an associated new benchmark for multi-turn text-to-vector sketch generation. • A novel multi-turn process-reward GRPO algorithm for training, enabling us to train a sketching agent with novel capabilities: multi-turn vector sketch generation and progressive editing of sketches with text guidance. The qualitative and quantitative experiment results show the potential of our data pipeline combined with VLM in the field of text-to-vector sketch synthesis. 2 Related Works 2.1 Text-to-Vector Sketch Synthesis Previous works on text-to-vector sketch generation fall into two main categories: learning-based approaches and test-time optimization-based approaches. Learning-based approaches Sketch-RNN [ 13 ] is one of the first works to explore this task by learning to generate polylines autoregressively. BézierSketch [ 7 ] improves upon Sketch-RNN by replacing polylines with Bézier curves for better smoothness. SketchODE [ 8 ] further extends autoregressive stroke generation by modeling sketches as continuous-time functions via Neural ODEs. More recently, inspired by the success of score-based generative modeling [ 15 , 31 ] , methods including ChiroDiff [ 9 ] and StrokeFusion [ 42 ] apply diffusion models to sketch synthesis and denoise all strokes simultaneously, offering no progressive, part-level control over the generation process. These methods are conditioned on pre-defined discrete class/attribute labels rather than free-form natural language, greatly limiting their real-world applicability. They also fall short of producing complex, high-fidelity sketches. Test-time optimization-based approaches These methods take a longer time to produce an output but offer greater flexibility and higher visual quality. CLIPDraw [ 10 ] pioneers text-guided vector sketch synthesis by optimizing SVG paths with a CLIP-based [ 26 ] objective. Later works utilize CLIP-based optimization for versatile image-to-sketch generation [ 35 , 34 ] . DiffSketcher [ 39 ] , AutoSketch [ 6 ] , and SketchDreamer [ 25 ] leverage a wider range of supervision, such as LPIPS [ 41 ] loss and score distillation sampling (SDS) [ 24 ] loss, to achieve higher visual quality. However, these methods optimize all strokes jointly for a single text input, producing sketches without meaningful stroke ordering or semantic part structure. The most directly relevant work to part-aware, text-guided sketch synthesis is SketchAgent [ 36 ] , which uses a closed-source Claude Sonnet model in a zero-shot prompting framework to perform text-guided sequential sketching. SketchAgent’s zero-shot nature constrains it to doodle-style outputs that cannot be adapted to higher visual fidelity or specific domains. It also exhibits low spatial grounding accuracy. 2.2 Reinforcement Learning for Large Language Models

Read the complete text