VectorGym: A Multitask Benchmark for SVG Code Generation, Sketching, and Editing
Juan Rodriguez, Haotian Zhang, Abhay Puri, Tianyang Zhang, Rishav Pramanik, Meng Lin, Xiaoqing Xie, Marco Terral, Darsh Kaushik, Aly Shariff, Perouz Taslakian, Spandana Gella, Sai Rajeswar, David Vazquez, Christopher Pal, Marco Pedersoli · 2026
Read VectorGym: A Multitask Benchmark for SVG Code Generation, Sketching, and Editing by Juan Rodriguez, Haotian Zhang, Abhay Puri, Tianyang Zhang, Rishav Pramanik, Meng Lin, Xiaoqing Xie, Marco Terral, Darsh Kaushik, Aly Shariff, Perouz Taslakian, Spandana Gella, Sai Rajeswar, David Vazquez, Christopher Pal, Marco Pedersoli (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-29852.txt.
Introduction
Rodriguez and co-authors launch VectorGym, a benchmark suite for SVG vector graphics tasks. It covers drawing from text, drawing from sketches, complex editing, and captioning. Expert human notes anchor all four tasks, unlike synthetic-edit sets. A joint reinforcement learning baseline trains across the tasks with rendering-based rewards. Its 8B model beats much larger open models and matches GPT-4o. A VLM-as-judge score, checked against human ratings, aids grading. Frontier models still show large gaps.
Catalog record
| CREATOR | Juan Rodriguez, Haotian Zhang, Abhay Puri, Tianyang Zhang, Rishav Pramanik, Meng Lin, Xiaoqing Xie, Marco Terral, Darsh Kaushik, Aly Shariff, Perouz Taslakian, Spandana Gella, Sai Rajeswar, David Vazquez, Christopher Pal, Marco Pedersoli |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65342 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.29852
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VectorGym: A Multitask Benchmark for SVG Code Generation, Sketching, and Editing Juan Rodriguez, Haotian Zhang, Abhay Puri, Tianyang Zhang, Rishav Pramanik, Meng Lin, Xiaoqing Xie, Marco Terral, Darsh Kaushik, Aly Shariff, Perouz Taslakian, Spandana Gella, Sai Rajeswar, David Vazquez, Christopher Pal, Marco Pedersoli Abstract We introduce VectorGym, a comprehensive benchmark suite for Scalable Vector Graphics (SVG) that spans generation from text and sketches, complex editing, and visual understanding. VectorGym addresses the lack of realistic, challenging benchmarks aligned with professional design workflows. Our benchmark comprises four tasks with expert human-authored annotations: the novel Sketch2SVG task (VG-Sketch); a new SVG editing dataset (VG-Edit) featuring complex, multi-step edits with higher-order primitives; Text2SVG generation (VG-Text); and SVG captioning (VG-Cap). Unlike prior benchmarks that rely on synthetic edits, VectorGym provides gold-standard human annotations that require semantic understanding and design intent. We also propose a multi-task reinforcement learning approach that jointly optimizes across all four tasks using rendering-based rewards. Our method, built on GRPO with curriculum learning, trains a Qwen3-VL 8B model that achieves state-of-the-art performance among open-source models, surpassing much larger models including Qwen3-VL 235B and matching GPT-4o. We also introduce a VLM-as-a-Judge metric for SVG generation, validated through human correlation studies. Our evaluation of frontier VLMs reveals significant performance gaps, positioning VectorGym as a rigorous framework for advancing visual code generation. VectorGym is publicly available on HuggingFace . Figure 1: Overview of VectorGym. VectorGym is a suite of human-authored datasets covering Sketch2SVG ( VG-Sketch ), SVG Editing ( VG-Edit ), Text2SVG ( VG-Text ), and SVG Captioning ( VG-Cap ). Unlike prior benchmarks, it is built from diverse real-world SVGs sourced from GitHub. Human experts annotate each SVG by hand-drawing sketches, creating complex edits, and writing detailed text descriptions, which are further cleaned and adapted into instruction-style prompts at varying levels of detail. We evaluate state-of-the-art models in VectorGym. 1 Introduction Scalable Vector Graphics (SVG) ( 7 ; 19 ) are widely used across the web, design tooling, and digital media. Unlike raster images ( 21 ; 22 ; 24 ) , SVGs are programs: their code exposes geometry, style, and structure, enabling precise editing, scalable rendering, and semantic manipulation. Evaluating models on SVG requires not only visual understanding but also reliable, syntax-aware code generation. Despite rapid progress in Vision-Language Models (VLMs), existing evaluations of SVG generation remain limited in scope. Prior datasets often target icons or basic shapes, rely on synthetic programmatic edits, rarely assess sketch-conditioned generation nor provide human-authored gold labels ( 20 ; 28 ; 31 ; 29 ; 30 ; 23 ) . As a result, the field lacks a unified, realistic benchmark that simultaneously stresses visual understanding, vector generation and structured SVG code manipulation. We introduce VectorGym , a new comprehensive multi-task benchmark for SVG generation and manipulation spanning four tasks: (1) Sketch2SVG ( VG-Sketch ), converting rough sketches to clean vector code; (2) SVG Editing ( VG-Edit ), applying natural-language edits to existing SVGs; (3) Text2SVG ( VG-Text ), generating SVGs from text; and (4) SVG Captioning ( VG-Cap ), describing SVG content. Our benchmark introduces the Sketch2SVG task and releases the first dataset of complex, human-authored SVG edits; all tasks use gold-standard human annotations. VectorGym covers in-the-wild diversity, including icons, diagrams, emojis, fonts, logotypes, and complex illustrations, sourced from SVG-Stack ( 20 ) . We pair this with careful human curation to ensure realistic and challenging task difficulty. We evaluate both proprietary and open-source frontier VLMs, providing a clear view of current capabilities and gaps. Our main contributions are: 1. We introduce a comprehensive, multi-task benchmark for real-world SVG code generation with gold-standard human annotations across all tasks; 2. We introduce the Sketch2SVG task and the first dataset of expert human-authored SVG edits with complex intent, involving rich primitives and non-trivial edits. 3. We introduce a reinforcement learning based method that jointly optimizes models across all four VectorGym tasks, achieving state-of-the-art performance among open-source models. In addition, we propose a task-specific VLM-as-a-judge evaluation suite for SVG generation, covering sketch, text, and editing tasks, and validate it through human correlation studies. 4. We provide extensive evaluation and analysis of current frontier VLMs across SVG generation tasks. Table 1 : SVG Datasets Comparison. We compare similar datasets across annotation quality, task complexity, and data source. We report the SVG type, total samples, whether each dataset supports multiple tasks, the level of SVG primitive coverage beyond simple paths (i.e. including higher-order primitives, like circle, text, gradients or animation logic), and whether it includes annotations for edits and sketches, plus whether these annotations reach expert human quality with complex intent. VectorGym is built from real SVGs collected from GitHub ( in-the-wild ), preserving original structure and primitive detail, and provides targets created by expert humans with the goal of producing complex annotations that capture semantic understanding, design intent, and multi-step reasoning. Dataset SVG Type # Samples Multi-Task Primitives Edits Sketches Human SVGBench ( 20 ) In-the-wild 10k ✗ ✓ ✗ ✗ ✗ VGBench ( 32 ) Emojis 9.5k ✓ ✗ ✗ ✗ ✗ SVGEditBench ( 15 ) Emojis 1.6k ✗ ✗ ✓ ✗ ✗ SVGEditBenchV2 ( 16 ) Emojis 1.6k ✗ ✗ ✓ ✗ ✗ UniSVG ( 10 ) Icons 525k ✓ ✗ ✗ ✗ ✗ SVGenius ( 4 ) Icons 100k ✓ ✗ ✓ ✗ ✗ VectorGym (ours) In-the-wild 7k ✓ ✓ ✓ ✓ ✓ 2 Related Work Vector Graphics Generation. Classical vectorization methods based on shape-fitting algorithms ( 11 ; 27 ) struggle with complex tasks beyond image vectorization. Recent approaches introduce learning-based components, relying on latent variable models with differentiable rendering and attention architectures ( 3 ; 2 ; 13 ) , as well as sketch abstraction ( 26 ) and text-conditioned SVG synthesis ( 8 ) . However, these methods are still not general enough to support a wide range of SVG tasks. VLMs for SVG Generation. Modern VLMs ( 17 ; 6 ) can now produce structured code from visual inputs. StarVector ( 20 ) frames SVG creation as a visual to code generation task, jointly testing visual understanding and program synthesis. Subsequent work further supports this direction ( 31 ; 1 ; 30 ) . SVG Datasets and Benchmarks. Foundational SVG datasets include DeepSVG icons ( 3 ) , FIGR-8 ( 5 ) , and SVG-Stack ( 20 ) . Several benchmarks address different SVG related tasks. UniSVG ( 10 ) unifies 525k SVGs for understanding and generation. VGBench ( 32 ) aggregates multiple sources to evaluate image to SVG, text to SVG, and diagram code generation. SVGEditBench ( 15 ) and its V2 version ( 16 ) target instruction based editing using synthetic LLM generated edits or edits derived from similar SVGs. SVGenius ( 4 ) covers a wide set of tasks, notably editing through algorithmic transform based operations. Here we propose VectorGym , which focuses on edits created by humans following instructions that make the edits complex and closer to the actions of real design professionals, requiring semantic understanding. We also introduce the novel Sketch2SVG task from human drawn sketches, and we collect human validated text captions that allow evaluation of both Text2SVG and SVG captioning on realistic, high difficulty edits. See Table 1 for a dataset comparison, and refer to Appendix A