Vision2Code: A Multi-Domain Benchmark for Evaluating Image-to-Code Generation

Ajay Vikram Periasami 1 , Junlin Wang 1 , Bhuwan Dhingra 1 1 Duke University [email protected], [email protected], [email protected] · 2026

Read Vision2Code: A Multi-Domain Benchmark for Evaluating Image-to-Code Generation by Ajay Vikram Periasami 1 , Junlin Wang 1 , Bhuwan Dhingra 1 1 Duke University [email protected], [email protected], [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-11307.txt.

Introduction

Ajay Vikram Periasami, Junlin Wang, and Bhuwan Dhingra introduce Vision2Code, a multi-domain benchmark for evaluating image-to-code generation. Image-to-code generation asks whether a vision-language model can recover an image's structure as executable code.

Existing benchmarks focus on narrow domains, depend on paired reference code, or rely on generic rubrics missing domain errors. Vision2Code is a reference-free multi-domain image-to-code benchmark. It holds test examples spanning charts, geometry, graphs, scientific imagery, documents, and 3D scenes. Models generate programs, rendered and scored against the source image by a VLM judge using dataset-specific rubrics and guardrails for severe failures. Diagnostics separate execution failures from reconstruction quality. Human checks show this protocol beats a generic visual rubric and embedding-similarity baselines. Across nine models, performance is domain-dependent: leading models do well on regular charts and graphs but stay weak elsewhere.

Catalog record

CREATORAjay Vikram Periasami 1 , Junlin Wang 1 , Bhuwan Dhingra 1 1 Duke University [email protected], [email protected], [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE84829
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.11307

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Vision2Code: A Multi-Domain Benchmark for Evaluating Image-to-Code Generation Ajay Vikram Periasami 1 , Junlin Wang 1 , Bhuwan Dhingra 1 1 Duke University [email protected], [email protected], [email protected] Abstract Image-to-code generation tests whether a vision-language model (VLM) can recover the structure of an image enough to express it as executable code. Existing benchmarks either focus on narrow visual domains, depend on paired executable reference code, or rely on generic rubrics that miss domain-specific reconstruction errors. We introduce Vision2Code , a reference-code-free benchmark and evaluation framework for multi-domain image-to-code generation. Vision2Code contains 2,169 test examples from 15 source datasets that span charts and plots, geometry, graphs, scientific imagery, documents, and 3D spatial scenes. Models generate executable programs, which we render and score against the source image using a VLM rater with dataset-specific rubrics and deterministic guardrails for severe semantic failures. We report render-success diagnostics that separate code execution failures from reconstruction quality. Human validation shows that this evaluation protocol aligns better with human judgments than either a generic visual rubric or embedding-similarity baselines. Across nine open-weight and proprietary models, we find that image-to-code performance is domain-dependent: leading models perform well on regular chart- and graph-like visuals but remain weak on spatial scenes, chemistry, documents, and circuit-style diagrams. Finally, we show that evaluator-filtered model outputs can serve as training data to improve image-to-code capability, with Qwen3.5-9B improving from 1.60 to 1.86 on the benchmark without paired source programs. Vision2Code provides a reproducible testbed for measuring, diagnosing, and improving image-to-code generation. Our code and data are publicly available at https://image2code.github.io/vision2code/ . 1 Introduction Image-to-code generation tests whether a vision-language model (VLM) can turn an image into executable code whose rendered output matches the source. This is different from pixel generation in that the output is editable, verifiable, interpretable, and reusable. A user can change a label, move a node, adjust an axis, or refactor a layout by editing code rather than manipulating pixels. This is the same reason tools such as LaTeX, Excalidraw, and Figma are useful in practice: they represent visual content as structured artifacts that can be verified, edited, and reused. Generating code that recreates an image is also a strict test of visual understanding. A model may answer a question about a chart while missing its axis scale, or describe a circuit while reversing its connectivity. To recreate the image as code, the model must preserve the structure that makes the image meaningful. Prior image-to-code benchmarks have made progress, but they leave important settings uncovered. Chart and plot benchmarks focus only on data visualization ( Yang et al. 2025a ; Wu et al. 2025 ; Zhang et al. 2026a ; Tang et al. 2025 ) , while web and UI benchmarks focus only on screenshots of web pages or widgets ( Si et al. 2025 ; Yun et al. 2024 ; Zhang et al. 2026b ) . Broader benchmarks such as Omni-I2C ( Zhou et al. 2026 ) study high-fidelity reconstruction across multiple technical graphic types, but they use paired executable reference code and relatively generic evaluation criteria. Reference code is useful when available because it enables code-level symbolic evaluation, but it also limits evaluation to images whose source programs exist or can be curated. Many realistic visual datasets, such as textbook diagrams, scanned documents, hand-drawn or semi-structured figures, provide only images and task metadata. For these settings, evaluation must rely only on the source image and rendered output, and must score domain-specific errors rather than only generic text, entity, color, or layout differences. Other visually conditioned code benchmarks, such as HumanEval-V ( Zhang et al. 2024a ) and MMCode ( Li et al. 2024 ) , evaluate whether code solves a task specified by an image, not whether it visually reconstructs the image. These gaps leave existing benchmarks either narrow in visual scope, dependent on paired reference code, or reliant on generic rubrics that miss domain-specific reconstruction errors. We introduce Vision2Code , a reference-code-free benchmark for image-to-code generation across 15 source datasets and six visual domains: charts and plots, geometry, graphs, scientific imagery, documents, and 3D spatial scenes. The benchmark contains 2,169 test examples, along with a 539-example test-mini subset for lower-cost evaluation. By evaluating generated programs through their rendered outputs rather than paired source code, Vision2Code can include visual datasets whose original source programs are unavailable. To make rendered-output evaluation meaningful across domains, Vision2Code uses dataset-specific rubrics rather than a single generic rubric. We build each rubric by identifying the visual information that must be preserved for that dataset: axes and legends for charts, topology for graphs, symbols and bonds for chemistry, text and layout for documents, and depth relations for spatial scenes. The VLM rater scores the rendered output against the source image using four weighted rubric categories on a 0–5 scale, where 5 indicates a near-exact reconstruction and 0 indicates no meaningful match or failed output. The highest-weight semantic categories are treated as critical: if they receive very low scores, deterministic guardrails cap the final score so that severe semantic mistakes cannot receive high ratings. We report render-success diagnostics for code execution. We validate the evaluation protocol against human judgments and show that the dataset-specific rubric aligns better with human ratings than a generic rubric or embedding cosine-similarity baselines. The dataset-specific rubric reaches Pearson/Spearman correlations of 0.723/0.703, compared with 0.626/0.633 for a generic rubric and lower correlations for embedding cosine-similarity baselines. We evaluate nine open-weight and proprietary models. The results show that image-to-code generation remains domain-dependent: the best test model reaches 4.06/5.0 overall, but the best spatial score is only 2.42/5.0. Models perform well on regular chart and graph-like visuals, yet remain weak on spatial scenes, chemistry, documents, and circuit-style diagrams. Finally, we use the evaluator as a source of training signal. While the source datasets provide images, they do not provide ground-truth reconstruction code. We therefore generate programs from training-pool images, score their rendered outputs, and group them by reconstruction quality and self-consistency. The most useful tier contains programs that reconstruct the source well once but are not stably reconstructed again, suggesting that they expose visual structures the model has not learned robustly. Fine-tuning Qwen3.5-9B on these rater-filtered data improves its score from 1.60 to 1.86 on test-mini, while repeated test-time scaling gives only limited gains. In summary, our contributions are: (1) Vision2Code , a reference-code-free benchmark for image-to-code generation across 15 source datasets and six visual domains that span charts and plots, geometry, graphs, scientific imagery, documents, and 3D spatial scenes, enabling evaluation beyond settings where paired rendering programs are available; (2) a domain-aware evaluation framework that renders generated code, scores outputs with a VLM rater and dataset-specific rubrics while applying deterministic guardrails for severe semantic failures; and (3) an evaluator-guided training pipeline that uses rendered-output scores to filter model-generated programs into useful supervision, improving VLM image-to-code generation capability without paired source code. Figure 1 : Vision2Code spans 15 source datasets across six domains: charts and plots, geometry, graphs, science, documents, and spatial scenes. The domains target complementary reconstruction challenges, including axes and graphical marks, label-to-object binding, topology and directionality, domain-specific notation, dense text layout, and 3D spatial relations. 2 Related Work Code as a Medium for Visual Reasoning. Visual reasoning in VLMs is often studied

Read the complete text