Unified Visual-to-Code Generation via Symbolic Rewards and Reference-Guided Code Optimization
Yaozhi Zheng, Yilei Jiang, Manyuan Zhang, Yuxuan Wan, Kaituo Feng, Tianshuo Peng, Bo Zhang, Xiangyu Yue · 2026
Read Unified Visual-to-Code Generation via Symbolic Rewards and Reference-Guided Code Optimization by Yaozhi Zheng, Yilei Jiang, Manyuan Zhang, Yuxuan Wan, Kaituo Feng, Tianshuo Peng, Bo Zhang, Xiangyu Yue (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-31732.txt.
Introduction
Zheng and seven colleagues tackle visual-to-code generation, where a model turns charts, vector graphics or webpages into code that redraws them. Standard fine-tuning leaves small errors in colors, positions and labels. Reinforcement learning could help, but its rewards are often too coarse and its search can stall before any valid code appears.
Their framework, UniCoder, adds two fixes. A helper model parses generated code into discrete attributes such as hex colors, giving dense element-level rewards. A second trick injects correct example trajectories into weak rollout groups, guiding exploration. On four benchmarks the resulting 8-billion-parameter model beats open-source rivals and matches proprietary systems, the authors report.
Catalog record
| CREATOR | Yaozhi Zheng, Yilei Jiang, Manyuan Zhang, Yuxuan Wan, Kaituo Feng, Tianshuo Peng, Bo Zhang, Xiangyu Yue |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.31732
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Unified Visual-to-Code Generation via Symbolic Rewards and Reference-Guided Code Optimization Yaozhi Zheng, Yilei Jiang, Manyuan Zhang, Yuxuan Wan, Kaituo Feng, Tianshuo Peng, Bo Zhang, Xiangyu Yue Abstract Visual-to-Code generation, which transforms scientific plots, vector graphics, and webpages into executable scripts, demands a level of pixel-precise alignment that standard Multimodal Large Language Models (MLLMs) fail to achieve through Supervised Fine-Tuning (SFT) alone. While Reinforcement Learning (RL) offers a theoretical pathway to bridge this gap, its application is hindered by two fundamental obstacles: (1) Reward Coarseness , where semantic metrics like CLIP scores fail to penalize fine-grained element deviations, and (2) Exploration Stagnation , where the sparse, heterogeneous code search space prevents the policy from bootstrapping valid trajectories. To overcome these limitations, we introduce UniCoder, a unified RL framework that integrates two novel mechanisms. First, we propose Symbolic Attribute Alignment , which employs a lightweight auxiliary LLM to parse generated code into discrete visual attributes (e.g., hex colors, coordinate limits), enabling dense, element-wise reward computation. Second, to escape local optima, we devise Reference-Guided Code Optimization , a strategy that dynamically injects ground-truth trajectories into low-performing rollout groups, transforming blind exploration into guided policy improvement. Extensive experiments on ChartMimic, UniSVG, Design2Code and ScreenBench benchmarks demonstrate that our 8B-parameter model not only surpasses all open-source baselines but also achieves state-of-the-art performance comparable to proprietary models, establishing a new paradigm for generalized visual-to-code synthesis. Our code will be open-sourced at https://github.com/JimmyZhengyz/unicoder . 1 1 footnotetext: Equal contribution. 2 2 footnotetext: Corresponding author. 1 Introduction The capability to translate visual information into executable code stands as a pivotal frontier in Multimodal Large Language Model (MLLM) research. From reverse-engineering scientific charts into Python scripts to converting design mockups into functional webpages, this task bridges the gap between visual intent and digital implementation. Unlike general visual captioning where semantic ambiguity is frequently tolerated, visual-to-code generation demands rigorous precision. The generated code must not only be syntactically executable but also render a visual output that is structurally and stylistically isomorphic to the input. Recent advancements in MLLMs have been largely driven by large-scale Supervised Fine-Tuning (SFT). While SFT enables models to learn high-level correspondences between image layouts and code structures, it often plateaus in fine-grained alignment. SFT-trained models frequently struggle with precise coordinate reasoning in Scalable Vector Graphics (SVG) or specific attribute extraction in scientific plotting. Consequently, they often default to hallucinated values that fit the training distribution rather than the specific instance. Reinforcement Learning (RL) offers a promising paradigm to transcend these limitations by optimizing directly for outcome-based feedback. Techniques such as Group Relative Policy Optimization (GRPO) have demonstrated remarkable success in logical reasoning and code generation. However, adapting RL to the visual-to-code domain introduces two fundamental challenges: (1) The Reward Alignment Problem: Constructing a reward signal that is both dense and accurate is notoriously difficult. Pixel-level metrics are overly sensitive to high-frequency rendering artifacts where a minor padding shift can penalize a structurally correct solution. Conversely, semantic metrics are often too coarse to penalize hallucinations in text labels. (2) Exploration Stagnation: The search space for executable code is sparse and brittle. A model facing a complex visual input often suffers from a “cold start” problem where it fails to generate any executably valid trajectories during early training. This leads to gradient variance collapse and learning stagnation. To address these challenges, we propose a unified reinforcement learning framework designed to master diverse visual-to-code tasks simultaneously. We introduce Symbolic Attribute Alignment , a mechanism that leverages a lightweight and auxiliary LLM to parse generated code and extract discrete visual attributes such as fill colors and axis limits. This enables the calculation of precise and element-level rewards that decouple structural correctness from rendering noise. Furthermore, we propose Reference-Guided Code Optimization to mitigate exploration stagnation. This strategy dynamically injects ground-truth reference code into the policy’s rollout buffer during low-performance episodes. This effectively bootstraps the exploration process by transforming the optimization landscape from sparse blind search to guided imitation. Our main contributions are summarized as follows: • We identify the critical bottlenecks of reward coarseness and exploration stagnation when applying RL to multimodal code generation. • We propose UniCoder, a novel framework integrating an LLM-based Attribute Extractor for fine-grained reward modeling and a Reference Injection strategy for stable policy exploration. • We demonstrate that our 8B-parameter model achieves state-of-the-art performance among open-source models on multiple diverse benchmarks including ChartMimic, UniSVG, Design2Code and ScreenBench. Our method significantly closes the gap with proprietary models like GPT-5. 2 Related Work 2.1 Multimodal Large Language Models for Visual-to-Code Generation The emergence of Multimodal Large Language Models (MLLMs) has significantly advanced the field of joint visual-text reasoning. Early explorations such as LLaVA 11 laid the groundwork for aligning visual and textual representations by introducing a simple linear projection layer to map CLIP visual features into the LLaMA word embedding space. Qwen-VL 1 and DeepSeek-VL 12 were fine-tuned to further enhance high-resolution processing and OCR capabilities essential for reading code-related visual elements. Contemporary foundation models like GPT-5.1 and Gemini 3 Pro have dramatically expanded the frontiers of multimodal understanding, excelling at tasks that require generating high-fidelity code from complex imagery. Parallel efforts pursue unified reasoning across modalities, with OneThinker 5 consolidating image and video understanding into a single model. Beyond general-purpose reasoning, increasing attention has been dedicated to specialized domains. Python Plots : Early works like DePlot 10 converted charts into textual tables for downstream reasoning. More recently, Plot2Code 16 and ChartMimic 19 introduced benchmarks requiring models to generate executable Python code (e.g., Matplotlib) that strictly reproduces the visual appearance of reference charts. SVG : DeepSVG pioneered deep generative models for vector icons using hierarchical transformers to predict path commands. Building on this, IconShop 18 and StarVector 13 explored text-guided SVG generation. For large-scale icon-to-SVG translation evaluation, UniSVG 9 established a comprehensive framework highlighting the difficulty MLLMs face in maintaining topological correctness and geometric precision without explicit reinforcement. Webpage : WebSight 2 managed to utilize millions of synthetic screenshot-code pairs for fine-tuning. Design2Code 15 provided a rigorous benchmark for real-world frontend generation, demonstrating that while SFT models can capture high-level layouts, they often struggle with fine-grained alignment and complex DOM tree nesting. Recent attempts such as LayoutCoder integrated multi-agent framework to further enhance webpage fidelity for UI-to-Code generation. Beyond static frontend synthesis, agentic coding systems such as OpenGame 7 extend executable code generation to interactive game environments. 2.2 Reinforcement Learning for Multimodal Alignment Group Relative Policy Optimization (GRPO) 14 has rapidly emerged as a paradigm-shifting approach for post-training alignment, demonstrating exceptional efficacy in unlocking complex reasoning capabilities, particularly for code generation tasks. Recent studies further show that group-based policy optimization can benefit from external gold references when sampled responses provide weak contrastive signals 17 , suggesting the importance of reliable guidance in low-reward regimes. Complementary work explores dedicated reasoning reward models for agentic systems 4 , underscoring the broader challenge of designing reward functions that reflect task-specific correctness. However, applying such advanced optimization to domain-specific visual synthesis remains constrained by the quality of reward signals. Existing approaches in this field predominantly rely on coarse-grained objectives: Leading approaches for SVG generation like VectorFusion 6 and StarVector 13 predominantly rely on basic semantic distillation (e.g., CLIP Score) or coordinate reconstruction