Advancing vision-language models in front-end development via data synthesis
Tong Ge, Yashu Liu, Jieping Ye, Tianyi Li, Chao Wang · 2025
Read Advancing vision-language models in front-end development via data synthesis by Tong Ge, Yashu Liu, Jieping Ye, Tianyi Li, Chao Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01619.txt.
Introduction
Tong Ge and co-authors present a data making workflow in 2025 to teach vision language models to write front end code from design images. Front end work has its own traps: parts must stay modular, data and visuals must stay in sync, and shared components must flex across uses. Even top models struggle to turn a design picture into working code for such frameworks.
Their workflow uses agents that reflect on their own output to build high quality paired image and code data. It pulls out self contained code pieces and captures the range of real front end traits. Training on that made data is meant to close the gap between a picture of a page and code that runs. The paper reports the gains from the approach. Better code from images could speed web building a lot.
Catalog record
| CREATOR | Tong Ge, Yashu Liu, Jieping Ye, Tianyi Li, Chao Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 102 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-01619
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Advancing vision-language models in front-end development via data synthesis Tong Ge, Yashu Liu, Jieping Ye, Tianyi Li, Chao Wang Abstract 1 Abstract Modern front-end (FE) development, especially when leveraging the unique features of frameworks like React and Vue, presents distinctive challenges. These include managing modular architectures, ensuring synchronization between data and visual outputs for declarative rendering, and adapting reusable components to various scenarios. Such complexities make it particularly difficult for state-of-the-art large vision-language models (VLMs) to generate accurate and functional code directly from design images. To address these challenges, we propose a reflective agentic workflow that synthesizes high-quality image-text data to capture the diverse characteristics of FE development. This workflow automates the extraction of self-contained 1 1 1 A self-contained code snippet is one that encapsulates all necessary logic, styling, and dependencies, ensuring it functions independently without requiring external imports or context. code snippets from real-world projects, renders the corresponding visual outputs, and generates detailed descriptions that link design elements to functional code. To further expand the scope and utility of the synthesis, we introduce three data synthesis strategies: Evolution-based synthesis, which enables scalable and diverse dataset expansion; Waterfall-Model-based synthesis, which generates logically coherent code derived from system requirements; and Additive Development synthesis, which iteratively increases the complexity of human-authored components. We build a large vision-language model, Flame, trained on the synthesized datasets and demonstrate its effectiveness in generating React code via the pass @ k \text{pass}@k metric. Our results suggest that a code VLM trained to interpret images before code generation may achieve better performance. 2 Introduction The astonishing progresses of large language models (LLMs) have ushered a new era of automatic code generation ( Chen et al. 2021 ; Fried et al. 2022 ; Nijkamp et al. 2022 ; Li et al. 2022 ; Zheng et al. 2023 ; Roziere et al. 2023 ; Li et al. 2023a ; Luo et al. 2023 ; Phung et al. ; Guo et al. 2024 ; Zhu et al. 2024 ; CodeGemmaTeam 2024 ; Hui et al. 2024 ) . Developers can now write high-quality codes simply by a couple of instruction messages, substantially increasing productivity. The GitHub Copilot, a commercialized code editor extension built upon the code models, has been widely acknowledged by software developers and had 1.8 million paid subscribers as reported in May 2024 ( Shaw 2024 ) . However, natural language instructions alone are insufficient for comprehensive human-machine interaction. This is particularly true in FE development, where certain ideas or designs are hard to articulate precisely or require cumbersome descriptions for accurate conveyance. In real-world FE development scenarios, UI design mockups are often necessary to facilitate developers’ understanding of the web design requirements. These requirements span various dimensions, including modular development, state management, and data-driven declarations. Accurately describing such multifaceted aspects through text alone is time-intensive and prone to ambiguity, thereby limiting the practical utility of code LLMs in these contexts. Furthermore, state-of-the-art vision-language models (VLMs) for code generation reveal substantial limitations when applied to the intricacies of FE development. Models like GPT-4o, though capable of generating basic components, are unsuitable for modern front-end workflows. Their outputs are often static, lacking modularity, reusability, and dynamic behavior—essential for scalable and interactive user interfaces—resulting in inefficient and incompatible code that deviates from established development practices. To bridge the gap between natural language instructions and the intricacies of front-end design, we introduce a self-reflective agentic workflow that synthesizes high-fidelity training data, enabling the development of a large vision-language model for end-to-end front-end code generation. Our key contributions are as follows: • Self-reflective agentic workflow for data synthesis. A major challenge in developing a strong large-scale VLM for front-end (FE) code generation is the scarcity of high-quality image-text data. To address this, we propose an automated pipeline that extracts, renders, and annotates self-contained FE code snippets. As a key contribution, we introduce three novel data synthesis methods—Evolution-Based, Waterfall-Model-Based, and Additive Development synthesis—which enable the generation of large-scale, diverse, and high-fidelity training data. Moreover, the proposed pipeline flexibly supports data generation for both single-image and multi-image code generation, making it broadly applicable to diverse front-end development scenarios. • Large Vision-Language Model for Front-End Code Generation. We develop Flame: F ront-end l anguage a ssistant with m ultimodal e xpertise, a large vision language model that integrates the code generation capability of a code LLM with the visual understanding capability of a vision encoder, enabling direct translation of design concepts into executable front-end code. We explore optimal practice for training a VLM for FE code generation through ablation studies. Empirical studies also suggesting that model tuned to interpret the input image before code generation can potentially greatly enhance performance, analogous to how Chain-of-Thought ( Wei et al. 2022 ) improves large language models (LLMs). • Benchmarking React Code Generation. To the best of our knowledge, we construct Flame-React-Eval, the first benchmarking dataset designed to assess syntactic precision, functional correctness, and visual consistency in React code generation across a range of design specifications. • Open Source. To advance the field of front-end code generation, we publicly release our data synthesis pipeline, the synthesized training data, the evaluation dataset, and the Flame models 2 2 2 https://github.com/Flame-Code-VLM/Flame-Code-VLM . 3 Related Work 3.1 Code Models The progress of LLMs for code has been made at a surprisingly fast pace these years. Codex ( Chen et al. 2021 ) showed that LLMs were produce workable solutions to real programming problems, pioneered the use of LLMs for code. In the development of the open-source code LLMs in recent years, the model size has been explored from 0.5 billion to 236 billion and the scale of training code data has grown to more than 3 trillion tokens, and as a result the SOTA code model can achieve more than 90% pass@1 on the HumanEval dataset. Besides the work on open LLMs for code, techniques for code instruction-tuning have emerged. The Self-Instruct ( Wang et al. 2022 ) approach generates instruction-tuning datasets through an iterative process: prompting a powerful LLM with samples from a seed task pool, generating instructions and corresponding responses, filtering the results, and progressively expanding the task pool. It is widely adopted in the code generation community. For example, CodeAlpaca ( Chaudhary 2023 ; Taori et al. 2023 ) and CodeLlama ( Roziere et al. 2023 ) were instruction-tuned on the datasets constructed under variants of the dea. WizardCoder ( Luo et al. 2023 ) employed an evolutionary algorithm ( Xu et al. 2024 ) to create diverse and complex code instruction data. OctoCoder ( Muennighoff et al. 2023 ) showed that instructions dataset made from Git commits was also beneficial for code generation performance. The advancements in LLMs for code generation have paved the way for AI software engineers. Leveraging the coding capabilities of LLMs, autonomous multi-agent systems such as MetaGPT ( Hong et al. 2024a ; Hong et al. 2024b ) , BabyAGI ( Yohei Nakajima ) , and ChatDev ( Qian et al. 2023 ) have demonstrated the ability to accomplish complex tasks, like writing a CLI blackjack game. 3.2 Large Vision-Language Models To fully leverage the reasoning capability of an LLM and the visual perception capability of a vision model, Flamingo