Multimodal graph representation learning for website generation based on visual sketch

Tung D. VuCollege of Engineering and Computer ScienceVinUniversityHanoi, Vietnam [email protected], Chung HoangDepartment of Computer ScienceHanoi University of Science and TechnologyHanoi, Vietnam [email protected], Truong-Son HyDepartment of Computer ScienceThe University of Alabama a · 2025

Read Multimodal graph representation learning for website generation based on visual sketch by Tung D. VuCollege of Engineering and Computer ScienceVinUniversityHanoi, Vietnam [email protected], Chung HoangDepartment of Computer ScienceHanoi University of Science and TechnologyHanoi, Vietnam [email protected], Truong-Son HyDepartment of Computer ScienceThe University of Alabama a (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-18729.txt.

Introduction

Turning a designer's sketch into working web code is slow, skilled work that automation has yet to master. Tung Vu, Chung Hoang, and Truong-Son Hy proposed a learning method in 2025 that reads a page design as a graph. It joins what the page looks like with how its parts are structured. That dual view aims to catch both the fine visual details and the layout relations that simpler methods miss.

Trained this way, their system generates HTML more often correct in meaning and sound in structure than rival methods. It works faster as well. The authors present multimodal graph learning as a step toward real design-to-code automation and release their code openly.

Catalog record

CREATORTung D. VuCollege of Engineering and Computer ScienceVinUniversityHanoi, Vietnam [email protected], Chung HoangDepartment of Computer ScienceHanoi University of Science and TechnologyHanoi, Vietnam [email protected], Truong-Son HyDepartment of Computer ScienceThe University of Alabama a
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-18729

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multimodal graph representation learning for website generation based on visual sketch Tung D. VuCollege of Engineering and Computer ScienceVinUniversityHanoi, Vietnam [email protected], Chung HoangDepartment of Computer ScienceHanoi University of Science and TechnologyHanoi, Vietnam [email protected], Truong-Son HyDepartment of Computer ScienceThe University of Alabama at BirminghamBirmingham, Alabama, United States [email protected] Abstract The Design2Code problem, which involves converting digital designs into functional source code, is a significant challenge in software development due to its complexity and time-consuming nature. Traditional approaches often struggle with accurately interpreting the intricate visual details and structural relationships inherent in webpage designs, leading to limitations in automation and efficiency. In this paper, we propose a novel method that leverages multimodal graph representation learning to address these challenges. By integrating both visual and structural information from design sketches, our approach enhances the accuracy and efficiency of code generation, particularly in producing semantically correct and structurally sound HTML code. We present a comprehensive evaluation of our method, demonstrating significant improvements in both accuracy and efficiency compared to existing techniques. Extensive evaluation demonstrates significant improvements of multimodal graph learning over existing techniques, highlighting the potential of our method to revolutionize design-to-code automation. Code available at https://github.com/HySonLab/Design2Code . 1 Introduction The Design2Code problem, which involves converting UI designs into functional source code, is a pivotal challenge in software development that lies at the intersection of computer vision, natural language processing, and programming. This task is particularly demanding when generating HTML code from webpage designs, as it requires not only the interpretation of visual elements but also an understanding of their spatial arrangements and hierarchical relationships. While significant advances have been made in both vision-language models and code generation models, combining the strengths of these two fields to solve Design2Code remains an unsolved challenge. One of the fundamental difficulties is that current vision-language models, which excel at generating descriptive text from images ( Hakimov & Schlangen 2023 ; Iscen et al. 2024 ; Lin et al. 2024 ) , often fall short when tasked with understanding the intricate structural layout of a design. These models, which typically focus on high-level image features, struggle with capturing the precise relationships between elements such as deeply nested blocks, grid layout, spatial alignment of diverse visual components. On the other hand, code generation models ( Zan et al. 2023 ; Sun et al. 2024 ; Lyu et al. 2024 ) , despite their proficiency in generating syntactically correct code from textual descriptions, often lack the visual comprehension needed to map these descriptions to complex layouts. This results in syntactically correct code, yet often fails to reflect the intended design. The key to overcoming these challenges lies in developing a model architecture that not only captures the global features of an image but also pays close attention to these critical attributes and relationship, ensuring both breadth and depth in its analysis. Given these motivations, we introduce a novel framework for the Design2Code problem that integrates multimodal information from webpage screenshots and component-level relationships. Using Optical Character Recognition (OCR) and segmentation models, our approach extracts textual and visual components, organizing them into a multimodal graph that captures the webpage’s structure and serves as a blueprint for generating HTML code. By isolating textual components early through OCR, we enable the segmentation model to more accurately focus on visual components without interference from overlapping text. The multimodal graph maps both textual and visual components, capturing their spatial and semantic relationships. Textual elements are linked semantically, while visual components are connected based on spatial proximity, forming the foundation for code generation. Our model integrates the visual elements and structural relationships within a graph representation, using a Vision-Language Model (VLM) to encode the webpage’s appearance. This combined multimodal data ensures both visual and structural details contribute to generating accurate HTML code. The contributions of this paper are threefold: • We propose a Graph-enhanced Vision-Language Model that seamlessly integrates graph modality, vision modality, and text modality to generate accurate HTML code. • We propose a novel approach to construct graph representations of user interfaces, enabling the model to learn latent structures that effectively bridge visual layouts and code semantics, thereby enhancing UI-to-code translation. • We perform extensive experiments to demonstrate that our approach significantly improves both content accuracy and layout fidelity compared to baseline models, offering a robust solution to the Design2Code problem. In the following sections, we first explore the existing research on the problem of converting sketches into website source code in Section 2 . Section 3 then details the methodology employed in this study. The experimental results are discussed in Section 4 , followed by conclusions, limitations, and future research in Section 5 and Section 6 , respectively. 2 Related work 2.1 Design2Code Approaches to Automating Design2Code Generation. The Design2Code (or UI2Code) problem has gained significant attention as researchers work to automate the conversion of user interface (UI) designs into source code. Early methods primarily relied on rule-based systems that assigned UI components to predefined code templates ( Nguyen & Csallner 2015 ) , offering limited flexibility. As the complexity of web design increased, newer approaches adopted deep learning techniques for enhanced automation and accuracy. Notable models like Pix2Code ( Beltramelli 2018 ) and Sketch2Code ( Robinson 2019 ) utilize convolutional neural networks (CNNs) to transform UI screenshots into single-file code, with Pix2Code demonstrating cross-platform versatility and Sketch2Code focusing on wireframe sketches. Recent efforts, such as those by Soselia et al. 2023 , treat image-to-code generation as a Reinforcement Learning problem, using the Intersection over Union (IoU) score as a reward signal. Evaluations on benchmarks such as Design2Code Si et al. 2024 have utilized open-access models from the GPT, Claude, and Gemini families, achieving strong performance without the need for fine-tuning. Bo et al. 2023 adopts a hybrid approach that leverages computer vision techniques to better capture the structure and components of UI images, in combination with deep learning models for code generation. Other works draw inspiration from the spatial or hierarchical organization of UI components to guide the model or adjust the inference strategy ( Zhou et al. 2024 ; Gui et al. 2025 ) . The work most similar to ours is Jiang et al. 2024 , which also employs computer vision techniques and rule-based methods to construct a graph, followed by training a Graph Neural Network to understand the high-level structure of UI components. However, their primary objective is not code generation. In contrast, our work enhances Visual-Language Models (VLMs) of suitable size by incorporating graph learning, aiming to better capture structural features and improve accuracy in the Design2Code task. Datasets for Design2Code. A crucial factor in advancing Design2Code methods is the availability of suitable datasets and benchmarks. One foundational dataset in this field is Pix2Code, an open-source, synthesized collection of 1,750 UI screenshots paired with corresponding source code. The Sketch2Code dataset ( ShantamVijayputra 2022 ) builds on this by converting Pix2Code screenshots into hand-drawn wireframe representations, adding an additional layer of abstraction for model training. Additionally, the Websight dataset from Hugging Face ( Laurençon et al. 2024 ) offers a larger scale, comprising 2 million triplets of HTML code, screenshots, and generated descriptions. While these synthesized datasets are critical for enabling large-scale training, they often lack the diversity and

Read the complete text