Sketch-to-Layout: Sketch-Guided Multimodal Layout Generation
Riccardo Brioschi, Aleksandr Alekseev, Emanuele Nevali, Berkay Döner, Omar El Malki Blagoj MitrevskiLeandro Kieliger Mark Collier Andrii Maksai Jesse Berent Claudiu Cristian Musat Efi Kokiopoulou · 2025
Read Sketch-to-Layout: Sketch-Guided Multimodal Layout Generation by Riccardo Brioschi, Aleksandr Alekseev, Emanuele Nevali, Berkay Döner, Omar El Malki Blagoj MitrevskiLeandro Kieliger Mark Collier Andrii Maksai Jesse Berent Claudiu Cristian Musat Efi Kokiopoulou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-27632.txt.
Introduction
Riccardo Brioschi and eleven co-authors introduce sketch-guided layout design in 2025. Layout generation builds neat pages, posters, and slides from set parts. Past tools guide that work with detailed written rules, which are slow and hard for users to write. A hand sketch gives a simpler guide: users draw rough boxes to show where parts should go.
Their model joins the sketch with the content parts in one transformer build, a neural design that weighs links among inputs. Hand sketches for training are scarce, so the team made fake sketches at scale. Tests on three public layout sets show the sketch guide works well. Rough drawings could make design tools far easier to steer.
Catalog record
| CREATOR | Riccardo Brioschi, Aleksandr Alekseev, Emanuele Nevali, Berkay Döner, Omar El Malki Blagoj MitrevskiLeandro Kieliger Mark Collier Andrii Maksai Jesse Berent Claudiu Cristian Musat Efi Kokiopoulou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-27632
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Sketch-to-Layout: Sketch-Guided Multimodal Layout Generation Riccardo Brioschi, Aleksandr Alekseev, Emanuele Nevali, Berkay Döner, Omar El Malki Blagoj MitrevskiLeandro Kieliger Mark Collier Andrii Maksai Jesse Berent Claudiu Cristian Musat Efi Kokiopoulou Abstract Graphic layout generation is a growing research area focusing on generating aesthetically pleasing layouts ranging from poster designs to documents. While recent research has explored ways to incorporate user constraints to guide the layout generation, these constraints often require complex specifications which reduce usability. We introduce an innovative approach exploiting user-provided sketches as intuitive constraints and we demonstrate empirically the effectiveness of this new guidance method, establishing the sketch-to-layout problem as a promising research direction, which is currently under-explored. To tackle the sketch-to-layout problem, we propose a multimodal transformer-based solution using the sketch and the content assets as inputs to produce high quality layouts. Since collecting sketch training data from human annotators to train our model is very costly, we introduce a novel and efficient method to synthetically generate training sketches at scale. We train and evaluate our model on three publicly available datasets: PubLayNet [ 43 ] , DocLayNet [ 32 ] and SlidesVQA [ 35 ] , demonstrating that it outperforms state-of-the-art constraint-based methods, while offering a more intuitive design experience. In order to facilitate future sketch-to-layout research, we release O(200k) synthetically-generated sketches for the public datasets above. 1 1 1 The datasets are available at https://github.com/google-deepmind/sketch_to_layout \footglue =.4em plus.15em minus.15em Figure 1 : Our sketch-to-layout approach leverages sketches to guide the generation of multimodal layouts in a natural and intuitive way. 1 Introduction Designing aesthetically pleasing and usable layouts for graphic design is a fundamental challenge. Layouts should represent a visually pleasing arrangement of text and image elements with appropriate sizes and positions, while at the same time capturing the right information hierarchy. Assets should have consistent semantic relationships such as an engaging reading order. Manual design can be time-consuming and automated layout generation aims to reduce this burden. Recent research has explored various approaches to layout generation, including image generation methods such as GANs [ 21 , 18 ] and LLM-based methods [ 36 , 26 , 41 ] . Some of these approaches try to incorporate user-defined constraints to guide the generation, but require complex details such as specifying precise element dimensions [ 26 , 16 ] , complex positional relationships [ 26 , 16 , 41 , 18 ] , grid-based guidelines [ 6 ] and detailed textual descriptions [ 25 ] . It is cumbersome for users to come up with such complex constraints. On the contrary, sketching, a common design practice where users quickly outline the structure of a design, offers a more intuitive alternative. Sketching is a widely used technique for creative tasks that captures the high-level essence of a layout without requiring overwhelming detail. Studies on designer behavior [ 30 , 31 ] show that sketching is an integral starting point for almost all designers in various domains [ 4 ] . This paper proposes using sketches as intuitive constraints to guide the generation of multimodal (image and text) layouts. We start by demonstrating empirically the effectiveness of sketches as a new guidance method when compared against other forms of user-defined constraints. This empirical result suggests that the sketch-to-layout problem is a promising direction for constrained layout generation, which has been under-explored. To tackle the sketch-to-layout problem, we propose a solution leveraging Vision-Language Models (VLMs); see also Fig 1 . Our method takes as input a user constraint in a form of a sketch along with image and text assets, to produce visually pleasing high quality layouts, capturing the structure suggested by the user while maintaining aesthetic appeal. Although VLMs have shown impressive performances on a wide range of tasks, recent research [ 23 ] demonstrated that generating the correct layout from sketch inputs in a single step is challenging even for state-of-the-art VLMs, which amplifies the need of collecting sketch training data for improving performance. However, collecting sketch data from human annotators to train a VLM model is very costly and time-consuming. In order to address this challenge, we propose a novel and efficient technique to synthetically generate sketches at scale, unblocking the fine-tuning of VLMs to tackle the sketch-to-layout problem. In order to accelerate research progress on sketch-to-layout, we release a dataset of O(200k) synthetic sketches generated by our proposed method. Our VLM-based approach is general and applicable to any VLM. In our empirical study we use PaLIGemma 3B [ 3 ] as an example open-source VLM. We train and evaluate our model on three publicly available datasets: PubLayNet [ 43 ] , DocLayNet [ 32 ] and SlidesVQA [ 35 ] , demonstrating that it outperforms state-of-the-art constraint-based methods, while offering a more intuitive design experience. By evaluating our model on both synthetic and human-produced sketches, we arrive at comparable performance, which validates the use of synthetic sketches as a reliable proxy for actual human-produced sketches when used as training data for VLMs in the sketch-to-layout task. In summary, our contributions are as follows: • We demonstrate the value of sketches as a novel guidance method for layout generation, establishing the sketch-to-layout as an effective research direction for guided layout generation. • We introduce a novel methodology to create large-scale synthetic datasets with sketches of documents and layouts, unblocking efficient VLM training and evaluation. • We release our large collection of O(200k) synthetically generated sketches for three publicly available datasets, in order to facilitate future research in this previously data-scarce domain. • We provide experimental results showing that our method leveraging PaLIGemma outperforms state-of-the-art constraint-based layout generation methods by more than 40% in terms of Maximum IoU on three publicly available datasets. Our empirical results also highlight the importance of the content-awareness aspect of our method. • We introduce Content Ordering Score (COS), a new metric inspired by the order loss [ 22 ] , designed to assess the content-awareness of a generated layout. 2 Related Work Unconstrained Generation. Early research in layout generation primarily focused on unconditional generation. CanvasVAE [ 40 ] models documents as a combination of canvases and elements, adopting a VAE to capture the distribution of their attributes. Gupta et al. 2021 propose an auto-regressive transformer to frame layout generation as a sequence-to-sequence task, and show the effectiveness of their approach on different domains. Constrained Generation. We focus on constrained generation of layouts, which has been investigated before with different types of constraints. LayoutVAE [ 17 ] proposes a two-stage VAE model that takes the set of labels as input constraint. Similarly, LayoutGAN [ 21 ] synthesizes layouts given the set of labels with each label having a separate probability distribution in the generator. Further work from Li et al. 2020b uses the area, aspect ratio and reading-order of the input elements as the input constraints. [ 19 ] takes relational constraints between elements (such as specifying a text block to be on the left of an image) and models the relationships using a graph-based model. [ 18 ] uses a transformer-based GAN that can additionally take beautification constraints, such as alignment and non-overlap. Later work using diffusion models [ 13 , 6 ] , transformers [ 16 ] , and LLMs