7Bench: a Comprehensive Benchmark for Layout-guided Text-to-image Models

Elena Izzo✉, Luca Parolari, Davide Vezzaro Lamberto Ballan · 2025

Read 7Bench: a Comprehensive Benchmark for Layout-guided Text-to-image Models by Elena Izzo✉, Luca Parolari, Davide Vezzaro Lamberto Ballan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-12919.txt.

Introduction

Elena Izzo and co-authors present in 2025 7Bench, a test set for image makers that take a layout map as a guide. Layout guides give more hold over where things sit in a made shot. No past test set scored both word match and layout match as one.

Word and layout pairs span seven hard scenes, from thing counts to hue truth and space links. A new layout score joins past text scores in the grade plan. Top spread models were run through the set, and their strong and weak zones came out by scene type. The set aims to guard fake data grade.

Catalog record

CREATORElena Izzo✉, Luca Parolari, Davide Vezzaro Lamberto Ballan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-12919

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

7Bench: a Comprehensive Benchmark for Layout-guided Text-to-image Models Elena Izzo✉, Luca Parolari, Davide Vezzaro Lamberto Ballan Abstract Layout-guided text-to-image models offer greater control over the generation process by explicitly conditioning image synthesis on the spatial arrangement of elements. As a result, their adoption has increased in many computer vision applications, ranging from content creation to synthetic data generation. A critical challenge is achieving precise alignment between the image, textual prompt, and layout, ensuring semantic fidelity and spatial accuracy. Although recent benchmarks assess text alignment, layout alignment remains overlooked, and no existing benchmark jointly evaluates both. This gap limits the ability to evaluate a model’s spatial fidelity, which is crucial when using layout-guided generation for synthetic data, as errors can introduce noise and degrade data quality. In this work, we introduce 7Bench, the first benchmark to assess both semantic and spatial alignment in layout-guided text-to-image generation. It features text-and-layout pairs spanning seven challenging scenarios, investigating object generation, color fidelity, attribute recognition, inter-object relationships, and spatial control. We propose an evaluation protocol that builds on existing frameworks by incorporating the layout alignment score to assess spatial accuracy. Using 7Bench, we evaluate several state-of-the-art diffusion models, uncovering their respective strengths and limitations across diverse alignment tasks. The benchmark is available at https://github.com/Elizzo/7Bench . 1 Introduction Layout-guided text-to-image models extend the standard text-to-image generation paradigm by introducing explicit layout control. Among the various layout representations, bounding boxes have emerged as the most commonly adopted format due to their simplicity and ease of annotation. This format enables users to intuitively specify the position, scale, and appearance of objects, offering a more controllable generation process. Such fine-grained control over the image content fosters new opportunities for leveraging synthetic data, consequently increasing the demand and usage of layout-guided diffusion models in other computer vision tasks, e.g . synthetic dataset generation [ 17 , 11 ] . Despite the high potential of these tools, recent studies have identified several limitations [ 6 ] . In fact, models often struggle to generate images that accurately adhere to the given layout and textual prompt, especially when given uncommon spatial arrangements [ 26 ] . Additionally, they inherit fundamental issues from diffusion models, including catastrophic neglect, attribute binding, and attribute leaking, which affect both text and layout alignment. Quantifying and identifying errors in text-image and layout-image faithfulness is crucial to properly evaluate and compare different layout-guided diffusion models. However, benchmarks collected for evaluating textual and layout alignment are still missing. As a result, the strategies adopted by the researchers to evaluate layout-guided generative models are heterogeneous and mainly based on traditional text-to-image benchmarks [ 1 , 9 , 23 ] . These include tests for text-image alignment in scenarios involving colors, attributes, or relations, but either lack any dedicated evaluation of layout alignment or suffer from inconsistencies among different evaluation protocols. For instance, layout alignment is assessed using a random subset of annotated datasets for object detection or grounding [ 13 , 3 , 19 ] or by researchers collecting their own sets of box-and-text pairs [ 26 ] . A recent effort [ 4 ] introduces the evaluation of spatial skills ( e.g . position, size or shape) but overlooks semantic evaluation. This lack of a unified approach leads to a significant misalignment among studies in terms of evaluation metrics, the number of samples investigated, and the specific layout skills being tested. In this paper, we propose 7Bench, a comprehensive benchmark for layout-guided models characterized by pairs of textual prompts and sets of bounding boxes to evaluate models on 7 different scenarios. The aim is to investigate models’ abilities in objects’ generation, colors, attributes, relationships, and spatial control. We design a structured prompt collection annotated with bounding boxes, collecting a dataset with 224 samples equally distributed in the 7 scenarios. We introduce an evaluation method based on two complementary measures: the text-alignment and the layout-alignment scores. Our evaluation protocol extends an existing framework that focuses on text-image alignment by introducing an assessment of layout accuracy. We systematically evaluate state-of-the-art layout-guided text-to-image diffusion models using our benchmark, comparing both fine-tuned models on grounding information and training-free approaches. Results obtained on 7Bench demonstrate room for improvement in all models, validating the scientific rationale of our approach. In summary, our main contributions include: (i) 7Bench, a novel benchmark for layout-guided text-to-image models, designed to evaluate both text-image and layout-image alignment across seven diverse and challenging scenarios; (ii) a meticulous collection of 224 text and bounding boxes pairs, ensuring a broad coverage of various semantic and spatial arrangements; (iii) an evaluation protocol based on two complementary measures: text-alignment and layout-alignment scores, which assess the alignment of generated images with respect to both textual descriptions and spatial layouts; and (iv) a comprehensive evaluation of four state-of-the-art layout-guided text-to-image diffusion models using 7Bench, providing insights into the strengths and weaknesses of different approaches, including a comparison between fine-tuned and training-free models. 2 Related Works Benchmarks for Text-to-image Generation. Text-to-image generation is the process of creating images from textual descriptions. The rise of diffusion models [ 8 ] revolutionised this task, outperforming traditional GAN-based approaches [ 21 ] in generating high-quality and diverse images. Despite the outstanding performance, these models often struggle to produce images that accurately align with the description, failing to bind attributes or specific details to objects [ 6 ] . As a result, benchmarks and evaluation metrics have been introduced to measure the faithfulness of a generated image to its text input on various challenging scenarios: colors, attributes, counting, relations, or objects [ 9 , 10 , 1 , 25 , 5 , 23 ] . However, with the introduction of layout-guided text-to-image models, all these benchmarks miss two key elements: the input layouts of the object described in the textual prompt and an evaluation metric to quantify the layout alignment. Layout-guided Text-to-image Diffusion Models. Layout-guided text-to-image diffusion models incorporate predefined spatial layouts for finer control over image generation. Generally, these methods inject grounding information into pre-trained models by adding new trainable layers or tokens [ 13 , 27 ] , or manipulating cross-attention layers at inference time to steer the generation process without requiring additional training [ 26 , 3 , 19 ] . Evaluating these models requires assessing both text and layout alignment, but the absence of comprehensive benchmarks has led researchers to develop their own evaluation strategies. Evaluation of Layout-guided Diffusion Models. The evaluation of layout-guided models is often a two-step framework assessing text and layout faithfulness separately. Text alignment is evaluated by benchmarks from the traditional text-to-image domain. Instead, layout alignment is often assessed by generating images from bounding boxes and captions sampled from COCO2014 [ 15 ] or Flickr30K Entities [ 20 ] validation sets. The generation quality is then quantified with metrics like FID [ 7 ] and YOLO score [ 14 ] . However, this evaluation is heterogeneous among papers and does not follow a shared protocol. For instance, in [ 13 ] a subset of 30 k samples are generated while in [ 19 ] only 5 k. Moreover, in [ 26 ] , the authors point out the need for a distinct evaluation dataset, independent of COCO images, since it is commonly used to

Read the complete text