OverLayBench: A Benchmark for Layout-to-Image Generation with Dense Overlaps

Bingnan Li Chen-Yu Wang Haiyang Xu Xiang Zhang Ethan Armand, Divyansh Srivastava Xiaojun Shan Zeyuan Chen Jianwen Xie Zhuowen Tu · 2025

Read OverLayBench: A Benchmark for Layout-to-Image Generation with Dense Overlaps by Bingnan Li Chen-Yu Wang Haiyang Xu Xiang Zhang Ethan Armand, Divyansh Srivastava Xiaojun Shan Zeyuan Chen Jianwen Xie Zhuowen Tu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-19282.txt.

Introduction

Ask an image generator to follow a layout of boxes, and it copes until the boxes overlap heavily. Bingnan Li and nine co-authors study that failure. Dense overlaps hurt in two ways: large shared regions, and overlapping objects that look nearly alike. Their analysis shows existing benchmarks mostly contain easy, low-overlap layouts, so models look better than they are. They define a score that measures how complex a layout's overlaps are.

Their new benchmark, OverLayBench, balances cases across overlap levels with high-quality annotations. Scores on it expose how sharply quality drops as overlap grows. As a first step toward a fix, the team fine-tunes a model on a curated dataset of amodal masks, which show each object's full shape even where hidden. The work lays groundwork for layout-following generators that survive realistic, crowded scenes.

Catalog record

CREATORBingnan Li Chen-Yu Wang Haiyang Xu Xiang Zhang Ethan Armand, Divyansh Srivastava Xiaojun Shan Zeyuan Chen Jianwen Xie Zhuowen Tu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-19282

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

OverLayBench: A Benchmark for Layout-to-Image Generation with Dense Overlaps Bingnan Li Chen-Yu Wang Haiyang Xu Xiang Zhang Ethan Armand, Divyansh Srivastava Xiaojun Shan Zeyuan Chen Jianwen Xie Zhuowen Tu Abstract Despite steady progress in layout-to-image generation, current methods still struggle with layouts containing significant overlap between bounding boxes. We identify two primary challenges: (1) large overlapping regions and (2) overlapping instances with minimal semantic distinction. Through both qualitative examples and quantitative analysis, we demonstrate how these factors degrade generation quality. To systematically assess this issue, we introduce OverLayScore, a novel metric that quantifies the complexity of overlapping bounding boxes. Our analysis reveals that existing benchmarks are biased toward simpler cases with low OverLayScore values, limiting their effectiveness in evaluating model performance under more challenging conditions. To bridge this gap, we present OverLayBench, a new benchmark featuring high-quality annotations and a balanced distribution across different levels of OverLayScore. As an initial step toward improving performance on complex overlaps, we also propose CreatiLayout-AM, a model fine-tuned on a curated amodal mask dataset. Together, our contributions lay the groundwork for more robust layout-to-image generation under realistic and challenging scenarios. Project link: https://mlpc-ucsd.github.io/OverLayBench . Figure 1 : Examples from OverLayBench with difficulty increasing from left to right. 1 Introduction Figure 2 : Failure cases from state-of-the-art layout-to-image models. Each row presents an example with overlapping instances, and image captions are shown below. More examples and more detailed failure descriptions can be found in Appendix C . With the advancement of text-to-image generative models ( Ramesh et al. 2021 ; Nichol et al. 2021 ; Rombach et al. 2022 ; Chen et al. 2023 ; Xue et al. 2024 ) , there has been growing interest in controllable image generation ( Li et al. 2023b ; Zhang et al. 2023 ) . A recent line of work proposes generating images conditioned on layouts, commonly referred to as Layout-to-Image (L2I) generation, which allows users to directly specify spatial locations ( Xie et al. 2023b ; Wang et al. 2024b ; Li et al. 2023b ) and object counts ( Binyamin et al. 2024 ; Yang et al. 2023 ) in the generated outputs. While existing frameworks ( Xie et al. 2023b ; Wang et al. 2024b ; Li et al. 2023b ) can achieve satisfactory spatial and numerical control over image generation, these approaches fail to generate distinct, coherent objects when multiple bounding boxes overlap in layout and their associated categories are semantically similar. As illustrated in Figure 2 , such scenarios lead to artifacts including object blending, spatial ambiguity, and visual distortion. To quantify the effect of bounding box overlap on generation, we introduce OverLayScore ( Equation 1 ), a simple metric that captures the difficulty of generation based on spatial and semantic overlap between bounding boxes. OverLayScore is computed as the sum of IoUs for all instance pairs, weighted by their semantic similarity, measured via the dot product of CLIP embeddings of instance annotations. We empirically demonstrate that generation quality degrades with higher OverLayScore, i.e., large overlap between bounding boxes and high category-level semantic similarity. Henceforth, we will interchangeably refer to layouts with high OverLayScore score as complex layouts and low OverLayScore score as simple layouts. Existing benchmarks for Layout-to-Image (L2I) generation ( Cheng et al. 2024 ; Zhang et al. 2024 ) primarily focus on image quality, offering limited evaluation of complex layouts or the accuracy of generated instance relationships. Our analysis reveals a strong bias towards simple layouts in these benchmarks, which restricts their utility in assessing model performance under more challenging, realistic scenarios. To address this gap, we introduce OverLayBench , a new benchmark specifically designed to evaluate L2I models on their ability to reconstruct complex layouts and instance-level relationships. OverLayBench features rich annotations such as detailed image and dense instance captions, enrichment of complex images with higher OverLayScore, improved semantic grounding of bounding boxes using Qwen ( Bai et al. 2023 ) , and quality instance relationships for evaluation. We conduct extensive evaluations of state-of-the-art L2I models ( Li et al. 2023a ; Wang et al. 2024a ; Zhou et al. 2024a ; Cheng et al. 2024 ; Zhou et al. 2024b ; Zhang et al. 2025 ; Zhang et al. 2024 ) on OverLayBench and verify their effectiveness in addressing layout-level challenges. These results provide strong baselines and highlight areas for improvement. Finally, we demonstrate that fine-tuning CreatiLayout ( Zhang et al. 2024 ) with amodal mask supervision on complex layouts helps mitigate generation artifacts caused by instance occlusion. This new baseline, CreatiLayout-AM , provides initial evidence that explicit mask-level guidance improves generation quality under high-overlap conditions, offering a promising direction for future research. Our contributions are summarized as follows: (1) We propose OverLayScore , a novel metric that empirically quantifies the difficulty of L2I generation by measuring the IoU and semantic similarity between bounding boxes in layouts; (2) We introduce OverLayBench , a challenging benchmark with high-quality annotations and balanced difficulty distribution, designed to evaluate complex relationships between instances in layouts; (3) We demonstrate that training with amodal masks helps alleviate generation artifacts in overlapping regions. Specifically, we present a simple yet effective baseline that aligns attention maps in diffusion models with amodal mask supervision. 2 Related Work 2.1 Layout-to-Image Generation Generative models have recently become highly popular ( Wang et al. 2024d ; Zhao et al. 2025 ; Zeng et al. 2025 ; Srivastava et al. 2025a ; Xu et al. 2024 ; Wang et al. 2024c ; Chen et al. 2025 ) , and controllable generation is attracting growing interest. In particular, Layout-to-Image (L2I) generation has gained attention as it enables structured and spatially grounded image synthesis. Prior work on L2I generation has primarily focused on fine-tuning foundational text-to-image (T2I) generative models ( Labs 2024 ; Esser et al. 2024 ; Podell et al. 2023 ) , introducing various techniques to inject layout conditioning into pre-trained architectures. U-Net-based approaches ( Li et al. 2023a ; Zhou et al. 2024a ; Wang et al. 2024a ; Wu et al. 2024 ; Cheng et al. 2024 ; Zheng et al. 2023 ) typically incorporate layout information through layer insertions (either in series or parallel) or by manipulating attention maps and masks. While these methods have shown promising results, their generation quality is often constrained by the representational capacity of U-Net backbones. More recent methods ( Zhang et al. 2025 ; Zhang et al. 2024 ) leverage powerful diffusion Transformer (DiT) architectures ( Peebles and Xie 2023 ) , fine-tuning models such as Flux ( Labs 2024 ) or SD3 ( Esser et al. 2024 ) , and achieve improved image fidelity. These layouts can be provided by the user or generated automatically from text using text-to-layout models ( Feng et al. 2023a ; Gani et al. 2023 ; Lian et al. 2023 ; Srivastava et al. 2025b ) , forming a two-stage pipeline for controllable image synthesis. In parallel, a growing line of training-free approaches ( Xie et al. 2023a ; Gani et al. 2023 ; Yang et al. 2024b ; Lee et al. 2024 ; Chen et al. 2024a ; Chen et al. 2024c ) has emerged, which utilize guidance mechanisms within diffusion models to enforce spatial

Read the complete text