CompAlign: Improving Compositional Text-to-Image Generation with a Complex Benchmark and Fine-Grained Feedback

Yixin Wan, and, Kai-Wei Chang · 2025

Read CompAlign: Improving Compositional Text-to-Image Generation with a Complex Benchmark and Fine-Grained Feedback by Yixin Wan, and, Kai-Wei Chang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11178.txt.

Introduction

Yixin Wan and Kai-Wei Chang present a benchmark and feedback method for compositional text-to-image generation, in their 2025 arXiv paper. Image models render single subjects well but stumble on prompts with several objects, attributes and spatial relations. Their benchmark, CompAlign, holds nine hundred complex prompts stressing number counts and 3D spatial layouts.

Their evaluation framework, CompQuest, breaks each prompt into atomic sub-questions and has a multimodal model answer each one about a generated image, giving fine-grained feedback. That feedback then serves as preference signals to train diffusion models. Testing nine image models showed compositional tasks, especially 3D spatial ones, remain hard, and open models trail closed ones.

Catalog record

CREATORYixin Wan, and, Kai-Wei Chang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-11178

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CompAlign: Improving Compositional Text-to-Image Generation with a Complex Benchmark and Fine-Grained Feedback Yixin Wan, and, Kai-Wei Chang Abstract State-of-the-art Text-to-Image models (e.g., DALL-E) are capable of generating high-resolution images given textual prompts. However, these models still struggle with accurately depicting compositional scenes that specify multiple objects, attributes, and spatial relations. We present CompAlign , a challenging benchmark with an emphasis on assessing the depiction of 3D-spatial relationships, for evaluating and improving models on compositional image generation. CompAlign consists of 900 complex multi-subject image generation prompts that combine numerical and 3D-spatial relationships with varied attribute bindings. Compared with prior works, our benchmark is remarkably more challenging, incorporating generation tasks with 3+ generation subjects with complex 3D-spatial relationships. Additionally, we propose CompQuest , an interpretable and accurate evaluation framework that first decomposes complex prompts into atomic sub-questions, then utilizes a Multimodal Large Language Model to provide fine-grained binary feedback on the correctness of each aspect of generation elements in model-generated images. This enables precise quantification of alignment between generated images and compositional prompts. Furthermore, we propose an alignment framework that uses CompQuest’s feedback as preference signals to improve diffusion models’ compositional image generation abilities. Using adjustable per-image preferences, our method is easily scalable and flexible for different tasks. Using CompAlign, we evaluate 9 state-of-the-art text-to-image models. Results show that: (1) models remarkable struggle more with compositional tasks with more complex 3D-spatial configurations, and (2) a noticeable performance gap exists between open-source accessible models and closed-source commercial models. Further empirical study on using CompAlign for model alignment yield promising results: post-alignment diffusion models achieve remarkable improvements in compositional accuracy, especially on complex generation tasks, outperforming previous approach across different compositional settings. 1 Introduction Despite impressive recent progress, Text-to-Image (T2I) models still frequently fall short in accurately visualizing details in compositional prompts. For instance, examples in Figure 1 show how state-of-the-art models like Stable Diffusion 3 [ 5 ] and DALL-E 3 [ 18 ] struggle with depicting multiple subjects in compositional instructions involving numeracy, 3D-spatial positionings, and attribute bindings like color and texture. Such drawbacks in T2I models greatly hinder downstream applications such as design and entertainment, which require better textual control over nuanced and complicated details in generated images. Prior works [ 12 , 13 ] proposed approaches to assess compositional T2I abilities, but face 2 major limitations: (1) they mostly evaluate models on simple T2I tasks depicting 2 objects, and does not include more challenging settings like depicting 3+ objects with 3D-spatial relationships. (2) their metrics rely on opaque model-generated scores, lacking both interpretability and robustness in evaluation outcomes. Figure 1: Example showing that while T2I models are capable of depicting simple compositional settings in a previous benchmark, they fall short in correctly generating images that accurately conform to complex compositional instructions in our challenging CompAlign benchmark. To bridge these gaps, we introduce CompAlign , a more challenging benchmark for advanced evaluation of compositional T2I generation. CompAlign consists of 900 complex generation prompts for multi-subject image generation, combining numerical, 3D-spatial, and attribute bindings to reflect realistic complexity in image generation tasks. We further propose CompQuest , a vision-language-based evaluation framework that utilizes 3 key steps— Atomic Question Decomposition , Atomic Multimodal Large Language Model (MLLM) Feedback , and Aggregated Compositional Accuracy —to achieve fine-grained evaluation of the compositional image generation task. Using CompAlign, we conducted evaluation experiments on 9 state-of-the-art T2I models. Results show that: (1) most models still struggle with accurately depicting compositional scenes, especially for tasks involving multiple generation subjects and complex 3D-spatial positioning configurations; (2) remarkable performance gaps can be observed between open-source and closed-source models. This indicates that limitations on computational resources and the quality of openly accessible data remain a challenge for building openly accessible generative intelligence. We further explore methods to improve open-source T2I models on complex compositional generation tasks. Previous works have leveraged human preference optimization to boost model performance. However, for compositional images with multiple objects associated with different attributes and relations, relying on simple human preferences is not enough to provide sufficient guidance for diffusion models. In cases where multiple objects and relationships are present, it might be subjective to decide whether a partially-correct generated image is “preferred” over another one. To address this limitation, we construct an alignment framework that adjustably adopts aggregated atomic feedback from CompQuest as per-image preference signals for improving diffusion models on compositional image generation. Experimental results with our alignment method on open-weight diffusion models demonstrate substantial improvements in compositional accuracy, outperforming previous methods for improving the compositional abilities of diffusion models. 2 CompAlign Benchmark Assessing the compositionality of T2I models, or the ability to accurately and coherently depict multiple concepts in a unified scene based on textual instructions, has been an increasingly challenging research topic [ 13 ] . For instance, Huang et al. [12] , Huang et al. [13] constructed textual prompts with simple attribute bindings and spatial relationships. Their evaluation benchmark mostly consists of straightforward generation prompts involving 2 subjects—for example, “a boy near a airplane” , as shown in Figure 1 . However, the figure shows that recent T2I models like SD2 and DALL-E 3 already achieve satisfactory performance on such simple generation tasks. To further challenge recent stronger T2I models on more complex compositional image generation tasks, we propose the CompAlign benchmark , which consists of a variety of detailed image generation prompts involving multiple objects, 3D-spatial relationships, and precise attributes such as color and texture. Our benchmark notably extends the complexity and scope of Huang et al. [13] ’s work through an emphasis on generation tasks for compositional scenarios that depict up to 6 subjects, combined with specific attribute bindings and 3D-spatial positioning requirements. Such complicated tasks challenge T2I models to understand and accurately depict not only multiple entities with different features, but also the nuanced 3-D positioning of them in the generated scenes. 1 1 1 We will release our code and data under the CC By 4.0 License upon acceptance. 2.1 Prompt Construction We adopt a hierarchical heuristic-based pipeline to generate reasonable and diverse compositional image generation prompts. Specifically, the prompt construction pipeline begins with assigning top-level numerical and 3D-spatial relationships between potential entities. Then, we iterate through a variety of candidate entities and corresponding attributes to be combined with every assignable 3D-spatial positioning in each generation setting. 2.1.1 Numerical and 3D-Spatial Relationships To create generation settings with complex 3D-spatial positioning of entities, we manually define 5 distinct 3D-spatial configurations, escalating in both numerical and spatial complexity: 1 row \times 2 subjects, 1 row \times 3 subjects, 2 rows \times 1 subject, 2 rows \times 2 subjects, and 2 rows \times 3 subjects. We phrase these positional configurations into natural prompt templates for further combination with generation subjects. For instance, for the configuration with 1 row and 2 subjects, the corresponding prompt template is: “An image with 2 objects: a { (optional) attribute A } { subject A } on the left, and a { (optional) attribute B } { subject B } on the right.” Specifications of the spatial attributes are detailed in the first section of Appendix A , Table 3 . The final CompAlign benchmark is equally divided between the 5 numerical and 3D-spatial positioning assignments.

Read the complete text