ComposeAnything: Composite Object Priors for Text-to-Image Generation

Zeeshan Khan, Shizhe Chen, Cordelia Schmid · 2025

Read ComposeAnything: Composite Object Priors for Text-to-Image Generation by Zeeshan Khan, Shizhe Chen, Cordelia Schmid (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24086.txt.

Introduction

Zeeshan Khan, Shizhe Chen, and Cordelia Schmid help image generators place many things in one scene as the words ask. Models garble hard layouts, above all in depth. Their 2025 frame, ComposeAnything, plans the scene first, then paints it.

A language model drafts a layout with boxes, depth notes, and rich captions for each thing. From that plan, the team builds a rough collage of the things in place. The collage seeds the painting run in place of random noise. Two guides keep the paint true to the plan while still letting it fix plan slips. Tests on hard prompt sets beat top rivals. Human judges also scored the scenes high for match to the words.

Catalog record

CREATORZeeshan Khan, Shizhe Chen, Cordelia Schmid
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-24086

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ComposeAnything: Composite Object Priors for Text-to-Image Generation Zeeshan Khan, Shizhe Chen, Cordelia Schmid Abstract Generating images from text involving complex and novel object arrangements remains a significant challenge for current text-to-image (T2I) models. Although prior layout-based methods improve object arrangements using spatial constraints with 2D layouts, they often struggle to capture 3D positioning and sacrifice quality and coherence. In this work, we introduce ComposeAnything , a novel framework for improving compositional image generation without retraining existing T2I models. Our approach first leverages the chain-of-thought reasoning abilities of LLMs to produce 2.5D semantic layouts from text, consisting of 2D object bounding boxes enriched with depth information and detailed captions. Based on this layout, we generate a spatial and depth aware coarse composite of objects that captures the intended composition, serving as a strong and interpretable prior that replaces stochastic noise initialization in diffusion-based T2I models. This prior guides the denoising process through object prior reinforcement and spatial-controlled denoising, enabling seamless generation of compositional objects and coherent backgrounds, while allowing refinement of inaccurate priors. ComposeAnything outperforms state-of-the-art methods on the T2I-CompBench and NSR-1K benchmarks for prompts with 2D/3D spatial arrangements, high object counts, and surreal compositions. Human evaluations further demonstrate that our model generates high-quality images with compositions that faithfully reflect the text. Figure 1: The proposed ComposeAnything framework enables text-to-image generation for complex compositions involving surreal spatial relationships and high object counts. It enhances both visual quality and faithfulness to the input text compared to diffusion-based models ( e.g. , SD3 [ 11 ] , FLUX [ 3 ] ) and 2D layout conditioned models ( e.g. , RPG [ 56 ] and CreatiLayout [ 58 ] ). 1 Introduction Text-to-image (T2I) models, particularly diffusion-based ones such as SDXL [ 45 ] , SD3 [ 11 ] and Flux [ 3 ] , have achieved remarkable success in generating individual concepts with high fidelity. However, they struggle with complex object compositions [ 21 ] , especially novel arrangements that deviate from their training distribution, often resulting in unnatural mixing of objects, incorrect 2D/3D spatial positioning, and inaccurate object counts, as shown in Figure 1 . To improve compositional T2I generation, layout control has emerged as a key strategy [ 13 , 29 , 41 ] . This approach leverages various 2D layout representations, such as bounding boxes [ 29 ] and blobs [ 41 ] , typically generated by large language models (LLMs) [ 42 ] from textual prompts. Layout-controlled T2I methods can be broadly categorized into training-based and training-free approaches. Training-based methods [ 58 , 29 , 59 , 52 ] incorporate layout conditioning by training adapter modules over pretrained T2I models. While they provide strong spatial control, they require extensive training and often degrade image coherence and quality [ 60 ] due to rigid layout constraints, as seen with CreatiLayout [ 58 ] in Figure 1 . In contrast, training-free methods mainly apply inference-time layout control [ 56 ] or latent optimization [ 10 ] , to guide spatial aware generation, such as manipulating region-to-text cross-attention maps [ 4 , 10 ] or employing region-wise denoising [ 56 ] . These methods better preserve image quality but offer weaker control, making it difficult to follow complex compositional instructions such as RPG [ 56 ] in Figure 1 . Furthermore, all existing training-free methods rely solely on coarse 2D layouts, which not only lack 3D spatial relationships but also fail to visually represent object appearance, limiting their effectiveness in guiding T2I generation. To address these limitations, we propose ComposeAnything , a novel inference-time T2I framework that enhances compositional generation without requiring model retraining. Our approach leverages LLMs with chain-of-thought reasoning to decompose the input text into a 2.5D semantic layout comprising objects and background. Each object is represented by a caption, a 2D bounding box, and a depth value to reflect relative 3D spatial relations. We then synthesize a coarse image by generating individual object images from their captions with existing T2I models [ 11 ] and arranging them spatially according to this layout. This results in a composite object prior that captures object appearance, count, size and 2.5D spatial positioning. To effectively utilize it we replace random noise initialization with noisy object prior and propose a prior-guided diffusion module that combines object prior reinforcement and spatial-controlled denoising. The prior reinforcement preserves the influence of foreground object priors in early diffusion steps, while automatically generating background to enable coherent generation. The spatial-controlled denoising further strengthens the spatial arrangement of the composite prior via mask-guided attention in early diffusion steps where global structure is determined. After these initial steps, we switch to standard diffusion to refine the image quality and coherence, achieving both faithful composition and high visual fidelity. ComposeAnything outperforms state-of-the-art methods on two challenging compositional T2I benchmarks: T2I-CompBench [ 21 ] and NSR-1K [ 13 ] , under automatic metrics. It achieves both high image quality and strong faithfulness to input text, as further validated by human evaluations. Ablation studies highlight the effectiveness of the composite object prior and prior-guided diffusion. To summarize, our contributions are as follows: (i) We introduce ComposeAnything , a training-free framework that enhances the compositional capabilities of existing diffusion-based T2I models by generating structured 2.5D semantic layouts and composite object priors from text using LLMs with chain-of-thought reasoning. (ii) We propose a prior-guided diffusion method that integrates composite object priors into the denoising process via object prior reinforcement and spatially-controlled denoising in initial diffusion steps to balance faithful object compositions and high-quality image synthesis. (iii) ComposeAnything enables interpretable and robust image generation, and outperforms the state of the art on two compositional T2I benchmarks, especially for surreal compositions. We will publicly release our code. 2 Related Works Compositional generation aims to produce images that are faithfully aligned with complex texts [ 21 , 60 , 22 , 25 , 52 , 13 , 56 , 53 , 9 , 31 ] . While diffusion models [ 45 , 11 , 3 ] have demonstrated strong generative capabilities, they still struggle with compositional generation. To address this, prior work in this area can be broadly categorized into two main directions. Training-based methods fine-tune pretrained diffusion models [ 45 , 11 , 3 , 5 , 6 , 43 ] on large-scale datasets to improve semantic alignments [ 7 , 57 ] . Some approaches introduce additional training objectives, such as grounding loss [ 53 ] or text-to-image alignment [ 23 ] , to strengthen compositional accuracy, while others enrich semantic features by using LLM-generated information [ 19 ] . To incorporate explicit spatial control, training-based layout-controlled methods train adapter modules to inject new conditioning for spatial control [ 29 , 59 , 52 , 14 , 57 , 58 , 32 , 61 , 62 , 15 , 40 , 27 ] such as bounding boxes, segmentation masks, and keypoints. Despite their effectiveness, these methods face two key limitations: a) large-scale training is required to learn object compositions comprehensively; and b) hard conditioning often degrades image quality as indicated in [ 60 ] . Training-free methods provide an alternative by manipulating diffusion models at inference time, without requiring retraining. Inspired by text-based image editing methods

Read the complete text