Improving Compositional Generation with Diffusion Models Using Lift Scores

Chenning Yu, Sicun Gao · 2025

Read Improving Compositional Generation with Diffusion Models Using Lift Scores by Chenning Yu, Sicun Gao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13740.txt.

Introduction

Chenning Yu and Sicun Gao present a 2025 score to help image models meet all parts of a mixed prompt at once. Diffusion models can draw fine images, but they often drop one trait when a prompt asks for two or more. Checking each trait one by one, then joining the checks, could steer the draw toward full matches.

Their score gauges how much a sample lifts the odds of each single trait, then composes those lifts. A key point is that the score can be guessed from the base model alone, with no extra training or side tools. A tuned form keeps the added cost low at draw time. Tests on made-up scenes, a visual logic task, and text-to-image draws all showed clear gains in match to the full prompt.

Catalog record

CREATORChenning Yu, Sicun Gao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE75 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-13740

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Improving Compositional Generation with Diffusion Models Using Lift Scores Chenning Yu, Sicun Gao Abstract We introduce a novel resampling criterion using lift scores, for improving compositional generation in diffusion models. By leveraging the lift scores, we evaluate whether generated samples align with each single condition and then compose the results to determine whether the composed prompt is satisfied. Our key insight is that lift scores can be efficiently approximated using only the original diffusion model, requiring no additional training or external modules. We develop an optimized variant that achieves relatively lower computational overhead during inference while maintaining effectiveness. Through extensive experiments, we demonstrate that lift scores significantly improved the condition alignment for compositional generation across 2D synthetic data, CLEVR position tasks, and text-to-image synthesis. Our code is available at rainorangelemon.github.io/complift . 1 Introduction Despite the success of diffusion models in generating high-quality images ( Sohl-Dickstein et al., 2015 ; Ho et al., 2020 ; Nichol & Dhariwal, 2021 ; Song et al., 2020 ) , they sometimes fail to produce coherent and consistent results when the condition is complex or involves multiple objects ( Huang et al., 2023 ; Feng et al., 2022 ; Chefer et al., 2023 ) . This limitation is particularly evident in tasks that generate the images with specific combinations of attributes or objects, where the generated samples may only partially satisfy the conditions or exhibit inconsistencies across different attributes. To mitigate this issue, several methods have been proposed to improve the compositional generation with diffusion models. These methods can be described as 2 main categories: (1) training-based methods, which train or finetune the diffusion model to improve its ability to generate samples that satisfy complex prompts ( Huang et al., 2023 ; Bao et al., 2024 ) , and (2) training-free methods, which either refine the sampling process, or apply rejection sampling to refine the generated samples without modifying the underlying model ( Feng et al., 2022 ; Chefer et al., 2023 ) . These 2 categories of methods are not necessarily mutually exclusive, and they can be combined to achieve better performance. We introduce CompLift , a training-free rejection criterion for improving compositional generation in diffusion models. The concept of lift score ( Brin et al., 1997 ) builds on a simple observation: if a generated sample truly satisfies a condition, the model should perform better at denoising it when given that condition compared to unconditional denoising. This insight leads to an efficient criterion for accepting or rejecting samples. Similar to Diffusion Classifier ( Li et al., 2023 ) , we show that lift scores can be efficiently approximated using only the original diffusion model’s predictions, requiring no additional training or external modules. By decomposing the satisfaction of a complex prompt into the satisfaction of multiple simpler sub-conditions using lift scores, our approach can boost the model’s alignment with complex prompts without changing the sampling process. Through extensive experiments on 2D synthetic data, CLEVR position tasks, and text-to-image generation, we demonstrate that CompLift significantly improves compositional generation while maintaining computational efficiency. Contributions. Our contributions are threefold: ❶ We introduce CompLift , as a novel resampling criterion using lift scores for compositional generation, requiring no additional training ( Section 3 ). ❷ We explore the design space of CompLift , including noise sampling strategies, timestep selection, and caching techniques to optimize computational efficiency ( Section 4 ). ❸ We scale our approach to text-to-image generation, demonstrating its effectiveness in improving compositional generation in this challenging domain ( Sections 5 and 6 ). 2 Related Work Recent studies have revealed significant limitations in diffusion models ( Dhariwal & Nichol, 2021 ) for handling compositional generation, particularly as missing objects, incorrect spatial relationships, and attribute inconsistencies ( Huang et al., 2023 ; Feng et al., 2022 ; Chefer et al., 2023 ) . Current approaches to address these challenges fall into several categories. One line of work leverages the cross-attention mechanisms, modifying attention values during synthesis to emphasize overlooked tokens ( Feng et al., 2022 ; Chefer et al., 2023 ; Wu et al., 2023 ; Wang et al., 2024 ) . Another approach introduces layout control for precise spatial arrangement of objects ( Feng et al., 2024 ; Chen et al., 2024 ) . While effective, these methods require specific model architectural knowledge, unlike our model-agnostic approach. The diffusion process itself can also be modified to improve compositional generation. Composable Diffusion ( Liu et al., 2022 ) guides generation by combining individual condition scores. Discriminator Guidance ( Kim et al., 2022 ) employs an additional discriminator to reduce the distribution mismatch of the generated samples. Recent methods introduce advanced sampling techniques like Markov Chain Monte-Carlo ( Du et al., 2023 ) , Restart Sampling ( Xu et al., 2023 ) , rejection sampling ( Na et al., 2024 ) , and particle filtering ( Liu et al., 2024 ) . We complement these methods by providing a technique that can be integrated without assumptions about the underlying sampling process. Most closely related to our work are resampling strategies that operate as either selection of the final image ( Karthik et al., 2023 ) or search of the initial noise ( Qi et al., 2024 ; Ma et al., 2025 ) . However, these methods typically require external models for quality assessment. In contrast, our approach leverages the denoising properties of the diffusion model itself ( Li et al., 2023 ; Chen et al., 2023 ; Kong et al., 2023 ) , eliminating the need for additional models. Figure 1: An illustration of product, mixture, and negation compositional models, and the improved sampling performance using CompLift . Left to right: Component distributions, ground truth composed distribution, composable diffusion samples, samples accepted by CompLift . Top: product, center: mixture, bottom: negation. \varnothing represents the empty set - no samples are generated or accepted. Each component distribution is trained independently using a 2D score-based diffusion model. Accuracy is evaluated based on whether generated samples fall into the support or within the 3\sigma region of the composed distribution (details in Section 6 , Appendix E ). Our work is not the first to apply lift scores to diffusion models. Condition Alignment Score ( Hong et al., 2024 ) proposes an equivalence to rank the alignment of images with a whole single prompt. Kong et. al ( Kong et al., 2023 ) regards lift scores as point-wise mutual information for the explanability of the generated images. In this work, we investigate the use of lift scores as compositional criterion for rejection and resampling in compositional generation. 3 Using Lift Scores for Rejection Sampling 3.1 Diffusion Model Preliminaries Diffusion model is a class of generative models that generate samples by denoising a noisy input iteratively. Given a clean sample {\bm{x}}_{0} , the forward diffusion process q({\bm{x}}_{t}|{\bm{x}}_{t-1}) sequentially adds Gaussian noise to {\bm{x}}_{t-1} at each timestep t , whereas the learned reverse diffusion process p_{\theta}({\bm{x}}_{t-1}|{\bm{x}}_{t},{\bm{c}}) denoises {\bm{x}}_{t} to reconstruct {\bm{x}}_{t-1} , optionally conditioned on a condition {\bm{c}} . The conditional probability of {\bm{x}}_{0} can be defined as: p_{\theta}({\bm{x}}_{0}|{\bm{c}})=\int_{{\bm{x}}_{1:T}}p({\bm{x}}_{T})\prod_{t=1}^{T}p_{\theta}({\bm{x}}_{t-1}|{\bm{x}}_{t},{\bm{c}})\,d{\bm{x}}_{1:T}, where p({\bm{x}}_{T}) is typically a standard Gaussian \mathcal{N}(\mathbf{0},\mathbf{I}) . The diffusion model is often parameterized as a neural network \epsilon_{\theta} to represent the denoising function.

Read the complete text