VSC: Visual Search Compositional Text-to-Image Diffusion Model

Do Huu Dat 1* Nam Hyeonu 2 Po-Yuan Mao 3 Tae-Hyun Oh 2,4 · 2025

Read VSC: Visual Search Compositional Text-to-Image Diffusion Model by Do Huu Dat 1* Nam Hyeonu 2 Po-Yuan Mao 3 Tae-Hyun Oh 2,4 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-01104.txt.

Introduction

Do Huu Dat and co-authors present in 2025 a text to image method that keeps colors and traits glued to the right objects. Diffusion models paint striking pictures from prompts, yet with several trait and object pairs they often swap the bindings: the red car and blue bike come out crossed. Common text encoders blur exactly those links.

Their model, VSC, breaks a hard prompt into sub prompts and paints each piece first. Visual prototypes drawn from those drafts are fused with the text encoding to steady the meaning. Training with segmentation based localization further lines up the model's attention with the right regions. On a standard composition benchmark the method beat rival compositional models, earned better marks from human judges, and held up as prompts stacked more binding pairs. Faithful trait binding matters anywhere generated images must match a spec.

Catalog record

CREATORDo Huu Dat 1* Nam Hyeonu 2 Po-Yuan Mao 3 Tae-Hyun Oh 2,4
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-01104

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VSC: Visual Search Compositional Text-to-Image Diffusion Model Do Huu Dat 1* Nam Hyeonu 2 Po-Yuan Mao 3 Tae-Hyun Oh 2,4 Abstract Text-to-image diffusion models have shown impressive capabilities in generating realistic visuals from natural-language prompts, yet they often struggle with accurately binding attributes to corresponding objects, especially in prompts containing multiple attribute-object pairs. This challenge primarily arises from the limitations of commonly used text encoders, such as CLIP, which can fail to encode complex linguistic relationships and modifiers effectively. Existing approaches have attempted to mitigate these issues through attention map control during inference and the use of layout information or fine-tuning during training, yet they face performance drops with increased prompt complexity. In this work, we introduce a novel compositional generation method that leverages pairwise image embeddings to improve attribute-object binding. Our approach decomposes complex prompts into sub-prompts, generates corresponding images, and computes visual prototypes that fuse with text embeddings to enhance representation. By applying segmentation-based localization training, we address cross-attention misalignment, achieving improved accuracy in binding multiple attributes to objects. Our approaches outperform existing compositional text-to-image diffusion models on the benchmark T2I CompBench, achieving better image quality, evaluated by humans, and emerging robustness under scaling number of binding pairs in the prompt. Figure 1 : Left: Original pipeline and generated image of Diffusion Model. Right: VSC individually “search” for visual information of each binding pair by generating images for each separate pair (“red car” and “yellow bicycle”). Finally, VSC uses them as references to generate the correct compositional image. * * footnotetext: Work is done as an intern at POSTECH 1 Introduction Compositionality is a fundamental ability that enables humans to naturally comprehend the world. It allows us to quickly recognize new objects through their components or construct complex sentences by combining words. This skill empowers us to make “infinite use of finite mean” [ 6 ] , endlessly reusing and recombining familiar concepts in diverse ways. In terms of developing compositional AI systems, diffusion models rise as the most feasible component due to their inherent enormous mode coverage and flexible controllability. The diffusion model’s possession of compositional generalization has been proved with synthetic data and the dataset CelebA, where the images are easily decomposed into gender and race [ 22 ] . Additionally, diffusion models for generating images from text prompts are capable of creating remarkably realistic visuals, allowing users to shape the content of generated images by crafting detailed and complex natural-language prompts. Despite these capabilities, these models often fail to fully utilize compositionality and produce inaccurate images when provided with more intricate descriptions. A prevalent issue is improper binding, as illustrated in the left side of Fig. 1 , where descriptive words fail to correctly influence the visual characteristics of specific nouns they are intended to modify. For instance, the prompt specifying “A photo of a red car and a yellow bicycle” may produce an image with mixed or misplaced attributes. One likely cause for these inconsistencies is the reliance on text encoders like CLIP within diffusion models. Although CLIP demonstrates strong performance in aligning images with textual descriptions, it faces significant challenges in handling complex linguistic structures. Specifically, it struggles with encoding hierarchical relationships and modifiers, which are crucial for understanding nuanced textual inputs. Previous studies [ 17 , 30 , 29 ] have highlighted that CLIP tends to simplify textual inputs by representing them as a “bag-of-concepts,” disregarding the intricate dependencies and relationships between words. These inherent weaknesses pose limitations for diffusion models that rely on CLIP for text encoding, as they hinder the model’s capacity to fully capture and respect the precise linguistic relationships specified in prompts. Serial search theory [ 31 ] in cognitive psychology suggests that when humans encounter objects with multiple features (e.g., color and shape), our brain processes them individually to bind these features into a cohesive perception. The theory has been successfully applied to improve the capability of large vision-language models [ 3 ] . Noticeably, CLIP likewise strongly materializes compositionality in the single binding setting [ 17 ] and this property is well-preserved in the latent diffusion model (refer to Section 6.1 ). Providing spatial information such as bounding box, segmentation maps, blobs, [ 7 , 21 , 36 , 10 ] can drastically improve the compositional generation by controlling the attention to be bounded in a specific regions while maintaining a reasonable layout. Those methods can be interpreted as disentangling the desired image into object-level generation. This leads to our motivation to generate images from every single attribute-object pair as references for improving the compositional generalization of the text-to-image diffusion model. On the other hand, the diffusion model can merge and edit images, namely subject-driven image generation, by quickly fine-tuning the model [ 26 ] that encodes a unique identifier that refers to the subject. Notably, Xiao et al. [ 34 ] propose a tuning-free method via training the diffusion model alongside an image encoder and an MLP. Besides, training an MLP has also been shown to be effective in improving the compositionality of the text-to-image model. Therefore, taking the psychological theory as inspiration, with the above previous works in diffusion models, our key idea, illustrated on the right side of Fig. 1 , is to develop a framework that generates images for each binding pair individually and ultimately fuses these images into a complete image from a given prompt containing several attribute-object binding pairs. The fusing process requires the use of an external MLP layer, so we create a synthetic dataset and fine-tune the MLP. Moreover, we extract the segmentation maps of the object and deploy cross-attention localization training as in [ 34 ] to fix the erroneous cross-attention maps, aiming to address the extreme decrease while increasing the number of attribute-object pairs. Our main contributions are summarized as follows: • We present VSC, a novel compositional text-to-image diffusion generation method based on a tuning-free, subject-driven image generation framework. Remarkably, VSC requires training only an MLP and the final layers of the image encoder. • We leverage the remarkable ability of the pre-trained text-to-image diffusion model in handling single attribute binding prompts to create synthetic images as reference images throughout training and inference. • We achieve state-of-the-art performance on the attribute binding benchmark compared to other compositional text-to-image diffusion models, and provide extensive analysis as well as ablation studies. 2 Related Work Compositional Text-to-Image Diffusion Model : In recent work, Huang et al. [ 13 ] introduce a benchmark to test compositionality in text-to-image models, highlighting the vulnerability of open-source models to simple compositional prompts. They also propose a fine-tuning approach to enhance compositionality in these models. Several existing studies suggest different methods for enhancing compositionality in text-to-image models in both inference time and training time. In terms of inference time, one popular way is controlling the attention maps of the cross-attention layers [ 24 , 32 , 19 , 1 ] , which would ensure the maps of binding tokens are overlapping. However, because the attention scores are illustrated to be erroneous [ 35 ] , then scaling the number of pairs leads to a drastic decrease in performance in those works. Furthermore, combining the guidance in a structured approach [ 18 , 9

Read the complete text