Aligning Text to Image in Diffusion Models is Easier Than You Think
Jaa-Yeon Lee, Byunghee Cha, Jeongsol Kim, Jong Chul Ye · 2025
Read Aligning Text to Image in Diffusion Models is Easier Than You Think by Jaa-Yeon Lee, Byunghee Cha, Jeongsol Kim, Jong Chul Ye (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08250.txt.
Introduction
Lee and co-authors offer a lighter way to make text-to-image models follow their prompts more faithfully. Even strong diffusion models sometimes produce pictures that miss details in the wording. Existing fixes often need costly fine-tuning on specially built datasets.
Their method, SoftREPA, fine-tunes the model with a contrastive approach that reuses the training data as both matching and mismatched text-image pairs. It adds fewer than one million trainable parameters, a tiny fraction of the model, yet improves the link between text and image features. Tests on image generation and text-guided editing showed better semantic consistency. The appeal is practical: better prompt alignment at a fraction of the usual computing cost.
Catalog record
| CREATOR | Jaa-Yeon Lee, Byunghee Cha, Jeongsol Kim, Jong Chul Ye |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-08250
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Aligning Text to Image in Diffusion Models is Easier Than You Think Jaa-Yeon Lee, Byunghee Cha, Jeongsol Kim, Jong Chul Ye Abstract While recent advancements in generative modeling have significantly improved text-image alignment, some residual misalignment between text and image representations still remains. Some approaches address this issue by fine-tuning models in terms of preference optimization, etc., which require tailored datasets. Orthogonal to these methods, we revisit the challenge from the perspective of representation alignment—an approach that has gained popularity with the success of REPresentation Alignment (REPA) [ 46 ] . We first argue that conventional text-to-image (T2I) diffusion models, typically trained on paired image and text data (i.e., positive pairs) by minimizing score matching or flow matching losses, is suboptimal from the standpoint of representation alignment. Instead, a better alignment can be achieved through contrastive learning that leverages existing dataset as both positive and negative pairs. To enable efficient alignment with pretrained models, we propose SoftREPA —a lightweight contrastive fine-tuning strategy that leverages soft text tokens for representation alignment. This approach improves alignment with minimal computational overhead by adding fewer than 1M trainable parameters to the pretrained model. Our theoretical analysis demonstrates that our method explicitly increases the mutual information between text and image representations, leading to enhanced semantic consistency. Experimental results across text-to-image generation and text-guided image editing tasks validate the effectiveness of our approach in improving the semantic consistency of T2I generative models. Project Page: https://softrepa.github.io/ . Figure 1: Representative results for image generation and image editing . SoftREPA provides much improved text-to-image alignment by introducing a negligible size of learnable soft tokens. 1 Introduction Achieving effective alignment between modalities is essential in multimodal generative modeling. Specifically, latent diffusion models [ 33 ] enable various conditioning mechanisms, such as text, during the generation process for the semantic alignment between the two modalities. For UNet-based denoisers [ 33 , 29 , 31 ] , image representations are updated to align with fixed text representations encoded by pre-trained CLIP models through cross-attention layers. In contrast, transformer-based denoisers [ 30 , 8 ] jointly update both text and image representations by concatenating them and processing them through self-attention layers. However, there is still room for further improvement, and to mitigate the remaining misalignment between text and image representations, a more effective representation learning approach is necessary. Recent advancements in representation alignment for Diffusion Transformers (DiT) [ 30 ] have notably enhanced their ability to learn semantically meaningful internal representations [ 46 , 11 ] . Notably, REPA [ 46 ] showed that aligning the internal representations of DiT with an external pre-trained visual encoder during training significantly improves both discriminative and generative performance. To further enhance text-image alignment in text-to-image (T2I) generative models, we leverage these ideas for aligning DiT’s internal representations. Most vision-language generative foundation models have focused on improving multimodal alignment through architectural modifications. These include MLP-based projection layers (e.g., LLaVA-style visual instruction tuning) [ 25 , 26 ] , cross-attention-based fusion mechanisms (e.g., Flamingo, Stable Diffusion 1.5) [ 2 , 29 ] , and early fusion models that integrate text and vision features within a shared representation space (e.g., Chameleon, Stable Diffusion 3, FLUX) [ 37 , 8 ] . In fact, many of these approaches have been largely adopted in T2I generative models. However, we explore an efficient yet effective way of further enhancing the representation alignment for given pre-trained models. Specifically, we focus on the contrastive learning framework, a widely adopted strategy for multimodal alignment in vision-language representation learning, as seen in models such as CLIP [ 32 ] , ALBEF [ 21 ] , BLIP [ 22 ] , and ALIGN [ 12 ] . One of the key contributions of this work is the introduction of soft text tokens , which are optimized via contrastive image-text training. These soft tokens allow the model to dynamically adapt its text representations, improving alignment with generated images without requiring full model fine-tuning. This simple yet effective approach significantly enhances text-to-image alignment in both text-to-image generation and text-guided image editing tasks. Notably, the method is highly flexible and can be seamlessly integrated with any pretrained text-to-image (T2I) generative model. From a theoretical standpoint, we show that this approach explicitly increases the mutual information between text and image representations, resulting in improved semantic consistency across modalities. We refer to our method as SoftREPA , short for Soft REPresentation Alignment. Our contributions can be summarized as follows. • We propose SoftREPA, a novel text-image representation alignment method that leverages a lightweight fine-tuning strategy with soft text tokens. This approach improves text-image alignment while adding fewer than 1M additional parameters, ensuring efficiency with minimal computational overhead. • SoftREPA is simple yet flexible so that it can be used with any pretrained T2I generative models to improve performance of image generation, editing, etc. • We show that our method explicitly increases mutual information between image and text, leading to better semantic consistency in multi-modal representations. 2 Preliminaries Flow Models Suppose that we have access to samples from target distribution X_{0}\sim q and source distribution X_{1}\sim p . The goal of flow model is to generate X_{0} starting from X_{1} . Specifically, we define a velocity field v_{t}({\bm{x}}) of a flow \psi_{t}({\bm{x}}):[0,1]\times\mathbb{R}^{d}\rightarrow\mathbb{R}^{d} that satisfies \psi_{t}(X_{0})=X_{t} and \psi_{1}(X_{0})=X_{1} . Here, the \psi_{t} is uniquely characterized by a flow ODE: \displaystyle d\psi_{t}({\bm{x}})=v_{t}(\psi_{t}({\bm{x}}))dt (1) where the flow velocity v_{t} is fitted to the parameterized neural network v_{t,\theta} via flow matching: \displaystyle\mathcal{L}_{FM}=\mathbb{E}_{t\in[0,1],{\bm{x}}_{t}\sim p_{t}}\|v_{t}({\bm{x}}_{t})-v_{t,\theta}({\bm{x}}_{t})\|^{2}. (2) However, this is computationally expensive to solve due to the integration with respect to X_{0} , so the authors in [ 24 ] proposed a conditional flow matching that has the same gradient with the original objective function: \displaystyle\mathcal{L}_{CFM}=\mathbb{E}_{t\in[0,1],{\bm{x}}_{0}\sim q}\|v_{t}({\bm{x}}_{t}|{\bm{x}}_{0})-v_{t,\theta}({\bm{x}}_{t})\|^{2}, (3) where v_{t}({\bm{x}}_{t}|{\bm{x}}_{0}) defines a conditional flow \psi_{t}({\bm{x}}_{t}|{\bm{x}}_{0}) satisfying \psi_{t}({\bm{x}}_{1}|{\bm{x}}_{0})={\bm{x}}_{t} . In particular, the linear conditional flow defines the flow as {\bm{x}}_{t}=\psi_{t}({\bm{x}}_{1}|{\bm{x}}_{0})=(1-t){\bm{x}}_{0}+t{\bm{x}}_{1} . Then, the conditional velocity field is given by v_{t}({\bm{x}}_{t}|{\bm{x}}_{0})=\dot{\psi}_{t}(\psi_{t}^{-1}({\bm{x}}_{t}|{\bm{x}}_{0})|{\bm{x}}_{0})={\bm{x}}_{1}-{\bm{x}}_{0} [ 24 ] . Thus, the conditional flow matching loss is defined as \displaystyle\mathbb{E}_{t\in[0,1],{\bm{x}}_{0},{\bm{x}}_{1}\sim\pi_{0,1}}\|({\bm{x}}_{1}-{\bm{x}}_{0})-v_{t,\theta}({\bm{x}}_{t})\|^{2}. (4) Considering a marginal velocity field, \displaystyle v_{t}({\bm{x}}_{t}) \displaystyle=\int v_{t}({\bm{x}}_{t}|{\bm{x}}_{0})p({\bm{x}}_{0}|{\bm{x}}_{t})d{\bm{x}}_{0}=\mathbb{E}[v_{t}({\bm{x}}_{t}|{\bm{x}}_{0})|{\bm{x}}_{t}] (5) \displaystyle=\mathbb{E}[{\bm{x}}_{1}-{\bm{x}}_{0}|{\bm{x}}_{t}]=\mathbb{E}[{\bm{x}}_{1}|{\bm{x}}_{t}]-\mathbb{E}[{\bm{x}}_{0}|{\bm{x}}_{t}], the generation process via flow ODE is \displaystyle d{\bm{x}}_{t}=v_{t}({\bm{x}}_{t})dt=(\mathbb{E}[{\bm{x}}_{1}|{\bm{x}}_{t}]-\mathbb{E}[{\bm{x}}_{0}|{\bm{x}}_{t}])dt. (6) Accordingly, d{\bm{x}}_{t}=\left(\mathbb{E}\left[\frac{{\bm{x}}_{t}-(1-t){\bm{x}}_{0}}{t}|{\bm{x}}_{t}\right]-\mathbb{E}[{\bm{x}}_{0}|{\bm{x}}_{t}]\right)dt=\frac{{\bm{x}}_{t}-\mathbb{E}[{\bm{x}}_{0}|{\bm{x}}_{t}]}{t}dt (7) which is the probability-flow ODE (PF-ODE) of DDIM [ 35 , 15 ] when {\bm{x}}_{1}\sim p:=\mathcal{N}(0,\bm{I}_{d}) . In other words, flow models and score-based models can be used interchangeably. Contrastive Representation Learning Contrastive learning aims to maximize the similarity between semantically related text-image pairs while pushing apart unrelated pairs in a shared representation space [ 6 , 32 ] . Formally, given a batch of N image-text pairs {(\bm{I}_{i},\bm{T}_{i})}^{N}_{i=1} , contrastive learning optimizes a contrastive loss function, typically a variant of the InfoNCE loss: \mathcal{L}_{\text{CLIP}}=-\frac{1}{N}\sum^{N}_{i=1}\log\frac{\exp(\text{sim}(\bm{I}_{i},\bm{T}_{i})/\tau)}{\sum^{N}_{j=1}\exp(\text{sim}(\bm{I}_{i},\bm{T}_{j})/\tau)} (8) where \text{sim}(\bm{I}_{i},\bm{T}_{i}) is a similarity metric such as cosine similarity or inner product between image and text embeddings, and \tau denotes a temperature parameter that controls the distribution sharpness. This contrastive loss encourages image and text representations to form a joint multimodal representation space, where aligned pairs are close together, and unaligned pairs are separated. Several fundamental models have successfully applied contrastive learning for multi-modal representation learning [ 32 , 12 , 21 , 22 ] and self-supervised learning [ 6 ] . Despite the success of contrastive learning in representation learning, its application in