Quality-Aware Modulation for Diffusion Transformers
Luke Budny, Yuhong Guo, Kevin Cheung · 2026
Read Quality-Aware Modulation for Diffusion Transformers by Luke Budny, Yuhong Guo, Kevin Cheung (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30934.txt.
Introduction
Luke Budny, Yuhong Guo and Kevin Cheung add a quality-aware signal to text-to-image diffusion transformers. These models denoise an image step by step, steered by embeddings that report the current noise level and the prompt. Nothing in that steering says anything about quality, which can leave outputs unaligned with the prompt, inconsistent or low in fidelity. Their Quality Representation Module is a small transformer that reads the existing model inputs and learns a representation of quality, emitting vectors that adjust the adaptive layer normalisation inside each transformer block.
The module leaves the sampling schedule and the diffusion backbone untouched. Experiments include ablations over its training losses and architectures. Results showed consistent image quality improvements over baseline diffusion transformer models, with better prompt fidelity and visual detail in the samples shown.
Catalog record
| CREATOR | Luke Budny, Yuhong Guo, Kevin Cheung |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.30934
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Quality-Aware Modulation for Diffusion Transformers Luke Budny, Yuhong Guo, Kevin Cheung Abstract Modern text-to-image diffusion models, such as diffusion transformers (DiT), rely on timestep or prompt embeddings to modulate the strength of the denoising process in each timestep. While this modulation communicates the current noise level, it does not provide any quality-aware information, which can lead to generated images that are unaligned, visually inconsistent, and lacking in fidelity. In this paper, we propose the Quality Representation Module (QRM), a lightweight transformer module that learns a quality-aware representation based on existing model inputs, and produces a set of vectors M q r m M_{qrm} . These vectors adjust the adaptive LayerNorm modulation within the DiT transformer blocks, thereby injecting a quality-sensitive signal into the denoising parameters. The QRM introduces no significant changes to the sampling schedule or diffusion backbone. Experiments include ablations on QRM training losses and architectures, as well as empirical results demonstrating consistent image quality improvements over baseline DiT-based models. 1 Introduction Figure 1: Images generated using SD3.5 with our Quality-Aware Reward Modulation (QRM). QRM modifies only the AdaLN modulation offsets but produces significantly improved prompt fidelity and visual detail. The field of text-to-image generation is currently dominated by diffusion models [ 3 ] such as DALL·E [ 10 ] , Midjourney [ 9 ] , and Stable Diffusion [ 18 ] , which are capable of producing high-quality prompt-aligned images. These models generate outputs by iteratively denoising latent noise over a sequence of diffusion timesteps, gradually transforming noise into a structured image. Crucial to text-to-image generation, diffusion models condition the denoising process on textual prompts, allowing the generated image to reflect the semantic content of the input description. Recent advances have introduced transformer-based diffusion architectures known as Diffusion Transformers (DiT) [ 11 ] , which replace the convolutional U-Net backbone with transformer blocks. These models incorporate conditioning signals at every diffusion timestep that modulate the internal activations of the network. Modern systems such as Stable Diffusion 3.5 (SD3.5) [ 19 ] adopt this architecture, using learned timestep embeddings and pooled prompt embeddings to control the denoising process. While these conditioning signals communicate information about the prompt and the current noise level of the latent image, they do not provide any indication of the quality or fidelity of the intermediate generated image. In diffusion transformers, this conditioning is applied through modulation layers [ 11 ] . At each diffusion timestep, the conditioning signals are transformed into modulation parameters that scale, shift, and gate the activations of transformer blocks. These parameters adjust the strength of attention and feedforward computations, allowing the model to adapt its denoising behavior based on the prompt and timestep. While this mechanism significantly improves generation quality and stability, the conditioning signals themselves are limited to prompt semantics and timestep information. This work explores whether modulation can be enhanced beyond timestep and prompt embeddings by incorporating a representation of latent image quality. During the diffusion process, visual fidelity and prompt alignment may degrade due to error accumulation, incorrect prompt interpretation, or hallucinated features [ 23 ] . Because the baseline conditioning signals depend only on the timestep and prompt, the modulation mechanism has no way to directly correct these issues based on the evolving state of the latent image. To address this limitation, we introduce the Quality Representation Module (QRM), a lightweight transformer module [ 20 ] that learns a quality-aware representation during diffusion. The QRM receives the baseline modulation inputs used by diffusion transformers (timestep and pooled prompt embeddings), along with latent image features and the baseline modulation parameters M_{base} . Using these inputs, the QRM predicts a set of modulation updates M_{qrm} that refine the baseline modulation parameters at each diffusion timestep. The QRM is trained using a Reward Feedback Learning (ReFL) strategy [ 23 ] , which allows the module to learn how modulation changes influence the semantic alignment between the generated image and its associated prompt. The predicted vectors M_{qrm} are combined with the baseline modulation parameters M_{base} to form the final modulation parameters applied within the transformer blocks. By injecting quality-aware information directly into the modulation pathway, the QRM provides a mechanism for adjusting model activations during the denoising process without modifying the underlying diffusion backbone. Although the proposed method is designed for diffusion transformer architectures in general, we evaluate QRM using Stable Diffusion 3.5 (SD3.5) [ 19 ] as a representative DiT-based model. This allows us to study the effects of quality-aware modulation within a modern large-scale diffusion transformer while keeping the pretrained backbone weights frozen and the sampling schedule unchanged. Our contributions are as follows: • We propose the Quality Representation Module (QRM), a lightweight transformer-based module [ 20 ] that injects quality-aware modulation into diffusion transformer architectures. • We introduce a mechanism for augmenting diffusion transformer modulation layers with latent image quality signals without modifying the underlying diffusion backbone or sampling schedule. • We demonstrate that the predicted modulation updates M_{qrm} refine the denoising process at each timestep, improving prompt adherence and image fidelity. • We perform extensive evaluations and ablations using image evaluation metrics including CLIP Score [ 2 ] , ImageReward [ 23 ] , and Human Preference Score (HPS) v2.1 [ 21 ] to analyze the effects of QRM architecture, input features, and training losses. 2 Related Works 2.1 Diffusion Models for Text-to-Image Generation Diffusion models generate data by progressively transforming Gaussian noise into structured samples through an iterative denoising process [ 3 ] . In text-to-image generation, this denoising trajectory is conditioned on textual prompts, enabling the model to synthesize images that reflect the semantic content of the input description. Latent diffusion models (LDMs) [ 15 ] significantly improved the efficiency of diffusion-based generation by performing the diffusion process in a learned latent space rather than directly in pixel space. This approach uses a pretrained variational autoencoder (VAE) to encode images into compact latent representations, reducing computational cost while preserving semantic structure. Stable Diffusion [ 15 , 17 , 12 ] popularized this paradigm and demonstrated that high-quality images can be generated by conditioning latent diffusion models on text embeddings. More recent systems such as Stable Diffusion 3.5 (SD3.5) [ 19 ] and FLUX [ 5 ] further advance this framework by replacing the traditional convolutional U-Net backbone with Diffusion Transformers (DiT) [ 11 ] . These architectures use transformer blocks with adaptive normalization layers to incorporate conditioning signals throughout the denoising process, enabling more flexible and scalable generative models. 2.2 Diffusion Transformer Architecture Diffusion Transformer (DiT) architectures [ 11 ] replace the convolutional U-Net backbone traditionally used in diffusion models with transformer blocks based on the self-attention architecture introduced in [ 20 ] . Similar to earlier latent diffusion systems, these models typically operate in the latent space of a pretrained variational autoencoder (VAE) [ 4 ] , allowing diffusion to be performed on compact semantic representations rather than full-resolution images. A key component of DiT models is the incorporation of conditioning signals at every denoising step through adaptive normalization mechanisms. Conditioning information is typically derived from two sources: a pooled prompt embedding p , produced by pretrained text encoders such as CLIP [ 13 ] or T5 [ 14 ] ,