Learning Diffusion Models with Flexible Representation Guidance

Chenyu Wang Cai Zhou Sharut Gupta Zongyu Lin, Stefanie Jegelka Stephen Bates Tommi Jaakkola · 2025

Read Learning Diffusion Models with Flexible Representation Guidance by Chenyu Wang Cai Zhou Sharut Gupta Zongyu Lin, Stefanie Jegelka Stephen Bates Tommi Jaakkola (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-08980.txt.

Introduction

Blur to sharp tools learn to make shots from noise, and a guide that aims their mid marks at a taught eye's marks makes them learn fast and well. Chenyu Wang and co-authors set that guide's rules in a 2025 study. Their frame splits the tool's sum in ways that fix when and how the guide's marks join the run.

Two new plays come from the split. One pairs each shot with marks of its own or of a twin kind, and learns the pair as one. The twin sets the school pace twixt mark learn and shot learn. On shot, ware chain, and drug form tasks, the guide both lifts the ware and speeds the school, by a wide span on the head test.

Catalog record

CREATORChenyu Wang Cai Zhou Sharut Gupta Zongyu Lin, Stefanie Jegelka Stephen Bates Tommi Jaakkola
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE112 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-08980

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learning Diffusion Models with Flexible Representation Guidance Chenyu Wang Cai Zhou Sharut Gupta Zongyu Lin, Stefanie Jegelka Stephen Bates Tommi Jaakkola Abstract Diffusion models can be improved with additional guidance towards more effective representations of input. Indeed, prior empirical work has already shown that aligning internal representations of the diffusion model with those of pre-trained models improves generation quality. In this paper, we present a systematic framework for incorporating representation guidance into diffusion models. We provide alternative decompositions of denoising models along with their associated training criteria, where the decompositions determine when and how the auxiliary representations are incorporated. Guided by our theoretical insights, we introduce two new strategies for enhancing representation alignment in diffusion models. First, we pair examples with target representations either derived from themselves or arisen from different synthetic modalities, and subsequently learn a joint model over the multimodal pairs. Second, we design an optimal training curriculum that balances representation learning and data generation. Our experiments across image, protein sequence, and molecule generation tasks demonstrate superior performance as well as accelerated training. In particular, on the class-conditional ImageNet 256 × 256 256\times 256 benchmark, our guidance results in 23.3 23.3 times faster training than the original SiT-XL as well as four times speedup over the state-of-the-art method REPA [ 60 ] . The code is available at https://github.com/ChenyuWang-Monica/REED . 1 Introduction Figure 1: REED achieves superior performance and accelerated training on image generation and protein inverse folding. Diffusion models [ 20 , 49 ] have achieved significant empirical success across images, videos, audio, and biomolecules [ 45 , 6 , 43 , 36 , 1 , 55 ] . While their primary objective is to model the data distribution, recent work [ 10 , 31 , 57 ] show that diffusion models implicitly learn some discriminative features in their latent representations. Nevertheless, the learned representations still fall behind those of pretrained representation learning models, which potentially limits the performance of generative modeling. Consequently, integrating high-quality representations obtained from other pretraining tasks such as Self-supervised Learning (SSL) unlocks additional possibilities to improve diffusion training. Representations can be integrated through flexible designs, for example, via conditioning [ 33 , 34 ] or alignment [ 60 ] . Specifically, RCG [ 33 ] trains a generative model to produce semantic representations from a self-supervised encoder and conditions image generation on these representations, while REPA [ 60 ] aligns diffusion model hidden states with clean representations from pretrained encoders. Despite the empirical effectiveness of these methods, there is a lack of systematic understanding of the role of representation learning in training diffusion models. In this paper, we present a theoretical framework to analyze how high-quality, pretrained representations can enhance diffusion model training. Building on the DDPM [ 20 ] framework, we derive a variational bound on the negative log-likelihood that incorporates the pretrained representations to guide the generation through a probabilistic decomposition of the joint distribution and a weighted aggregation process. This leads to two key components: representation modeling , which promotes extracting representations from noisy input, and data generation , which fuses representation-conditioned and unconditional generation into a hybrid model governed by a weight schedule ( Section 2.1 ). The framework generalizes to multi-latent structures, enabling integration of diverse representations ( Section 2.2 ). Our unified perspective subsumes existing approaches like REPA [ 60 ] and RCG [ 33 ] , offering theoretical insight into representation-enhanced diffusion models. In particular, REPA emerges as a special case with a linear weight aggregation schedule and a single latent, where the output of the hybrid model is approximated by conditioning solely on representations extracted from noisy inputs to mitigate the training-inference gap ( Section 2.3 ). We also provide a TV distance error bound for the sample distribution of representation-aligned diffusion models ( Section C.2 ). Built on the theoretical insights, our general framework allows for a much broader design space for exploiting representation alignment [ 60 ] in diffusion models. In particular, we demonstrate the effectiveness of two novel strategies, designated as REED ( R epresentation- E nhanced E lucidation of D iffusion). First, we argue that diffusion models can profit from diverse representation sources, hence we incorporate multi-latent representations from different modalities to provide complementary information. Our new multimodal alignment scheme uses synthetic paired data and multimodal pretrained models trained on datasets from different corpora using discriminative or self-supervised tasks to generate the target representations ( Section 3.1 ). Second, we analyze optimal training curricula for representation-aligned diffusion models, demonstrating that representation learning is crucial in the early training stage to yield substantial advantages. Consequently, we introduce a novel curriculum that applies the representation learning loss from the outset, while progressively increasing the diffusion loss coefficient from zero to its maximum value as a phase-in protocol ( Section 3.2 ). Finally, we propose three practical instantiations of our framework that yield improvements in image, protein, and molecule generation ( Section 3.3 ). We extensively evaluate our proposed REED across image generation, protein inverse folding, and molecule generation tasks. For the class-conditional ImageNet 256\times 256 benchmark, aligning SiT-XL/2 with DINOv2 [ 41 ] and Qwen2-VL [ 54 ] representations using our improved curriculum significantly improves the generation quality measured by FID scores. REED achieves a 23.3\times training speedup over the original SiT-XL, reaching FID=8.2 in only 300K training iterations (without classifier-free guidance [ 19 ] ); and a 4\times speedup over REPA [ 60 ] , matching its classifier-free guidance performance at 800 epochs with only 200 epochs of training (FID=1.80). For protein inverse folding, aligning discrete diffusion models trained on the PDB dataset with AlphaFold3 [ 1 ] structure and sequence representations accelerates training by 3.6\times and yields significantly superior performance across metrics such as sequence recovery rate, RMSD and pLDDT. For molecule generation, aligning state-of-the-art geometric equivariant flow matching (SemlaFlow [ 24 ] ) with pretrained Unimol [ 62 ] representations improves metrics such as atom and molecule stability, validity, energy, and strain on the challenging Geom-Drug [ 3 ] datasets. We summarize our main contributions as follows: • We conduct rigorous analysis for representation-enhanced diffusion model training, delivering theoretical insights into prior empirical methods and informing the design of new strategies. • We systematically investigate the design space for representation guidance in training diffusion models, proposing a novel multimodal representation alignment scheme via synthesized paired multimodal data, an effective training curriculum, and practical domain-specific instantiations. • Extensive experiments demonstrate that our method consistently accelerates training and improves performance across various diffusion model types and diverse tasks, including image, protein sequence, and molecule generation tasks. 2 Theoretical Characterizations In this section, we introduce a general theoretical framework for representation-enhanced diffusion models. Building on the DDPM framework [ 20 ] , we explore how representations can offer additional information and enhance the generation process. Notably, due to the known equivalence between DDPM, score matching [ 49 ] , stochastic interpolants [ 2 ] , and flow matching [ 37 ] , our insights are applicable to a broad class of diffusion model variants. Proofs for this section are detailed in Appendix C . 2.1 Variational Bound with Decomposed Probabilistic Framework Problem Formulation. We consider a generative modeling setting involving discrete-time latent

Read the complete text