Canonical Latent Representations in Conditional Diffusion Models
Yitao Xu, Tong Zhang, Ehsan Pajouheshgar, Sabine Süsstrunk · 2025
Read Canonical Latent Representations in Conditional Diffusion Models by Yitao Xu, Tong Zhang, Ehsan Pajouheshgar, Sabine Süsstrunk (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-09955.txt.
Introduction
Yitao Xu and co-authors find compact class summaries hiding inside conditional diffusion models. Their 2025 paper notes that these generators learn whole data distributions, which tangles the features that define a class with background clutter. The team defines canonical latent codes that keep the defining signal and drop the rest.
Decoding such a code yields a clean representative sample of its class, an interpretable summary of what the class means to the model. Their CaDistill method then uses only these codes, about a tenth of the training data, to teach a student network. The student ends up tougher against attacks and better at generalizing, attending to class signal over spurious cues. The authors reframe diffusion models as compact teachers, not just image factories.
Catalog record
| CREATOR | Yitao Xu, Tong Zhang, Ehsan Pajouheshgar, Sabine Süsstrunk |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 92 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-09955
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Canonical Latent Representations in Conditional Diffusion Models Yitao Xu, Tong Zhang, Ehsan Pajouheshgar, Sabine Süsstrunk Abstract Conditional diffusion models (CDMs) have shown impressive performance across a range of generative tasks. Their ability to model the full data distribution has opened new avenues for analysis-by-synthesis in downstream discriminative learning. However, this same modeling capacity causes CDMs to entangle the class-defining features with irrelevant context, posing challenges to extracting robust and interpretable representations. To this end, we identify C anonical LA tent Rep resentations (CLAReps), latent codes whose internal CDM features preserve essential categorical information while discarding non-discriminative signals. When decoded, CLAReps produce representative samples for each class, offering an interpretable and compact summary of the core class semantics with minimal irrelevant details. Exploiting CLAReps, we develop a novel diffusion-based feature distillation paradigm, CaDistill . While the student has full access to the training set, the CDM as teacher transfers core class knowledge only via CLAReps, which amounts to merely 10% of the training data in size. After training, the student achieves strong adversarial robustness and generalization ability, focusing more on the class signals instead of spurious background cues. Our findings suggest that CDMs can serve not just as image generators but also as compact, interpretable teachers that can drive robust representation learning. 1 Introduction Diffusion models (DMs) excel at generative modeling of images [ 16 , 28 , 58 , 65 , 66 ] . When conditioned on class labels [ 4 , 27 , 56 ] or text prompts [ 57 , 58 ] , conditional diffusion models (CDMs) faithfully generate samples with desired characteristics of the condition. This generative capability has sparked a wave of analysis-by-synthesis approaches [ 5 , 10 , 44 , 51 , 53 , 78 , 83 , 91 , 95 ] , where DMs are used to probe or improve downstream discriminative tasks. However, a key challenge remains: since DMs model the full data distribution, they often encode redundant or irrelevant information, which can obscure the discriminative signal. For example, in the Tench row of Figure 1 , modifying the CDM latent code of the training sample changes the angler in the background while the fish remains almost unchanged, showing that the model encodes background cues that correlate with the class but not semantically essential to it. This entanglement between class semantics and extraneous factors limits interpretability and hinders the effective use of CDMs in representation learning. This motivates our central question: How can we identify the underlying core categorical semantics in a conditional diffusion model? Figure 1 : Conditional diffusion models (CDMs) encode both core class features and irrelevant context in a CDM. Left : training samples. Middle : samples obtained by modifying CDM latent codes that preserve the class label but alter class-irrelevant parts. Right : Canonical Samples produced by CLARID, which retain the class-defining content while removing extraneous context . CLARID benefits the extraction of robust and interpretable representations from the CDMs. We answer this by introducing C anonical LA tent R epresentation ID entifier ( CLARID ), a method for identifying the latent codes in CDMs that capture essential categorical information and filter out irrelevant details. We call the resulting latent codes C anonical LA tent Rep resentations (CLAReps). We begin with the assumption that the essential semantics of a class typically lie on a low-dimensional manifold embedded in the high-dimensional data space [ 12 , 72 , 92 ] . We postulate that such semantic manifolds also exist within the latent space of diffusion models. Our key insight is that altering the latent code along the tangent directions of the manifold, referred to as extraneous directions, modifies visual appearance without affecting class identity. We find that projecting out the extraneous directions in the latent space of CDMs effectively eliminates class-irrelevant factors such as background clutter or co-occurring objects from other categories. When projected back to the data space, CLAReps produce representative samples of each category, namely Canonical Samples , providing an intuitive and interpretable summary of the essential categorical semantics. Additionally, the internal CDM features of CLAReps, i.e. Canonical Features , contain mostly the core class information. We first validate our method in a toy hierarchical generative model, where CLARID recovers a low-dimensional class manifold while standard classifier-free guidance (CFG) produces dispersed, high-likelihood samples. Scaling up, we apply CLARID to ImageNet-pretrained CDMs and develop strategies for finding the projection time step and the number of extraneous directions. Our method also generalizes to text-conditioned DMs and is compatible with different diffusion samplers. Building on the discovery of CLARep, we propose a novel feature distillation paradigm, CaDistill . This method leverages the interpretable nature of CLAReps, which encapsulates the core semantics of each class, to supervise a student network. CaDistill aligns the student network’s representations on both Canonical Samples and original training samples with Canonical Features using a novel feature distillation loss, which helps the student network encode the core class information. The student network’s representations of the original training samples are also forced to be close to those of the Canonical Samples, treating them as anchors in the student’s representation space. The student learns on the full training set, while the teacher CDM transfers the essential class knowledge by only exploiting CLAReps, which amounts to merely 10% of the training data in size. In contrast, existing state-of-the-art methods require transferring the teacher CDM’s knowledge using the entire training set. CaDistill improves the student’s adversarial robustness as well as the generalization capability on CIFAR10 [ 40 ] and ImageNet [ 15 ] . Our contributions are as follows: • We introduce C anonical LA tent Rep resentation (CLARep) in CDMs—latent codes whose internal CDM features, i.e. Canonical Features , encapsulate core categorical semantics with minimal irrelevant signals. When decoded to the data space, these latent codes produce Canonical Samples , which serve as compact and interpretable prototypes for each class. • To extract CLAReps , we propose C anonical LA tent R epresentation ID entifier (CLARID), a method that identifies these latent codes by projecting out non-discriminative directions in the CDM’s latent space. The optimal configurations of CLARID are selected through a systematic analysis of the CDM’s features. • Leveraging the CLAReps, we develop a novel diffusion-based feature distillation paradigm, CaDistill . While the student is being trained on the full training set, the CDM as the teacher transfers essential class knowledge only via exploiting CLAReps, which amounts to merely 10 % of the data. The resulting student network achieves strong adversarial robustness and generalization performance, focusing more on the core class information. 2 Related Works 2.1 Interpretability in diffusion models Recent research in diffusion models (DMs) reveals the interpretable semantic information in them. We categorize these efforts into two main groups. The first line of this work focuses on semantic editing by manipulating the reverse diffusion trajectory to produce semantically meaningful changes in generated images [ 9 , 22 , 41 , 55 ] . Kwon et al. 2023 uncover a semantic space inside a pre-trained DM, termed h-space, where particular vector directions yield high-quality image editing results. Park et al. 2023 analyze the latent input space, namely x-space, of DMs from a Riemannian geometry perspective. They define the pullback