What’s in a Latent? Leveraging Diffusion Latent Space for Domain Generalization

Xavier Thomas, Deepti Ghadiyaram · 2025

Read What’s in a Latent? Leveraging Diffusion Latent Space for Domain Generalization by Xavier Thomas, Deepti Ghadiyaram (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06698.txt.

Introduction

Models that work on one photo collection often stumble on another, a failure called poor domain generalization. Xavier Thomas and Deepti Ghadiyaram mine the inner space of image diffusion models for help in a 2025 paper.

Their method first hunts for hidden groupings inside a pre-trained feature space. Those groupings, found without any labels, capture how domains differ. A classifier then receives both its usual features and these extra domain-flavored ones. Features from diffusion models proved best at sorting domains without labels. Across five data sets, the simple add-on lifted accuracy on unseen domains by up to four points. It even beat most methods that are allowed to see domain labels in training.

Catalog record

CREATORXavier Thomas, Deepti Ghadiyaram
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-06698

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

What’s in a Latent? Leveraging Diffusion Latent Space for Domain Generalization Xavier Thomas, Deepti Ghadiyaram Abstract Domain Generalization aims to develop models that can generalize to novel and unseen data distributions. In this work, we study how model architectures and pre-training objectives impact feature richness and propose a method to effectively leverage them for domain generalization. Specifically, given a pre-trained feature space, we first discover latent domain structures, referred to as pseudo-domains , that capture domain-specific variations in an unsupervised manner. Next, we augment existing classifiers with these complementary pseudo-domain representations making them more amenable to diverse unseen test domains. We analyze how different pre-training feature spaces differ in the domain-specific variances they capture. Our empirical studies reveal that features from diffusion models excel at separating domains in the absence of explicit domain labels and capture nuanced domain-specific information. On 5 5 datasets, we show that our very simple framework improves generalization to unseen domains by a maximum test accuracy improvement of over 4% compared to the standard baseline Empirical Risk Minimization (ERM). Crucially, our method outperforms most algorithms that access domain labels during training. Code: https://xthomasbu.github.io/GUIDE . 1 Introduction Figure 1 : T-SNE visualization of the latent space from different pre-training objectives: CLIP [ 53 ] , DiT [ 49 ] , MAE [ 20 ] , ResNet-50 [ 18 ] on the domain generalization benchmark VLCS [ 15 ] . VLCS is curated from 4 different datasets, thus dataset-specific biases like spatial composition and object size variations serve as different domains. Note how the diffusion features separate domains effectively, suggesting that latent domain structures can be captured without explicit supervision. Best viewed in color. It is now a common practice to use models pre-trained on billion-scale data [ 18 , 20 , 56 , 49 , 47 , 53 , 42 ] as defacto backbones for diverse downstream tasks [ 39 , 65 ] . In order to make these large-scale models “foundational,” and offer rich feature representations, a variety of powerful pre-training strategies have been designed. Some of these objectives aim to eliminate the need for clean labeled data [ 12 , 7 , 75 , 8 , 19 ] , some reap the benefits from rich text representations by aligning them with corresponding visual signals [ 53 , 28 ] , while others force models to build a more meaningful understanding of scenes by learning to predict large hidden regions of images [ 20 ] . Despite such tremendous progress, what exactly is captured in the underlying latent landscape remains an open question. This question becomes more challenging in diffusion models mainly due to their iterative global denoising objective. This work aims to understand the feature landscape learnt from different pre-training models and objectives in the context of domain generalization. Robust generalization to unseen domains has been a long-standing goal in machine learning research [ 5 , 44 ] , particularly in scenarios where collecting domain-specific data is infeasible or expensive. In such cases, models must learn to generalize without relying on explicit domain labels even during training [ 36 ] . It has been established that most sophisticated models struggle when the test data distribution differs from that of training data [ 63 , 55 , 61 ] even in subtle ways, e.g., same visual scene but captured using different cameras, same patient but different brand imaging devices, same object but captured in different color schemes and so on. We posit that the first step to make fundamental progress towards designing foundational models is to examine and interpret how current state-of-the-art models structure visual information and uncover their strengths and limitations. For instance, how are object, scene, and domain-specific variations internally encoded in a latent space? Do domain-specific traits manifest in distinct regions of the latent space or are they engulfed along with low- to mid-level scene and object level information? We study these questions in great detail in this work. Specific to the task of domain generalization, we analyze how different pre-training objectives and architectures influence the granularity of visual information captured in their feature space. Our key insight is that certain internal states of diffusion models effectively capture abstract information such as photographic styles, camera angles, and so on. Building on this insight, we first develop an unsupervised method for discovering latent domain structures. Next, we alter a standard domain generalization classification [ 67 ] pipeline with one key difference: we augment the classifier’s representations with the discovered latent domain representations. We show through extensive empirical analysis that this simple tweak to the standard pipeline assists in training a model that generalizes well to unseen domains [ 4 ] . While most prior works focus on leveraging a single feature space to design a universal model [ 60 , 16 , 10 , 40 , 62 ] , we take an alterative approach and compliment existing classifier’s features with domain rich features and show that this auxiliary guidance makes the overall feature space more robust to unseen domains. Our framework dubbed GUIDE : G eneralization u sing I nferred D omains from Latent E mbeddings, offers a simple and effective method to “guide” a given feature space to adapt better to unseen domains. We summarize our key contributions below: • We propose a method of unsupervised pseudo-domain discovery from frozen pre-trained feature spaces and use them to improve any model’s ability to generalize to diverse domains, making it particularly useful in scenarios where domain labels used during training are unavailable or noisy (Sec. 3.3 ). • We analyze different pre-training objectives and architectures and investigate how they influence the structure of the feature latent landscape of both diffusion and conventional vision models (Sec. 4.3 ). • We shine light on the ability of diffusion models to capture domain-specific information, such as photographic and artistic styles, texture variations , and demonstrate their effectiveness to domain generalization (Sec. 4.4 ). We obtain an average test accuracy improvement of \mathbf{+2.6\%} on 5 datasets, notably beating ERM [ 67 ] by \mathbf{+4.3\%} on the TerraIncognita dataset [ 3 ] . 2 Related Work Diffusion features for representation learning: Diffusion models [ 58 , 24 ] have significantly advanced image and video generation, prompting extensive exploration of their intermediate representations and their utility for diverse downstream tasks such as detection [ 11 ] , segmentation [ 2 , 72 ] , classification [ 32 ] , semantic correspondence [ 41 ] , depth estimation [ 71 , 78 ] , and visual reasoning [ 70 ] , showcasing their utility in both discriminative and generative domains. Recent studies [ 30 , 41 , 69 ] demonstrate that features extracted across layers and timesteps encode rich semantic information, ranging from coarse patterns to fine-grained details. In this work, we analyze how the latent space of diffusion models captures class and domain-specific information and leverage these representations for the task of domain generalization. Domain generalization: First formalized in [ 5 ] , domain generalization is the challenging task of designing models capable of generalizing to unseen test domains. Various methods have been proposed to address this by learning domain-agnostic representations [ 45 , 27 ] , data or latent augmentation

Read the complete text