Training-Free Distribution Adaptation for Diffusion Models via Maximum Mean Discrepancy Guidance

Matina Mahdizadeh Sani, Nima Jamali, Mohammad Jalali, Farzan Farnia · 2026

Read Training-Free Distribution Adaptation for Diffusion Models via Maximum Mean Discrepancy Guidance by Matina Mahdizadeh Sani, Nima Jamali, Mohammad Jalali, Farzan Farnia (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-08379.txt.

Introduction

Matina Mahdizadeh Sani and three co-authors adapt diffusion models to target data without retraining. Their 2026 arXiv paper notes that pre-trained diffusion models act as strong generative priors. Their outputs often drift from the traits of a user's specific target data, though.

The mismatch bites hardest in domain adaptation, where only a few reference examples exist. Retraining the model there is usually infeasible. The authors guide generation with maximum mean discrepancy, a measure of distribution distance. Training-free guidance steers samples toward the target set at inference time. Medical and design users with small private datasets could benefit from adaptation that needs no new training run.

Catalog record

CREATORMatina Mahdizadeh Sani, Nima Jamali, Mohammad Jalali, Farzan Farnia
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-08379

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Training-Free Distribution Adaptation for Diffusion Models via Maximum Mean Discrepancy Guidance Matina Mahdizadeh Sani, Nima Jamali, Mohammad Jalali, Farzan Farnia Abstract Pre-trained diffusion models have emerged as powerful generative priors for both unconditional and conditional sample generation, yet their outputs often deviate from the characteristics of user-specific target data. Such mismatches are especially problematic in domain adaptation tasks, where only a few reference examples are available and retraining the diffusion model is infeasible. Existing inference-time guidance methods can adjust sampling trajectories, but they typically optimize surrogate objectives such as classifier likelihoods rather than directly aligning with the target distribution. We propose MMD Guidance , a training-free mechanism that augments the reverse diffusion process with gradients of the Maximum Mean Discrepancy (MMD) between generated samples and a reference dataset. MMD provides reliable distributional estimates from limited data, exhibits low variance in practice, and is efficiently differentiable, which makes it particularly well-suited for the guidance task. Our framework naturally extends to prompt-aware adaptation in conditional generation models via product kernels. Also, it can be applied with computational efficiency in latent diffusion models (LDMs), since guidance is applied in the latent space of the LDM. Experiments on synthetic and real-world benchmarks demonstrate that MMD Guidance can achieve distributional alignment while preserving sample fidelity. 1 Introduction The rapid advancement of generative artificial intelligence is largely driven by the paradigm of foundation models: pre-trained neural networks capable of generalization across diverse tasks and modalities [ 4 ] . These models, including large language models (LLMs) [ 5 ] and denoising diffusion models [ 16 , 39 , 36 ] , offer powerful priors that can be adapted to specific downstream applications with minimal computational overhead. While this adaptation can occur through fine-tuning or prompt engineering, training-free inference-time methods remain relatively underexplored, despite their practical advantages of no memory overhead and immediate deployment. A fundamental challenge in deploying generative models is distribution matching , i.e., aligning model outputs with a user’s target distribution that differs from the pre-training corpus. For example, consider a medical imaging scenario where a practitioner needs synthetic X-rays matching their hospital’s specific equipment characteristics, or a designer requiring images that follow their brand’s specific visual style, where both tasks are specified through a small number (e.g. 50-100) of reference examples. A generic text-conditioned image generation model will generate samples following its learned priors, leading to a distribution mismatch. Existing methods cannot fully address the distribution mismatch without computationally expensive retraining on a sufficiently large sample set from the target distribution, which is often unavailable to typical users of generative AI services. The mentioned gap between model capabilities and deployment requirements motivates the following question: Can we guide pre-trained diffusion models to match a user’s target distribution at inference time using limited reference samples from their distribution? Note that this distribution matching task can be considered in both the prompt-free (unconditional) and prompt-aware (conditional) data generation, where the distribution alignment targets the user’s distribution of interest. In particular, diffusion models are suited for such training-free adaptation due to their iterative denoising process. During the sampling phase of diffusion models, external signals can steer the generation trajectory through guidance in sample generation without modifying the parameters of the pre-trained neural net denoiser. Existing guidance techniques successfully bias samples toward specific objectives: classifier guidance [ 9 ] maximizes class probabilities, classifier-free guidance [ 17 ] amplifies conditioning, and score distillation [ 35 ] optimizes external losses. However, none of these methods can explicitly match a target distribution defined by reference samples, since they optimize surrogate objectives rather than directly minimizing distributional discrepancy. The key challenge is how one can quantify and minimize distribution mismatch within the reverse diffusion dynamics. Figure 1 : Image generation via a text-conditioned latent diffusion model (LDM) with no guidance vs. our proposed MMD guidance. The LDM (Stable Diffusion-XL) following our proposed MMD guidance over 100 reference samples of "cat" and "dog" images could exhibit the visual format of the target distribution, but the unguided LDM’s output samples differ in style from the target model. In this work, we propose the MMD Guidance approach, an inference-time mechanism that augments the reverse diffusion process with gradients from the Maximum Mean Discrepancy (MMD) [ 12 ] . This approach aims to address distribution matching by incorporating \nabla_{x_{t}}\text{MMD}^{2} into the sampling dynamics at each timestep. MMD is uniquely suited for this task. Unlike the family of f -divergences and Wasserstein distances, the MMD distance provides unbiased, low-variance estimates from limited samples without suffering from the curse of dimensionality. Moreover, the kernel-based formulation in MMD is efficiently differentiable, enabling scalable gradient computation with respect to generated samples. Our proposed MMD guidance method can smoothly extend to prompt-aware distribution matching in text-conditioned image generation. Users often need outputs that satisfy both textual semantics ("a portrait of a person") and distributional constraints (matching the style of their reference portraits). We attempt to achieve this property through the product kernel function k_{\text{multimodal}}\bigl([p,x],[p^{\prime},x^{\prime}]\bigr)=k_{\text{text}}\bigl(p,p^{\prime}\bigr)\cdot k_{\text{image}}\bigl(x,x^{\prime}\bigr) that jointly measure similarity in prompt and visual spaces, enabling prompt-aware control without any retraining of the denoiser in conditional diffusion model sample generation. For practical deployment, we implement MMD Guidance efficiently in the latent space of a latent diffusion model (LDM) [ 36 ] . By computing MMD and its gradients directly on the latent representation z\in\mathcal{Z} rather than pixel space, we achieve a significant speedup while maintaining alignment quality. This enhances the efficiency and scalability of the MMD guidance method for state-of-the-art conditional diffusion generation models. We validate MMD Guidance in our numerical experiments on synthetic distributions, standard image databases, and domain adaptation benchmarks. On synthetic Gaussian mixture models, the MMD guidance can recover the target mixture from a limited number of samples. In the case of image generation, the MMD Guidance framework reduces distribution discrepancy compared to the baseline training-free guidance methods while maintaining comparable fidelity scores in image generation. Also, the MMD guidance method could preserve sample quality through inference-time guidance in the latent space with proper generalization. Figure 1 illustrates an example of applying the prompt-aware MMD guidance with only 100 reference samples of "cat" and "dog" categories to the latent space of the SD-XL [ 34 ] latent diffusion model. In summary, our work’s main contributions are: • We introduce MMD Guidance, a training-free method for inference-time adaptation of diffusion models to a target distribution specified by a limited reference set. • We propose a practical extension for prompt-aware adaptation using product kernels, enabling joint alignment to both text prompts and visual styles. • We develop an efficient implementation that operates directly in the latent space of LDMs, ensuring scalability to state-of-the-art conditional diffusion models. • We provide empirical validation demonstrating that MMD guidance can achieve satisfactory distribution matching. 2 Related Work Guidance in diffusion models. Inference-time guidance has been central to the success of diffusion models. Classifier guidance augments reverse diffusion with classifier gradients, boosting conditional fidelity at the expense of diversity [ 9 ] . Classifier-free guidance (CFG) removes the external classifier by interpolating conditional and unconditional scores [ 17 ] . This line of work has been extended in several ways. [ 28 ] introduce \beta -CFG, which controls the impact of guidance during the denoising

Read the complete text