Uni-DAD: Unified Distillation and Adaptation of Diffusion Models for Few-step Few-shot Image Generation

Yara Bahram, Melodie Desbos, Mohammadhadi Shateri, Eric Granger · 2025

Read Uni-DAD: Unified Distillation and Adaptation of Diffusion Models for Few-step Few-shot Image Generation by Yara Bahram, Melodie Desbos, Mohammadhadi Shateri, Eric Granger (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-18281.txt.

Introduction

Yara Bahram, Mélodie Desbos, Mohammadhadi Shateri, and Eric Granger merge two slow steps in image generation into one. Diffusion models make sharp images but sample at high cost, and a sped up copy tends to stay stuck in its teacher's domain. Past fixes chain two stages, adapt then distill or the reverse, at a price in quality or range.

Their method, Uni-DAD, trains in a single stage. One signal pulls the student toward both a source teacher and a target teacher at once. A second signal, from a multi scale critic, pushes for real looking output and curbs overfit when few target images exist. The single stage keeps source range while it gains speed in the new domain.

Catalog record

CREATORYara Bahram, Melodie Desbos, Mohammadhadi Shateri, Eric Granger
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-18281

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Uni-DAD: Unified Distillation and Adaptation of Diffusion Models for Few-step Few-shot Image Generation Yara Bahram, Melodie Desbos, Mohammadhadi Shateri, Eric Granger Abstract Diffusion models (DMs) produce high-quality images, yet their sampling remains costly when adapted to new domains. Distilled DMs are faster but typically remain confined within their teacher’s domain. Thus, fast and high-quality generation for novel domains relies on two-stage training pipelines: Adapt-then-Distill or Distill-then-Adapt. However, both add design complexity and suffer from degraded quality or diversity. We introduce Uni-DAD , a single-stage pipeline that unifies distillation and adaptation of DMs. It couples two signals during training: (i) a dual-domain distribution-matching distillation objective that guides the student toward the distributions of the source teacher and a target teacher, and (ii) a multi-head generative adversarial network (GAN) loss that encourages target realism across multiple feature scales. The source domain distillation preserves diverse source knowledge, while the multi-head GAN stabilizes training and reduces overfitting, especially in few-shot regimes. The inclusion of a target teacher facilitates adaptation to more structurally distant domains. We perform evaluations 1 1 1 Our code is included in suppl. materials and will be made public. on a variety of datasetsfor few-shot image generation (FSIG). Uni-DAD delivers higher quality than state-of-the-art (SoTA) adaptation methods even with ≤ 4 \leq 4 sampling steps, and outperforms two-stage training pipelines in both quality and diversity. 1 Introduction Figure 1 : Uni-DAD ( Distill & Adapt ) vs. two-stage pipelines, Distill-then-Adapt , and Adapt-then-Distill . Adapt is performed by fine-tuning, and Distill by DMD2 49 . The source domain is represented by 70K diverse faces, and the target domain by 10 babies. Sampling steps are reduced from 25 to 3. DMs ( 41 ; 13 ; 44 ) have emerged as the dominant paradigm for generative modeling, achieving SoTA performance in image synthesis ( 8 ) and text-to-image generation ( 34 ; 36 ) . These models produce high-quality and diverse images even when adapted to novel domains and subjects, given only a handful of images. This makes them an attractive solution for FSIG ( 3 ; 52 ) and subject-driven personalization (SDP) ( 9 ; 35 ; 19 ) . However, DMs need an iterative denoising procedure over many time-steps for sampling, resulting in slow test-time generation. Adapted models inherit this cost, challenging real-time personalized use-cases. Distillation alleviates the slow inference by training a few-step student to mimic a larger teacher DM ( 37 ; 43 ; 50 ; 49 ; 5 ) . Ultimately, the ability to generate images in novel domains in few-shot contexts while requiring only a few denoising steps can facilitate the deployment of DMs in personalized real-time applications. In recent works, reducing the number of time-steps and adapting to new domains requires a two-stage pipeline: Distill-then-Adapt or Adapt-then-Distill . The former is more compute-friendly as a student can be adapted per task after a single compute-heavy distillation step. Yet, students often saturate their adaptation capacity, yielding over-smoothed outputs on few-shot target domains ( 27 ) . Further, fine-tuning a student under the teacher’s original diffusion loss negates the benefits of distillation ( 27 ) . Adapt-then-Distill can yield higher image quality and mitigate over-smoothing. However, the student remains tied to the adapted teacher’s performance and is prone to overfitting. Conversely, neither of the two-stage pipelines is an end-to-end process, and both are susceptible to losing diverse transferable source information during the training processes. We propose Uni fied D istillation and A daptation of D iffusion models ( Uni-DAD ), a single-stage pipeline that compresses a high-quality DM teacher into a few-step student, while adapting it to a few-shot target domain (Fig. 1 ). It couples two complementary signals: a dual-domain distribution–matching distillation (DMD) objective and a multi-head GAN loss. The dual-domain DMD guides the student’s generation with the scores of a frozen source-domain teacher and optionally an online target-domain teacher. The inclusion of the target teacher improves adaptation to structurally distant domains. This dual-domain design guides the student toward a common area between the two distributions while preserving source diversity. The multi-head GAN enforces target realism across multiple feature scales, reducing overfitting in few-shot regimes. An online fake teacher tracks the evolving student distribution and provides up-to-date negatives for the discriminator in the GAN framework. Uni-DAD training iterates among updating (i) the student, (ii) the online fake teacher and discriminator, and optionally (iii) an online target teacher. These objectives allow the student to preserve source-derived diversity while sharpening target realism. The end result is a few-step generator that produces diverse high-quality images of a novel domain in few-shot contexts. Uni-DAD is checkpoint-agnostic: a pre-adapted target DM can replace the online target teacher with no additional training needed, and a pre-distilled source DM can initialize the student. As a result, our method enables distillation of adapted models and adaptation of distilled models without any changes to the training loop. Uni-DAD is extensively validated on four target sets with varying levels of source proximity on the task of FSIG 28 with guided denoising diffusion probabilistic model (DDPM) 8 . It attains higher quality with substantially fewer sampling steps than related SoTA adaptation methods and outperforms two-stage pipelines in terms of quality and diversity. Our single-step pipeline offers a practical path to fast personalized image generation. Contributions. (i) This paper introduces the first single-stage pipeline that jointly distills and adapts a DM for fast, high-quality, and diverse generation in novel domains. (ii) Dual-domain DMD and multi-head GAN losses are proposed to help retain source-domain diversity while sharpening target domain realism under few-shot data. An optional target teacher facilitates adaptation to structurally distant domains. (iii) On FSIG benchmarks, our method achieves higher quality with substantially fewer sampling steps than prior non-distilled adaptation methods and outperforms two-stage pipelines in quality and diversity. 2 Related Work (a) Diffusion Distillation. To address the costly DM inference, efficient numerical solvers such as DDIM ( 42 ) and DPM-Solver ( 21 ) allow compressing the long denoising trajectory without retraining the DM \big(\text{neural function evaluations, NFE}\!\sim\!10\big) . Knowledge distillation, on the other hand, trains a few-step student generator to mimic a teacher DM ( 1\leq\text{NFE}\!\leq\!4 ). Progressive distillation halves steps by matching the teacher across adjacent time-steps ( 37 ) . Consistency-based methods directly learn a one-step mapping from noise to data ( 43 ; 22 ) . Recently, by combining score distillation with adversarial training, DMD ( 50 ; 49 ) , ADD ( 39 ; 38 ) , and FlashDiffusion ( 5 ) , yield few-step students that match or surpass their teacher in quality. However, the students remain tied to the teacher’s manifold, limiting flexibility under domain shift. Furthermore, the adversarial objective assumes access to large training corpora that is unavailable in few-shot applications, where the GAN’s discriminator easily memorizes the target set. We focus on distillation in the face of domain-shift and few-shot target sets. Figure 2 : Overview of Uni-DAD for few-step and few-shot image generation. A (frozen) source teacher \epsilon^{\text{src}} is adapted and distilled into a student G for fast sampling ( 1\leq\text{NFEs}\leq 4 ) on the target domain. At each training iteration, Uni-DAD alternates among three updates: (1) Student: optimize G with a dual-domain DMD objective on \epsilon^{\text{src}}

Read the complete text