One-Step Offline Distillation of Diffusion-based Models via Koopman Modeling
Nimrod Berman, Moshe Eliasof, Hedi Zisling, Omri Azencot · 2025
Read One-Step Offline Distillation of Diffusion-based Models via Koopman Modeling by Nimrod Berman, Moshe Eliasof, Hedi Zisling, Omri Azencot (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13358.txt.
Introduction
Nimrod Berman and co-authors look for a faster way to run diffusion models, the image generators that build a picture step by step. That step-by-step process gives good results but costs a lot of computer time. The team wants a model that can produce a clean image in one step instead.
Their approach, called the Koopman Distillation Model, rests on an old math idea for turning curved motion into straight motion in a new space. The model maps a noisy input into that space, moves it forward with a simple learned rule, then decodes it into a clean sample. Tests on standard benchmarks show results that rival other fast methods.
Catalog record
| CREATOR | Nimrod Berman, Moshe Eliasof, Hedi Zisling, Omri Azencot |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 94 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-13358
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
One-Step Offline Distillation of Diffusion-based Models via Koopman Modeling Nimrod Berman, Moshe Eliasof, Hedi Zisling, Omri Azencot Abstract Diffusion-based generative models have demonstrated exceptional performance, yet their iterative sampling procedures remain computationally expensive. A prominent strategy to mitigate this cost is distillation , with offline distillation offering particular advantages in terms of efficiency, modularity, and flexibility. In this work, we identify two key observations that motivate a principled distillation framework: (1) while diffusion models have been viewed through the lens of dynamical systems theory, powerful and underexplored tools can be further leveraged; and (2) diffusion models inherently impose structured, semantically coherent trajectories in latent space. Building on these observations, we introduce the Koopman Distillation Model ( KDM ) , a novel offline distillation approach grounded in Koopman theory—a classical framework for representing nonlinear dynamics linearly in a transformed space. KDM encodes noisy inputs into an embedded space where a learned linear operator propagates them forward, followed by a decoder that reconstructs clean samples. This enables single-step generation while preserving semantic fidelity. We provide theoretical justification for our approach: (1) under mild assumptions, the learned diffusion dynamics admit a finite-dimensional Koopman representation; and (2) proximity in the Koopman latent space correlates with semantic similarity in the generated outputs, allowing for effective trajectory alignment. KDM achieves highly competitive performance across standard offline distillation benchmarks. 1 Introduction Diffusion-based generative models [ 72 ] , including score-based and flow-matching variants [ 30 , 47 ] , have become ubiquitous across a wide range of domains—from images and videos to audio and time series [ 18 , 31 , 41 , 16 , 59 , 21 , 26 ] . These models now surpass approaches such as GANs and VAEs in terms of sample quality, while also exhibiting greater training stability. Despite these advantages, one of their key limitations remains the high computational cost associated with sampling [ 63 , 68 , 51 ] . Generating a single high-fidelity sample typically requires executing a lengthy iterative process, progressively refining random noise through dozens or even hundreds of model evaluations [ 30 ] . A widely adopted strategy to address this involves minimizing the number of inference steps [ 73 ] , for example by leveraging improved numerical solvers or designing more expressive noise schedules, enabling high-quality generation with significantly fewer denoising iterations. Another increasingly popular alternative is distillation [ 68 ] , where a student model learns to emulate the behavior of a teacher model. Distillation approaches vary in supervision and setup: in online distillation, the student directly learns from the teacher’s predictions during training, with the teacher providing trajectory information across time steps. In contrast, offline distillation relies on precomputed noise–image pairs or teacher-generated samples only, where the student learns without any further access to the teacher model, including its weights, evaluations, or internal states. Offline methods inherently offer key advantages over online tools. Training on precomputed, teacher-generated noise–image pairs avoids multi-stage training and repeated teacher queries, reducing both neural function evaluations and memory footprint [ 24 ] . It also enables pre-filtering or anonymization of potentially privacy violations [ 22 ] . Consequently, once the dataset is produced, the teacher can be discarded, shrinking the attack surface and supporting secure collaboration (e.g., outsourcing training without exposing proprietary information) ( 12 , 32 , 61 ) . Beyond privacy and efficiency, offline distillation is model-agnostic and architecture-general (as evidenced in our experiments), and it scales flexibly with parameter budgets. On the limitation side, offline setups require upfront generation and storage costs for the precomputed corpus and, lacking teacher access beyond the noise–image pairs, may constrain model expressivity and adaptability. By contrast, online methods may warm-start from teacher weights or query the teacher for stepwise supervision. Despite these trade-offs, offline distillation remains a practical, private, and sustainable approach to generative-model distillation. The two paradigms can be considered complementary, and the optimal choice depends on factors such as privacy requirements, computational cost, and deployment constraints. Towards developing a distillation method, we identify two key observations that guide our approach: (1) While denoising-based methods leverage dynamical systems perspectives for both general generative modeling [ 76 , 47 ] and distillation [ 89 , 13 ] , a wide range of powerful tools from dynamical systems theory remain underexplored; (2) Trained diffusion models impose a structured organization on the latent space, coherently mapping noisy inputs toward clean data. Specifically, our empirical analysis suggests that semantically related images tend to originate from nearby points in the noise space. Building on these observations, we propose leveraging a dynamical systems approach that has not yet been explored in the context of distillation: Koopman operator theory. This theory is a classical yet recently revitalized approach for representing nonlinear systems with linear dynamics in an embedded space [ 42 , 67 ] . The core idea is that by mapping the system into a suitable embedding space, its evolution becomes linear. Thus, learning the diffusion process then reduces to jointly learning these embeddings and the linear operator. While Koopman operators are typically infinite-dimensional, we prove that an exact finite-dimensional representation exists under mild assumptions [ 33 ] . Moreover, prior work suggests that the Koopman operator preserves the structure inherent to certain dynamical systems [ 28 , 9 ] . This motivates the use of Koopman theory to harness the observed coherent semantic structure in diffusion dynamics, offering a principled foundation for accurate teacher model distillation. To this end, we formally connect the observed structure in diffusion models to the Koopman framework by proving that this structure is preserved within the learned dynamical representation. [formula] [formula] [formula] [formula] [formula] Figure 1 : A Gaussian sample (bottom left) evolves into a clean image (bottom right) via nonlinear dynamics \Phi_{T} . Leveraging the Koopman framework, learned encoders E_{\varphi} and E_{\phi} transform noise and image into an embedding space where the evolution becomes linear under C_{\mu} (top). In practice, we propose KDM ( Koopman Distillation Model ), a framework for modeling diffusion processes through linear evolution in a learned embedding space. Given a noisy input, an encoder maps it into an embedding space where a learned Koopman operator evolves it forward, and a decoder reconstructs the clean sample. A schematic illustration of our framework is presented in Fig. 1 . The training objective encourages semantic preservation (via reconstruction loss), accurate latent linear dynamics (via Koopman consistency), and fidelity in output space (via prediction and adversarial losses). This design naturally yields fast, single-step generation, while preserving structural coherence distillation. We evaluate our method on the offline distillation benchmark, while adding two higher-resolution datasets, and achieve highly competitive results in both unconditional and conditional generation. These results demonstrate the effectiveness of our Koopman-based formulation in bridging the performance gap with online methods, offering a fast and high-fidelity offline diffusion model distillation. Our key contributions are: 1. Observing diffusion dynamics . We identify, analyze and formalize two key observations: (i) tools from dynamical systems theory, specifically Koopman operator theory, remain underexplored in diffusion modeling and distillation; (ii) training induces coherent semantic organization in noise space. These insights motivate a structure-promoting dynamics-aware approach to distillation. 2. A principled and practical Koopman-based distillation framework . We develop a theoretical foundation proving