Accelerated Diffusion Models via Speculative Sampling

Valentin De Bortoli, Alexandre Galashov, Arthur Gretton, Arnaud Doucet · 2025

Read Accelerated Diffusion Models via Speculative Sampling by Valentin De Bortoli, Alexandre Galashov, Arthur Gretton, Arnaud Doucet (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-05370.txt.

Introduction

Valentin De Bortoli, Alexandre Galashov, Arthur Gretton, and Arnaud Doucet speed up spread-based image makers with guess-and-check draws in a 2025 paper. The trick is a hit for text models: a fast draft pens bits, and the big model keeps or drops them by its own odds. Spread makers grow draws by smooth chains, not bits.

The team grows the trick to those chains, where the target is a dear but fine maker. One draft route needs no trained draft at all and fits any maker. Tests halve the step count at no loss of truth.

Catalog record

CREATORValentin De Bortoli, Alexandre Galashov, Arthur Gretton, Arnaud Doucet
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE144 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-05370

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Accelerated Diffusion Models via Speculative Sampling Valentin De Bortoli, Alexandre Galashov, Arthur Gretton, Arnaud Doucet Abstract Speculative sampling is a popular technique for accelerating inference in Large Language Models by generating candidate tokens using a fast draft model and then accepting or rejecting them based on the target model’s distribution. While speculative sampling was previously limited to discrete sequences, we extend it to diffusion models, which generate samples via continuous, vector-valued Markov chains. In this context, the target model is a high-quality but computationally expensive diffusion model. We propose various drafting strategies, including a simple and effective approach that does not require training a draft model and is applicable out-of-the-box to any diffusion model. We demonstrate significant generation speedup on various diffusion models, halving the number of function evaluations while generating exact samples from the target model. Finally, we also show how this procedure can be used to accelerate Langevin diffusions to sample unnormalized distributions. † † affiliationnotice: Equal contribution 1 Motivation Denoising diffusion models (DDMs), introduced by Sohl-Dickstein et al. (2015) and further developed by Ho et al. (2020) and Song et al. (2021) , are generative models exhibiting state-of-the-art performance in a wide variety of domains. The core concept behind DDMs is the progressive transformation of a data distribution into a Gaussian distribution through the addition of noise. Sample generation is achieved by simulating an approximation of the time-reversal of this noising process. This requires multiple evaluations of a neural network that approximates the scores of the noising process, and typically involves simulating a Markov chain over hundreds of steps. Since sample generation is computationally expensive, several techniques have been proposed to accelerate it. These include distillation techniques ( Salimans & Ho, 2022 ; Meng et al., 2023 ; Song et al., 2023 , e.g. ) , better sampling schemes ( Karras et al., 2022 ; Lu et al., 2022 ; Zhang & Chen, 2023 , e.g. ) and parallel simulation methods ( Shih et al., 2023 ; Chen et al., 2024 , e.g. ) . However, distillation techniques inherently require training a student model and often underperform compared to the teacher model ( Dieleman, 2024 ) . While better sampling schemes can improve performance, using too small a number of steps does degrade performance, see e.g. ( Karras et al., 2022 ) . Finally, parallel simulation methods relying on Picard iterations over a sliding window have been proposed ( Shih et al., 2023 ; Chen et al., 2024 ; Tang et al., 2024 ) . However, they are inherently iterative, requiring repeated parallel sampling within a window until errors fall below a pre-specified tolerance. In the context of Large Language Models (LLMs), various techniques have also been proposed to speed up inference. Notably, speculative sampling , first introduced by Leviathan et al. (2023) and later proposed independently by Chen et al. (2023) , has become prominent in this area and has spawned numerous extensions ( Xia et al., 2024 ) . Given a target LLM, this algorithm enables faster sampling than serial token decoding without compromising quality, as the sampled tokens remain exactly distributed according to the target model’s distribution. This is achieved by considering a smaller and faster LLM model generating a draft sequence. The target model is then used to compute in parallel the conditional probabilities of these draft tokens, and these probabilities are used to decide sequentially whether to accept or reject the draft tokens. Upon the first rejection, a new token is sampled using an adjusted distribution combining the draft and target distributions. Many extensions of speculative sampling have been proposed to reduce latency; see Xia et al. (2024) and further related works in Section 6 . In the present work, we adapt speculative sampling to accelerate DDMs. We assume a computationally cheap draft model that generates a sequence of draft states for the denoising Markov chain of a target DDM. The transition probability densities of these states under the target model are then computed in parallel and used to sequentially accept/reject the draft states. At rejection, a new state is sampled from an adjusted distribution dependent on both the draft and target distributions; see Figure 1 . As for LLMs, the procedure is designed such that it outputs samples distributed exactly according to the target DDM. Efficient Draft Verify in Parallel ignored ✓ ✓ ✓ \times Speculative sampling Iterate ✓ \times accepted samples rejected samples Figure 1: Speculative Sampling for diffusion models. Draft states are efficiently generated and verified in parallel. Upon the first rejection, a new state is sampled using an adjusted distribution combining draft & target models, and the remainder of the draft sequence is discarded. Wang et al. (2024) concurrently proposed an adaptation of speculative sampling for continuous-valued autoregressive processes, specifically for Masked Autoregressive models ( Li et al., 2024b ) . In this setting, they sample from the adjusted distribution appearing at rejection using a standard rejection sampling algorithm. However, as demonstrated in Section 3.2 , this approach is, on average, more computationally expensive than directly sampling from the target model in our context. Furthermore, it exhibits counter-intuitive performance degradation as the draft model more closely approximates the target model. We present a method to circumvent these issues while retaining the optimality properties of speculative sampling. Our contributions are summarized below. Proofs are in the Supplementary Material. • By leveraging the connections between speculative sampling and coupling techniques ( Lindvall, 1992 ) , first observed by Sun et al. (2023) in the context of LLMs, we show in Section 3.3 that we can sample efficiently from a novel adjusted distribution for DDMs using reflection maximal coupling ( Bou-Rabee et al., 2020 ) . Our procedure returns exact samples from the target model, and it is optimal in the sense that it maximizes the probability of accepting each draft state. • We investigate several drafting strategies ( Section 3.1 and Appendix B ). As with LLMs, one can rely on a “cheap” diffusion model as draft model, or use a draft model learned from the target model. We propose here instead a simple and effective approach that proposes a draft model relying solely on the target model. This eliminates any need for learning a separate draft model, and is readily applicable to any diffusion model. • We present a complexity analysis and a lower bound on the acceptance ratio of the draft states in Section 4 . • We explain in Section 5 how this method can be adapted to accelerate Langevin diffusions to sample unnormalized distributions. • The proposed method achieves significant speed-ups for image generation on CIFAR10, and LSUN using pixel space diffusion models, without any loss of quality ( Section 7 ). Furthermore, we show similar speed-ups in robotics for policy generation. 2 Speculative Sampling for LLMs We begin with a review of speculative sampling for LLMs. Consider two probability distributions q and p for sequences on some finite space \mathcal{X} . In this context, q corresponds to the joint distribution of tokens for the target LLM, while p represents the draft model. 2.1 Speculative Sampling for Autoregressive Targets Speculative sampling generates L candidate tokens according to the draft model p which are scored in parallel

Read the complete text