Differentiable Solver Search for Fast Diffusion Sampling

Shuai Wang, Zexian Li, Qipeng Zhang, Tianhui Song, Xubin Li, Tiezheng Ge, Bo Zheng, Limin Wang · 2025

Read Differentiable Solver Search for Fast Diffusion Sampling by Shuai Wang, Zexian Li, Qipeng Zhang, Tianhui Song, Xubin Li, Tiezheng Ge, Bo Zheng, Limin Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21114.txt.

Introduction

Shuai Wang and co-authors search for faster samplers for diffusion image models in a 2025 paper. Diffusion models make superb images but need many steps, each a costly network pass. Equation solvers cut the steps, yet classic solver designs borrow interpolation tricks that fit diffusion poorly.

The team shows the standard time-based interpolation is suboptimal and carves out a compact search space of step sizes and solver weights. Their differentiable search tunes those values directly for image quality. With only ten steps, several large models reached top-tier quality scores on a standard image benchmark. The found solver beat classic ones by a wide margin and transferred across model sizes, designs, and resolutions. The work turns solver design into a learnable problem.

Catalog record

CREATORShuai Wang, Zexian Li, Qipeng Zhang, Tianhui Song, Xubin Li, Tiezheng Ge, Bo Zheng, Limin Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-21114

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Differentiable Solver Search for Fast Diffusion Sampling Shuai Wang, Zexian Li, Qipeng Zhang, Tianhui Song, Xubin Li, Tiezheng Ge, Bo Zheng, Limin Wang Abstract Diffusion models have demonstrated remarkable generation quality, but at the cost of numerous function evaluations. Advanced ODE-based solvers have recently been developed to mitigate the substantial computational demands of reverse-diffusion solving under limited sampling steps. However, these solvers, heavily inspired by Adams-like multistep methods, rely solely on t-related Lagrange interpolation. We show that t-related Lagrange interpolation is suboptimal for diffusion models and reveal a compact search space comprised of time steps and solver coefficients. Building on our analysis, we propose a novel differentiable solver search algorithm to identify more optimal solver. Equipped with the searched solver, rectified-flow models, e.g., SiT-XL/2 and FlowDCN-XL/2, achieve FID scores of 2.40 and 2.35, respectively, on ImageNet- 256 × 256 256\times 256 with only 10 steps. Meanwhile, DDPM model, DiT-XL/2, reaches a FID score of 2.33 with only 10 steps. Notably, our searched solver significantly outperforms traditional solvers(even some distillation methods). Moreover, our searched solver demonstrates generality across various model architectures, resolutions, and model sizes. Figure 1 : Visualization of searched Solver Parameters of DDPM/VP and Rectified Flow. We limited the order of solver coefficients of the last two steps for 5/6 NFE. The left images show the absolute value of searched coefficients \{c_{i}^{j}\} . The right image shows the searched timesteps of different NFE and fitted curves. 1 Introduction Image generation is a fundamental task in computer vision research, which aims at capturing the inherent data distribution of original image datasets and generating high-quality synthetic images through distribution sampling. Diffusion models ( Ho et al. 2020 ; Song et al. 2020b ; Karras et al. 2022 ; Liu et al. 2022 ; Lipman et al. 2022 ; Wang et al. 2025 ) have recently emerged as highly promising solutions to learn the underlying data distribution in image generation, outperforming GAN-based models ( Brock et al. 2018 ; Sauer et al. 2022 ) and Auto-Regressive models ( Chang et al. 2022 ) by a significant margin. However, diffusion models necessitate numerous denoising steps during inference, which incur a substantial computational cost, thereby limiting the widespread deployment of pre-trained diffusion models. To achieve fast diffusion sampling, the existing studies have explored two distinct approaches. Training-based techniques by distilling the fast ODE trajectory into the model parameters, thereby circumventing redundant refinement steps. In addition, solver-based methods ( Lu et al. 2023 ; Zhang & Chen 2023 ; Song et al. 2020a ) tackle the fast sampling problem by designing high-order numerical ODE solvers. For training-based acceleration, ( Salimans & Ho 2022 ) aligns the single-step student denoiser with the multi-step teacher output, thereby reducing inference burdens. The consistency model concept, introduced by ( Song et al. 2023 ) , directly teaches the model to produce consistent predictions at any arbitrary timesteps. Building upon ( Song et al. 2023 ) , subsequent works ( Zheng et al. 2024 ; Kim et al. 2023 ; Wang et al. 2024a ; Song et al. 2025 ) propose improved techniques to mitigate discreet errors in LCM training. Furthermore, ( Lin et al. 2024 ; Kang et al. 2024 ; Yin et al. 2024 ; Zhou et al. 2024 ; Wang et al. 2023 ) leverage adversarial training and distribution matching to enhance the quality of generated samples. To improve the training efficiency of distribution matching. However, training-based methods introduce changes to the model parameters, resulting in an inability to fully exploit the pre-training performance. Solver-based methods rely heavily on the ODE formulation in the reverse-diffusion dynamics and hand-crafted multi-step solvers. ( Lu et al. 2023 ; Lu et al. 2022 ) and ( Zhang & Chen 2023 ) point out the semi-linear structure of the diffusion ODE and propose an exponential integrator to tackle faster sampling in diffusion models. ( Zhao et al. 2023 ) further enhances the sampling quality by borrowing the predictor-corrector structure. Thanks to the multistep-based ODE solver methods, high-quality samples can be generated within as few as 10 steps. To further improve efficiency, ( Gao et al. 2023 ) tracks the backward error and determines the adaptive step. Moreover, ( Karras et al. 2022 ; Lu et al. 2022 ) propose a handcrafted timesteps scheduler to sample respaced timesteps. ( Xue et al. 2024 ) argues that timesteps sampled in ( Karras et al. 2022 ; Lu et al. 2022 ) are suboptimal, thus proposing an online optimization algorithm to find the optimal sampling timesteps for generation. Apart from timesteps optimization, ( Shaul et al. 2023 ) learns a specific path transition to improve the sampling efficiency. In contrast to training-based acceleration methods, solver-based approaches do not necessitate parameter adjustments and preserve the optimal performance of the pre-trained model. Moreover, solvers can be seamlessly applied to any arbitrary diffusion model trained with a similar noise scheduler, offering a high degree of flexibility and adaptability. This motivates us to investigate the generative capabilities of pre-trained diffusion models within limited steps from a diffusion solver perspective. Current state-of-the-art diffusion solvers ( Lu et al. 2023 ; Zhao et al. 2023 ) adopt Adams-like multi-step methods that use the Lagrange interpolation function to minimize integral errors. We argue that an optimal solver should be tailored to specific pre-trained denoising functions and their corresponding noise schedulers. In this paper, we explore solver-based methods for fast diffusion sampling by improving diffusion solvers using data-driven approaches without destroying the pre-training internality in diffusion models. Inspired by ( Xue et al. 2024 ) , we investigate the sources of error in the diffusion ODE and discover that the interpolation function form is inconsequential and can be reduced to coefficients. Furthermore, we define a compact search space related to the timesteps and solver coefficients. Therefore, we propose a differentiable solver search method to identify the optimal parameters in the compact search space. Based on our novel differentiable solver search algorithm, we investigate the upper bound performance of pre-trained diffusion models under limited steps. Our searched solver significantly improves the performance of pre-trained diffusion models, and outperforms traditional solvers with a large gap. On ImageNet- 256\times 256 , armed with our solver, rectified-flow models of SiT-XL/2 and FlowDCN-XL/2 achieve 2.40 and 2.35 FID respectively under 10 steps, while DDPM model of DiT-XL/2 achieves 2.33 FID. Surprisingly, our findings reveal that when equipped with an optimized high-order solver, the DDPM can achieve comparable or even surpass the performance of rectified flow models under similar NFE constraints. To summarize, our contributions are • We reveal that the interpolation function choice is not important and can be reduced to coefficients through the pre-integral technique. We demonstrate that the upper bound of discretization error in reverse-diffusion ODE is related to both timesteps and solver coefficients and define a compact solver search space. • Based on our analysis, we propose a novel differentiable solver search algorithm to find the optimal solver parameter for given diffusion models. • For DDPM/VP time scheduling, armed with our searched solver, DiT-XL/2 achieves 2.33 FID under 10 steps, beating DPMSolver++/UniPC by a significant margin. • For Rectified-flow models, armed with our searched solver, SiT-XL/2 and FlowDCN-XL/2 achieve 2.40 and 2.35 FID respectively

Read the complete text