Parallel Diffusion Solver via Residual Dirichlet Policy Optimization
Ruoyu Wang, Ziyu Li, Beier Zhu, Liangyu Yuan, Hanwang Zhang, Xun Yang, Xiaojun Chang, Chi Zhang · 2025
Read Parallel Diffusion Solver via Residual Dirichlet Policy Optimization by Ruoyu Wang, Ziyu Li, Beier Zhu, Liangyu Yuan, Hanwang Zhang, Xun Yang, Xiaojun Chang, Chi Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-22796.txt.
Introduction
Ruoyu Wang and co-authors speed up diffusion models, whose image quality is excellent but whose step-by-step sampling is slow. Standard fast solvers lose quality at low step counts because their errors grow on sharply curving parts of the sampling path. The team's EPD-Solver evaluates several gradients in parallel at each step and combines them to follow the path's integral more faithfully.
Because those extra evaluations are independent, they run in parallel and keep latency low. Training happens in two stages. A small set of solver parameters is first fitted by distillation, then refined by reinforcement learning that treats the solver as a compact stochastic policy. The large backbone model is never changed. The method also works as a plug-in that upgrades existing samplers, with gains shown on demanding text-to-image tasks.
Catalog record
| CREATOR | Ruoyu Wang, Ziyu Li, Beier Zhu, Liangyu Yuan, Hanwang Zhang, Xun Yang, Xiaojun Chang, Chi Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 87 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-22796
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Parallel Diffusion Solver via Residual Dirichlet Policy Optimization Ruoyu Wang, Ziyu Li, Beier Zhu, Liangyu Yuan, Hanwang Zhang, Xun Yang, Xiaojun Chang, Chi Zhang Abstract Diffusion models (DMs) have achieved state-of-the-art generative performance but suffer from high sampling latency due to their sequential denoising nature. Existing solver-based acceleration methods often face significant image quality degradation under a low-latency budget, primarily due to accumulated truncation errors arising from the inability to capture high-curvature trajectory segments. In this paper, we propose the Ensemble Parallel Direction solver (dubbed as EPD-Solver ), a novel ODE solver that mitigates these errors by incorporating multiple parallel gradient evaluations in each step. Motivated by the geometric insight that sampling trajectories are largely confined to a low-dimensional manifold, EPD-Solver leverages the Mean Value Theorem for vector-valued functions to approximate the integral solution more accurately. Importantly, since the additional gradient computations are independent, they can be fully parallelized, preserving low-latency sampling nature. We introduce a two-stage optimization framework. Initially, EPD-Solver optimizes a small set of learnable parameters via a distillation-based approach. We further propose a parameter-efficient Reinforcement Learning (RL) fine-tuning scheme that reformulates the solver as a stochastic Dirichlet policy. Unlike traditional methods that fine-tune the massive backbone, our RL approach operates strictly within the low-dimensional solver space, effectively mitigating reward hacking while enhancing performance in complex text-to-image (T2I) generation tasks. In addition, our method is flexible and can serve as a plugin ( EPD-Plugin ) to improve existing ODE samplers. Extensive experiments demonstrate the effectiveness of EPD-Solver . On validation benchmarks, at the same latency level of 5 NFE, the distilled EPD-Solver achieves state-of-the-art FID scores of 4.47 on CIFAR-10, 7.97 on FFHQ, 8.17 on ImageNet, and 8.26 on LSUN Bedroom, surpassing existing learning-based solvers by a significant margin. On T2I benchmarks, our RL-tuned EPD-Solver significantly improves human preference scores on both Stable Diffusion v1.5 and SD3-Medium. Notably, it outperforms the official 28-step baseline of SD3-Medium with only 20 steps, effectively bridging the gap between inference efficiency and high-fidelity generation. Fig. 1 : Comparison of various solvers on diffusion models. We compare the FID versus latency (ms) across different NFE settings on a NVIDIA 4090. Our proposed EPD-Solver shows superior image quality without increasing latency. I Introduction Diffusion models (DMs) [ 54 , 13 , 46 ] have emerged as a leading paradigm in generative modeling, delivering state-of-the-art performance across image synthesis [ 46 , 49 , 23 ] and video generation [ 3 , 14 , 78 , 73 ] . These models generate data by iteratively refining noisy inputs through a sequential denoising process. While this mechanism produces high-fidelity outputs, the requirement for multi-step sequential evaluation introduces substantial latency, rendering real-time sampling inefficient. Fig. 2 : Computation graphs of various ODE solvers. (a) DDIM solver [ 55 ] (Euler’s method) adopts the rectangle rule that uses the gradient at the start point: {\mathbf{d}}_{t_{n+1}}=\bm{\epsilon}_{\theta}({\mathbf{x}}_{t_{n+1}},t_{n+1}) . disclose EDM solver [ 15 ] (Heun’s method) uses the trapezoidal rule that averages the gradients of both the start and the end timesteps, i.e ., {\mathbf{d}}_{t_{n+1}}=\bm{\epsilon}_{\theta}({\mathbf{x}}_{t_{n+1}},t_{n+1}) and {\mathbf{d}}_{t_{n}}^{\prime}=\bm{\epsilon}_{\theta}({\mathbf{x}}_{t_{n}}^{\prime},t_{n}) , where {\mathbf{x}}_{t_{n}}^{\prime} is the additional evaluation given by Euler’s method. (c) AMED solver [ 80 ] optimizes a small network g_{\phi}(\cdot) to output an intermediate timestep s_{n}\in(t_{n},t_{n+1}) to compute the gradient: {\mathbf{d}}_{s_{n}}=\bm{\epsilon}_{\theta}({\mathbf{x}}_{s_{n}},s_{n}) . Since AMED introduces a network in sequential computation, its latency is slightly higher than that of other solvers, as shown in fig. 1 . (d) Our EPD-Solver leverage K parallel gradients to achieve more accurate integral approximation. We optimize K intermediate timesteps \tau_{n}^{1},\dots,\tau_{n}^{K} , compute their gradients {\mathbf{d}}_{\tau_{n}^{1}},\dots,{\mathbf{d}}_{\tau_{n}^{K}} , and combine them via a simplex-weighted sum. To mitigate this inefficiency, recent research has focused on accelerating the sampling process through various approaches, such as solver-based, distillation-based and parallelism-based methods. Solver-based methods aim to develop fast numerical solvers to reduce the number of sampling steps [ 55 , 15 , 34 , 35 , 32 , 71 , 76 , 80 , 19 , 63 ] . However, aggressive step reduction often leads to significant truncation errors, causing quality degradation at low function evaluations (NFEs). Distillation-based methods establish a direct mapping between noise and data [ 79 , 36 , 33 , 1 , 50 , 40 , 56 , 37 , 18 ] , achieving extreme acceleration (e.g., one-step generation). Yet, they incur high training costs and lack the flexibility to trade speed for quality. Parallelism-based methods [ 53 , 27 , 26 , 5 ] attempt to trade computation for speed, but this direction remains under-explored for quality enhancement. In this paper, we seek to combine the strengths of these approaches by investigating solver-based methods under low-latency constraints. We propose the E nsemble P arallel D irection ( EPD ) solver, a novel approach that leverages extra parallel computation to minimize truncation errors in each ODE step without increasing wall-clock time. Unlike standard solvers that rely on a single gradient evaluation (e.g., DDIM [ 55 ] ) or sequential multi-point estimates (e.g., EDM [ 15 ] and AMED [ 71 ] ), our method concurrently evaluates gradients at multiple learned intermediate timesteps within a single integration interval (see Figure 2 ). By aggregating these parallel gradient estimates via a weighted combination, we achieve a significantly more accurate approximation of the integral direction, which is theoretically grounded in the mean value theorem for vector-valued functions [ 39 ] . Crucially, since these gradient computations are independent of each other, they can be efficiently parallelized on modern hardware. This allows EPD-Solver to enhance sampling fidelity with negligible latency overhead. As in Figure 1 , EPD-Solver consistently achieves better FID scores than existing ODE solvers at comparable inference latencies on CIFAR [ 22 ] . We adopt a two-stage optimization framework to decide the intermediate evaluation timesteps and their combining weights. (1) In the first stage, we distill a few-step EPD sampler by optimizing its learnable parameters to approximate the trajectories generated by a high-NFE teacher solver. However, in the extremely low-step regime, distillation alone is insufficient. It not only struggles to learn an accurate mapping from noise to teacher trajectories [ 59 ] but also falls short in aligning with human perceptual preferences. For large-scale text-to-image (T2I) diffusion models, human preference is better characterized by semantic and perceptual alignment rather than strict trajectory consistency. Motivated by this, (2) in the second stage , we perform Residual Dirichlet Policy Optimization (RDPO) , where the solver is reparameterized as a stochastic policy initialized using the parameters distilled in Stage 1. This parameterization induces a structured, simplex-constrained policy space that supports stable and efficient optimization of human-aligned rewards via a PPO [ 52 ] variant. Moreover, our framework is lightweight and plug-and-play: it optimizes only a few solver parameters while freezing the backbone, thereby reducing tuning cost, improving RL stability, and preserving generation robustness; the resulting solver can also be seamlessly integrated into existing ODE samplers as EPD-Plugin . We evaluate EPD-Solver across a diverse set of image generation models spanning resolutions from 32 to 1024, including standard unconditional and class-conditional benchmarks (CIFAR-10 [ 22 ] , FFHQ [ 17 ] , ImageNet [ 47 ] , LSUN Bedroom [ 70 ] ) and large-scale T2I models (Stable Diffusion v1.5 [ 46 ]