SAIP: A Plug-and-Play Scale-adaptive Module in Diffusion-based Inverse Problems
Lingyu Wang, Xiangming Meng · 2025
Read SAIP: A Plug-and-Play Scale-adaptive Module in Diffusion-based Inverse Problems by Lingyu Wang, Xiangming Meng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-24580.txt.
Introduction
Diffusion models can restore damaged images by sampling from a posterior, but the balance between prior and likelihood is set by a hand-tuned fixed scale. Lingyu Wang and Xiangming Meng argue that static balance is the weak point: the ideal scale shifts across timesteps and across tasks. Their module, SAIP, refines that scale adaptively at each timestep, with no retraining and no change to the diffusion backbone.
SAIP plugs into existing samplers built on common likelihood estimators. Across diverse restoration tasks, including hard cases, it consistently improved reconstruction quality. A small plug-and-play part thus buys broader generalization for a whole family of solvers.
Catalog record
| CREATOR | Lingyu Wang, Xiangming Meng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-24580
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SAIP: A Plug-and-Play Scale-adaptive Module in Diffusion-based Inverse Problems Lingyu Wang, Xiangming Meng Abstract Solving inverse problems with diffusion models has shown promise in tasks such as image restoration. A common approach is to formulate the problem in a Bayesian framework and sample from the posterior by combining the prior score with the likelihood score. Since the likelihood term is often intractable, estimators like DPS, DMPS, and π \pi GDM are widely adopted. However, these methods rely on a fixed, manually tuned scale to balance prior and likelihood contributions. Such a static design is suboptimal, as the ideal balance varies across timesteps and tasks, limiting performance and generalization. To address this issue, we propose SAIP, a plug-and-play module that adaptively refines the scale at each timestep without retraining or altering the diffusion backbone. SAIP integrates seamlessly into existing samplers and consistently improves reconstruction quality across diverse image restoration tasks, including challenging scenarios. † † address: Zhejiang University, ZJU-UIUC Institute, Haining,China 1 Introduction Image restoration is a typical inverse problem that aims to recover a clean, high-quality image x\in\mathbb{R}^{N\times 1} from a degraded observation y\in\mathbb{R}^{M\times 1} . Depending on the degradation process, typical tasks include denoising [ 1 ] , deblurring [ 2 ] , and inpainting [ 3 ] , all of which can be cast as a linear inverse problem: y=Ax+n, (1) where A\in\mathbb{R}^{M\times N} is a linear forward operator and n\sim\mathcal{N}(0,\sigma^{2}I) denotes Gaussian noise. Due to system limitations and environmental disturbances, the measurement y is often incomplete, noisy, or otherwise degraded. The resulting ill-posedness generally implies non-uniqueness of the solution x , making restoration fundamentally challenging. To resolve this ambiguity and select a plausible solution, one must incorporate prior knowledge or regularization about the target signal. From a Bayesian perspective [ 4 ] , this amounts to performing posterior inference p(x\mid y) via Bayes’ rule, p(x\mid y)\propto p(x)\,p(y\mid x), (2) where p(x) and p(y\mid x) are the prior and the likelihood, respectively. The prior p(x) encapsulates assumptions about natural images and thus plays a central role in inverse problems. Classical choices include hand-crafted, analytical priors (e.g., sparsity [ 5 ] , low-rank structure [ 6 ] , and total variation [ 5 ] ) as well as architecture-induced priors such as the Deep Image Prior [ 7 ] . However, these priors often fall short of capturing the full complexity of natural images. Recently, generative models, especially diffusion models [ 8 , 9 ] , have emerged as powerful data-driven priors for inverse problems [ 10 , 2 , 11 , 12 , 13 , 14 , 15 , 16 ] . Following Bayes’ rule ( 2 ), posterior inference with diffusion models requires the posterior score: \nabla_{x_{t}}\log p(x_{t}|y)=\nabla_{x_{t}}\log p(x_{t})+\lambda\nabla_{x_{t}}\log p(y|x_{t}), (3) where x_{t} is a noisy latent in the diffusion process and \lambda is a scaling factor. The prior score \nabla_{x_{t}}\log p(x_{t}) is provided by the pretrained diffusion model, whereas the likelihood score \nabla_{x_{t}}\log p(y\mid x_{t}) is generally intractable [ 2 , 17 , 11 , 10 ] . A number of approximations have been proposed, including Diffusion Posterior Sampling (DPS) [ 10 ] , Pseudoinverse-Guided Diffusion Models ( \pi GDM) [ 12 ] , and Diffusion Model Posterior Sampling (DMPS) [ 11 ] . Figure 1 : Representative results. Rows 1 and 3: FFHQ at 256\times 256 ; Rows 2 and 4: LSUN-Bedroom at 256\times 256 . Despite their success, existing methods typically rely on a fixed, manually tuned \lambda to balance the prior and likelihood contributions. This practice is tedious and often suboptimal. As observed in DMPS [ 11 ] , the final reconstruction quality is highly sensitive to \lambda : even small deviations can markedly degrade sampling performance or lead to instability. Moreover, a single, manually chosen coefficient lacks robustness across noise levels and timesteps, necessitating extensive per-setting tuning. This not only reduces efficiency but also exposes \lambda selection as a key bottleneck that limits the stability and generalization of current diffusion-based inverse solvers. To address this, inspired by the self-adaptive guidance in CFG-Zero* [ 18 ] for conditional generation, we extend the idea to inverse problems and propose a self-adaptive scaling module ( SAIP ). SAIP dynamically estimates \lambda during inference, allowing the guidance strength at each timestep to adapt to task and noise specific characteristics, thereby combining prior and likelihood in a more robust manner. SAIP is plug-and-play: it integrates into existing diffusion-based inverse problem solvers without retraining or modifying the diffusion backbone. Comparisons between the baseline and its ”SAIP +” version are shown in Fig. 1 . The main contributions are summarized as follows: (1) We extend the idea of adaptive scaling to inverse problems and introduce SAIP, a plug-and-play module that dynamically estimates the scale during inference. To the best of our knowledge, this is the first work to introduce adaptive scaling into diffusion-based inverse problem solvers. SAIP also can be directly applied to a wide range of methods that combine estimated likelihood and prior scores, such as DiffStateGrad [ 19 ] , without retraining. (2) We integrate SAIP into several representative diffusion-based solvers and evaluate it on classical image restoration tasks. Experiments span both standard settings and more challenging scenarios with severe degradation, high noise and latent domain. The results demonstrate consistent improvements over nearly all baselines, while incurring only a minor additional inference cost. 2 METHOD To address the fixed scale challenge in diffusion-based inverse problem solvers, we propose a plug-and-play module called SAIP , which dynamically adjusts the balance between the prior and likelihood scores during posterior sampling. Our design is inspired by the self-adaptive guidance introduced in CFG-Zero* [ 18 ] for classifier-free guidance [ 20 ] : \nabla_{x_{t}}\log\tilde{p}(x_{t}|y)=\omega\nabla_{x_{t}}\log p(x_{t}|y)+s(1-\omega)\nabla_{x_{t}}\log p(x_{t}), (4) where \omega is the guidance strength and s is a learnable scaling factor. To adapt this idea to inverse problems, we replace the posterior score in right side of Eq.( 4 ) with the sum of the likelihood score and the prior score. \displaystyle\nabla_{x_{t}}\log\tilde{p}(x_{t}|y) \displaystyle=\big[s(1-\omega)+\omega\big]\nabla_{x_{t}}\log p(x_{t}) (5) \displaystyle+\omega\nabla_{x_{t}}\log p(y|x_{t}). Then, we reorganize the equation by factoring out the prior score term with a coefficient of 1 from the posterior score expression. \displaystyle\nabla_{x_{t}}\log\tilde{p}(x_{t}|y) \displaystyle=\underbrace{\nabla_{x_{t}}\log p(x_{t})+\omega\nabla_{x_{t}}\log p(y|x_{t})}_{\text{as same as Eq.~\eqref{eq:estimatedlikelihood} when $\omega=\lambda$}} (6) \displaystyle+\underbrace{\bm{\left[(s-1)(1-\omega)\right]\nabla_{x_{t}}\log p(x_{t})}}_{\textbf{Additional part}}, Thus, compared to Eq. ( 3 ), the SAIP formulation essentially adds an adaptive offset (s-1)(1-\omega) to the prior score, making it possible to refine the balance between the two components adaptively. In particular, when setting s=1 , Eq. ( 6 ) reduces to the original formulation of the baseline method. The remaining problem is to calculating s , we minimize the gap between the estimated posterior score and the ground-truth posterior score: \mathcal{L}(s)=\left\|\nabla_{x_{t}}\log\tilde{p}_{\theta}(x_{t}|y)-\nabla_{x_{t}}\log p^{*}(x_{t}|y)\right\|^{2}, (7) where the first term is the estimated posterior score modulated by SAIP, as defined in Eq. ( 5 ), and p^{*}(x_{t}|y) denotes the true posterior. Since p^{*} is intractable, We replace the posterior score in Eq. ( 6 ) and introduce a new parameter \omega^{\star}=\omega-1 , which allows us to rewrite the loss as \mathcal{L}(s)=\left\|\begin{aligned} &\nabla_{x_{t}}\log\tilde{p}_{\theta}(x_{t}|y)-\nabla_{x_{t}}\log p^{*}(x_{t}|y)\\ &\quad+\omega^{\star}\left(-s\nabla_{x_{t}}\log p(x_{t})+\nabla_{x_{t}}\log p(y|x_{t})\right)\end{aligned}\right\|^{2}. (8) Applying the triangle inequality, we obtain \displaystyle\mathcal{L}(s)\leq \displaystyle\;\left\|\nabla_{x_{t}}\log p_{\theta}(x_{t}\mid y)\right\|^{2}+\left\|\nabla_{x_{t}}\log p^{*}(x_{t}\mid y)\right\|^{2} \displaystyle+\left\|\omega^{\star}\left(-s\nabla_{x_{t}}\log p(x_{t})+\nabla_{x_{t}}\log p(y\mid x_{t})\right)\right\|^{2}, (9) which shows that minimizing \mathcal{L}(s) is equivalent to minimizing the upper-bound \mathcal{L}_{\text{upper}}=\left\|-s\nabla_{x_{t}}\log p(x_{t})+\nabla_{x_{t}}\log p(y|x_{t})\right\|^{2}. (10) To