Accelerating Video Inverse Problem Solvers with Autoregressive Diffusion Models

Taesung Kwon, Jonghyun Park, Hyungjin Chung, Jong Chul Ye · 2026

Read Accelerating Video Inverse Problem Solvers with Autoregressive Diffusion Models by Taesung Kwon, Jonghyun Park, Hyungjin Chung, Jong Chul Ye (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20624.txt.

Introduction

Taesung Kwon and three colleagues speed up video inverse-problem solvers with autoregressive diffusion models. Diffusion priors help restore video without task-specific training, but full-video restoration brings high first-frame delay and repeated passes hurt throughput. Their AVIS solver restores video in a streaming way, starting reverse diffusion from a measurement-consistent guess. It cuts initial delay from 114 seconds to 4 seconds and lifts throughput from 0.71 to 1.18 frames per second.

Catalog record

CREATORTaesung Kwon, Jonghyun Park, Hyungjin Chung, Jong Chul Ye
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57909
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.20624

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Accelerating Video Inverse Problem Solvers with Autoregressive Diffusion Models Taesung Kwon, Jonghyun Park, Hyungjin Chung, Jong Chul Ye Abstract Diffusion models provide powerful priors for zero-shot video inverse problems, but their real-time deployment is hindered by two inefficiencies: high initial latency caused by holistic video restoration, and low throughput resulting from multiple VAE passes to enforce measurement consistency in pixel space. To overcome these limitations, we propose Autoregressive Video Inverse problem Solver (AVIS). The AVIS framework leverages autoregressive video diffusion models to restore videos in a streaming manner, naturally eliminating latency bottlenecks. Specifically, AVIS initializes reverse diffusion with a measurement-consistent estimate, reducing the required sampling steps. Compared to leading non-autoregressive solvers, AVIS drastically reduces initial latency from 114s to 4s and increases throughput from 0.71 to 1.18 FPS while achieving superior restoration quality. We further introduce a highly accelerated variant, dubbed AVIS Flash , that enforces measurement consistency solely on the first chunk. AVIS Flash substantially boosts throughput to 5.91 FPS on a single RTX 4090 GPU while maintaining competitive performance and achieving a favorable efficiency–performance trade-off, paving the way toward real-time deployment. Figure 1: The AVIS framework leverages autoregressive video diffusion models to restore videos in a streaming manner, naturally eliminating latency bottlenecks. Examples show diverse video inverse problems that AVIS can solve in a zero-shot manner. † † footnotetext: * First authors. † Corresponding authors. Project page is available here . 1 Introduction Video inverse problems aim to reconstruct a clean video \mathbf{x} from a degraded measurement \mathbf{y} ( e.g. , low-resolution, masked, or blurred). The degradation process is mathematically formulated as \mathbf{y}=\mathcal{A}(\mathbf{x})+\boldsymbol{n}, (1) where \mathcal{A} is a degradation operator and \boldsymbol{n} denotes measurement noise. These video restoration tasks are inherently ill-posed, as a single measurement \mathbf{y} can correspond to many possible clean videos \mathbf{x} . To resolve this ambiguity, diffusion and flow-based models [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 ] have recently emerged as powerful priors for solving video inverse problems in a zero-shot manner [ 10 , 11 , 12 , 13 , 14 , 15 ] , producing results that are both perceptually realistic and faithful to the measurements. Despite their successful restoration performance, these solvers remain far from real-time deployment due to two inefficiencies: 1. High initial latency: Recent Diffusion model-based Video Inverse problem Solvers (DVIS) [ 11 , 12 , 13 , 14 , 15 ] restore the entire video in a holistic manner, sampling the entire video sequence simultaneously. Consequently, the first frame cannot be displayed until the last frame has been restored, introducing latency equal to the time required to restore all frames. 2. Low throughput: Modern diffusion and flow-based video models [ 16 , 17 , 18 , 19 , 20 ] operate in the latent space of a variational autoencoder (VAE) [ 21 ] for computational efficiency. Recent DVIS based on such models [ 13 , 15 ] require multiple VAE passes during restoration, restricting throughput to below 1 FPS. While some works [ 22 , 23 ] bypass this with auxiliary latent operators, they require degradation-specific training, sacrificing zero-shot generality. Here, we identify a promising yet underexplored direction toward the efficient deployment of DVIS: harnessing the potential of autoregressive (AR) video diffusion models [ 24 , 25 , 26 ] . Unlike non-autoregressive solvers that process the entire video jointly (Figure 2 (a)), autoregressive solvers restore videos in a streaming manner, naturally eliminating the latency bottleneck. However, simply adopting an AR backbone does not automatically resolve the issue of low throughput, as it still requires multiple VAE passes for measurement consistency updates. To overcome this limitation, we propose A utoregressive V ideo I nverse problem S olver ( AVIS ), which accelerates restoration by initializing the reverse diffusion process with a measurement-consistent estimate, thereby reducing the required sampling steps. We draw inspiration from CCDF [ 27 ] , which demonstrates that initializing the reverse diffusion process from a coarse estimate, predicted by an auxiliary pretrained restorer, exploits stochastic contraction to significantly reduce the required sampling steps. Unlike CCDF, AVIS does not rely on an auxiliary pretrained restorer; instead, it obtains an initial restoration by directly minimizing the measurement consistency objective. By utilizing this estimate to initialize the reverse diffusion, AVIS begins the restoration process from a better starting point while enforcing measurement consistency for every video chunk (Figure 2 (b)). To push efficiency further, we introduce a highly accelerated variant, AVIS Flash . Unlike AVIS, which applies iterative measurement guidance to every chunk, AVIS Flash enforces measurement consistency only on the first video chunk. Interestingly, we observe that this is sufficient to restore the entire sequence (Figure 2 (c)). Once the first chunk is fully restored and accurately grounded to the measurement, subsequent chunks can be generated through autoregressive propagation from this corrected prefix, completely bypassing explicit measurement guidance. This eliminates iterative VAE passes for subsequent chunks, accelerating restoration by 5 \times while preserving zero-shot performance. Compared to the leading non-autoregressive solver [ 15 ] , AVIS reduces initial latency from 114s to 4s and improves throughput from 0.71 to 1.18 FPS while achieving superior restoration quality. Pushing efficiency further, AVIS Flash substantially boosts throughput to 5.91 FPS on a single RTX 4090 GPU while maintaining competitive performance, paving the way toward real-time deployment. Our main contributions can be summarized as follows: • AVIS. We investigate the potential of adopting autoregressive video diffusion models for zero-shot video restoration. AVIS accelerates restoration by initializing the reverse diffusion process with a measurement-consistent estimate, reducing the required sampling steps while enforcing measurement consistency for every video chunk. • AVIS Flash . We introduce a highly accelerated variant of AVIS that retains the initialization while enforcing measurement consistency solely on the first video chunk. This substantially increases throughput, paving the way for real-time video inverse problem solvers. Figure 2: Overview of our proposed AVIS and AVIS Flash framework. (a) Non-autoregressive restoration processes the entire video holistically, suffering from high initial latency. (b) AVIS restores videos in a streaming manner and reduces sampling steps via measurement-consistent initialization while enforcing measurement updates for every video chunk. (c) AVIS Flash retains the same initialization as AVIS but applies measurement updates only to the first video chunk. Subsequent chunks ( n\geq 2 ) are restored through autoregressive propagation from the corrected prefix, eliminating iterative VAE passes and dramatically accelerating throughput. 2 Related Work 2.1 Diffusion for Video Inverse Problems Diffusion model-based Inverse problem Solvers (DIS) [ 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 ] enable repurposing the diffusion or flow-based models [ 1 , 2 , 3 , 4 , 6 , 7 , 8 , 9 ] originally trained to model the data distribution p(\mathbf{x}) as powerful plug-and-play priors. By guiding the sampling trajectory toward the posterior p(\mathbf{x}|\mathbf{y}) , these methods achieve strong zero-shot performance across a wide range of image restoration tasks, producing solutions that are perceptually natural and consistent with the measurement \mathbf{y} . Motivated by this success, Diffusion model-based Video Inverse problem Solvers (DVIS) [ 11 , 12 , 13 , 14 , 15 ] extend this paradigm

Read the complete text