Stream-DiffVSR: Low-Latency Streamable Video Super-Resolution via Auto-Regressive Diffusion

Hau-Shiang Shiu, Chin-Yang Lin, Zhixiang Wang, Chi-Wei Hsiao, Po-Fan Yu, Yu-Chih Chen, Yu-Lun Liu · 2025

Read Stream-DiffVSR: Low-Latency Streamable Video Super-Resolution via Auto-Regressive Diffusion by Hau-Shiang Shiu, Chin-Yang Lin, Zhixiang Wang, Chi-Wei Hsiao, Po-Fan Yu, Yu-Chih Chen, Yu-Lun Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-23709.txt.

Introduction

Hau-Shiang Shiu and six co-authors make diffusion-based video upscaling fast enough for live streams. Diffusion super-resolution produces excellent quality, but ordinary versions peek at future frames and need many denoising steps, so they cannot run online. Stream-DiffVSR instead works strictly frame by frame, using only the past.

Three pieces make it work: a distilled denoiser that needs only four steps, a temporal guidance module that feeds motion-aligned cues into each frame's denoising. A lightweight decoder that sharpens detail while keeping consecutive frames coherent. On a high-end graphics card it processed high-definition frames in about a third of a second each, cut per-frame runtime by more than a hundredfold against earlier diffusion methods. Shrank the wait for the first frame from over an hour to a fraction of a second.

Catalog record

CREATORHau-Shiang Shiu, Chin-Yang Lin, Zhixiang Wang, Chi-Wei Hsiao, Po-Fan Yu, Yu-Chih Chen, Yu-Lun Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-23709

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Stream-DiffVSR: Low-Latency Streamable Video Super-Resolution via Auto-Regressive Diffusion Hau-Shiang Shiu, Chin-Yang Lin, Zhixiang Wang, Chi-Wei Hsiao, Po-Fan Yu, Yu-Chih Chen, Yu-Lun Liu Abstract Diffusion-based video super-resolution (VSR) methods achieve strong perceptual quality but remain impractical for latency-sensitive settings due to reliance on future frames and expensive multi-step denoising. We propose Stream-DiffVSR, a causally conditioned diffusion framework for efficient online VSR. Operating strictly on past frames, it combines a four-step distilled denoiser for fast inference, an Auto-regressive Temporal Guidance (ARTG) module that injects motion-aligned cues during latent denoising, and a lightweight temporal-aware decoder with a Temporal Processor Module (TPM) that enhances detail and temporal coherence. Stream-DiffVSR processes 720p frames in 0.328 seconds on an RTX4090 GPU and significantly outperforms prior diffusion-based methods. Compared with the online SOTA TMP ( Zhang et al. 2024b ) , it boosts perceptual quality (LPIPS +0.095) while reducing latency by over 130 × \times . Stream-DiffVSR achieves the lowest latency reported for diffusion-based VSR, reducing initial delay from over 4600 seconds to 0.328 seconds, thereby making it the first diffusion VSR method suitable for low-latency online deployment. Project page: https://jamichss.github.io/stream-diffvsr-project-page/ Figure 1 : Comparison of visual quality and inference speed across various categories of VSR methods. Stream-DiffVSR achieves superior perceptual quality (lower LPIPS) and maintains comparable runtime to CNN- and Transformer-based online models, while also demonstrating significantly reduced inference latency compared to existing offline approaches. Best and second-best results are marked in red and green . 1 Introduction Video super-resolution (VSR) aims to reconstruct high-resolution (HR) videos from low-resolution (LR) inputs and is vital in applications such as surveillance, live broadcasting, video conferencing, autonomous driving, and drone imaging. It is increasingly important in low-latency rendering workflows, including neural rendering and resolution upscaling in game engines and AR/VR systems, where latency-aware processing is crucial for visual continuity. Specifically, latency-sensitive processing involves two key aspects: per-frame inference time (throughput) and end-to-end system latency (delay between receiving an input frame and producing its output). Existing VSR methods often struggle with this trade-off. While CNN- and Transformer-based models offer a balance between efficiency and quality, they fall short in perceptual detail. Diffusion-based models excel in perceptual quality due to strong generative priors, but suffer from high computational cost and reliance on future frames, making them impractical for time-sensitive video applications. In this paper, we propose Stream-DiffVSR , a diffusion-based method specifically tailored to online video super-resolution, effectively bridging the gap between high-quality but slow diffusion methods and fast but lower quality CNN- or Transformer-based methods. Unlike previous diffusion-based VSR approaches (e.g., StableVSR ( Rota et al. 2024 ) and MGLD-VSR ( Yang et al. 2024 ) ) that typically require 50 or more denoising steps and bidirectional temporal information, our method leverages diffusion model distillation to significantly accelerate inference by reducing denoising steps to just four. Additionally, we introduce an Auto-regressive Temporal Guidance mechanism and an Auto-regressive Temporal-aware Decoder to effectively exploit temporal information from previous frames, significantly enhancing temporal consistency and perceptual fidelity. Fig. 1 illustrates the core advantage of our approach by comparing visual quality and runtime across various categories of video super-resolution methods. Our Stream-DiffVSR achieves superior perceptual quality (measured by LPIPS ( Zhang et al. 2018 ) ) and temporal consistency, outperforming existing unidirectional CNN- and Transformer-based methods (e.g., MIA-VSR ( Zhou et al. 2024b ) , RealViformer ( Zhang and Yao 2024 ) , TMP ( Zhang et al. 2024b ) ). Notably, Stream-DiffVSR offers significantly faster per-frame inference than prior diffusion-based approaches (e.g., StableVSR ( Rota et al. 2024 ) , MGLD-VSR ( Yang et al. 2024 ) ), attributed to our use of a distilled 4-step denoising process and a lightweight temporal-aware decoder. In addition, existing diffusion-based methods, such as StableVSR ( Rota et al. 2024 ) typically rely on bidirectional or future-frame information, resulting in prohibitively high processing latency that is not suitable for online scenarios. Specifically, for a 100-frame video, StableVSR (46.2 s/frame) would incur an initial latency exceeding 4600 seconds on an RTX 4090 GPU, as it requires processing the entire sequence before generating even the first output frame. In contrast, our Stream-DiffVSR operates in a strictly causal, autoregressive manner, conditioning only on the immediately preceding frame. Consequently, the initial frame latency of Stream-DiffVSR corresponds to a single frame’s inference time (0.328 s/frame), reducing the latency by more than three orders of magnitude compared to StableVSR. This significant latency reduction demonstrates that Stream-DiffVSR effectively unlocks the potential of diffusion models for practical, low-latency online video super-resolution. Table 1 : Comparison of diffusion-based VSR methods. We report online capability, inference steps, runtime (FPS on 720p, RTX 4090), maximum end-to-end latency (sec), and whether each method uses distillation, temporal modeling, or offline future frames. OOM denotes out-of-memory, and - indicates missing public inference results. Notably, Stream-DiffVSR is the only diffusion-based method that runs in a strictly online, past-only setting with the lowest latency. # of FPS Max Temporal Temporal Method Online Steps @720p latency Distill Input Decoder StableVSR ( Rota et al. 2024 ) ✗ 50 0.02 4620 ✗ Future/Bi-dir ✗ MGLD-VSR ( Yang et al. 2024 ) ✗ 50 0.02 218 ✗ Future/Bi-dir ✓ Upscale-A-Video ( Zhou et al. 2024a ) ✗ 30 OOM - ✗ Future/Bi-dir ✓ DiffVSR ( Li et al. 2025 ) ✗ – – - ✗ Future/Bi-dir ✓ VEnhancer ( He et al. 2024 ) ✗ 15 OOM - ✗ Future/Bi-dir ✓ Stream-DiffVSR (ours) ✓ 4 3.05 0.328 ✓ Past-only ✓ To summarize, the main contributions of this paper are: • We introduce the first diffusion-based framework explicitly designed for online, low-latency video super-resolution, achieving efficient inference through distillation from 50 denoising steps down to 4 steps. • We propose a novel Auto-regressive Temporal Guidance mechanism and a Temporal-aware Decoder to effectively leverage temporal information only from past frames, significantly enhancing perceptual quality and temporal consistency. • Extensive experiments demonstrate that our approach outperforms existing methods across key perceptual and temporal consistency metrics while achieving practical inference speeds, thereby making diffusion-based VSR applicable for real-world online scenarios. To contextualize our contributions, Table 1 compares recent diffusion-based VSR methods in terms of online inference capability, runtime efficiency, and temporal modeling. Our method uniquely achieves online low-latency inference while preserving high visual quality and temporal stability. This substantial latency reduction of over three orders of magnitude compared to prior diffusion-based VSR models demonstrates that Stream-DiffVSR is uniquely suited for low-latency online applications such as video conferencing and AR/VR. 2 Related Work Video Super-resolution. VSR methods reconstruct high-resolution videos from low-resolution inputs through CNN-based approaches ( Xue et al. 2019 ; Tian et al. 2020 ; Wang et al. 2019 ; Chan et al. 2021 ; Chan et al. 2022a ; Sun et al. 2025c ) , deformable convolutions ( Tian et al. 2020 ; Dai et al. 2017 ; Zhu et al. 2019 ) , online processing ( Zhang et al. 2024b ) , recurrent architectures ( Sajjadi et al. 2018 ; Fuoli et al. 2019 ; Isobe et al. 2020 ; Yi et al. 2019 ; Li et al. 2020 ) , flow-guided methods ( Youk et al. 2024 ; Guo et al. 2024 ; Liu et al. 2019 ) , and Transformer-based models ( Vaswani et al.

Read the complete text