InstaVSR: Taming Diffusion for Efficient and Temporally Consistent Video Super-Resolution

Jintong Hu, Bin Chen, Zhenyu Hu, Jiayue Liu, Guo Wang, Lu Qi · 2026

Read InstaVSR: Taming Diffusion for Efficient and Temporally Consistent Video Super-Resolution by Jintong Hu, Bin Chen, Zhenyu Hu, Jiayue Liu, Guo Wang, Lu Qi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-26134.txt.

Introduction

Hu and co-authors present InstaVSR, a light diffusion method for upscaling video. Diffusion models give sharp frames but can flicker between them and cost too much. InstaVSR trims the backbone to a pruned one-step design. Recurrent training with flow-guided rules steadies frame-to-frame motion. Adversarial training in both latent and pixel space guards image quality. On an RTX 4090, it handles a 30-frame 2K video in under a minute. Memory use stays at seven gigabytes. Transitions look smoother than in rival methods, the authors report.

Catalog record

CREATORJintong Hu, Bin Chen, Zhenyu Hu, Jiayue Liu, Guo Wang, Lu Qi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40534
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.26134

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

InstaVSR: Taming Diffusion for Efficient and Temporally Consistent Video Super-Resolution Jintong Hu, Bin Chen, Zhenyu Hu, Jiayue Liu, Guo Wang, Lu Qi Abstract Video super-resolution (VSR) seeks to reconstruct high-resolution frames from low-resolution inputs. While diffusion-based methods have substantially improved perceptual quality, extending them to video remains challenging for two reasons: strong generative priors can introduce temporal instability, and multi-frame diffusion pipelines are often too expensive for practical deployment. To address both challenges simultaneously, we propose InstaVSR, a lightweight diffusion framework for efficient video super-resolution. InstaVSR combines three ingredients: (1) a pruned one-step diffusion backbone that removes several costly components from conventional diffusion-based VSR pipelines, (2) recurrent training with flow-guided temporal regularization to improve frame-to-frame stability, and (3) dual-space adversarial learning in latent and pixel spaces to preserve perceptual quality after backbone simplification. On an NVIDIA RTX 4090, InstaVSR processes a 30-frame video at 2K × \times 2K resolution in under one minute with only 7 GB of memory usage, substantially reducing the computational cost compared to existing diffusion-based methods while maintaining favorable perceptual quality with significantly smoother temporal transitions. † † ∗ Contributing equally to this work. † Corresponding Author: Lu Qi, [email protected] 1 Introduction Figure 1 : Visual and quantitative comparison of video super-resolution methods on real-world content. Left: super-resolution results showing the low-quality input and the outputs of different methods, with zoomed-in patch comparisons. Right: comparison of computational efficiency (latency, parameters, and VRAM) and visual quality metrics. Our method achieves a strong trade-off between efficiency and perceptual quality compared with RealViformer [ 62 ] and SeedVR2 [ 47 ] . Video super-resolution (VSR) aims to reconstruct high-resolution frames from their low-resolution counterparts and has become an essential technique in a wide range of applications including video streaming [ 31 , 63 , 60 ] , surveillance enhancement [ 13 , 32 , 11 ] , film restoration [ 1 , 36 ] , and medical imaging enhancement [ 46 , 18 , 3 ] . Over the past few years, significant progress has been made in addressing both synthetic and real-world degradations [ 26 , 57 , 6 , 19 ] . Early CNN-based approaches, such as EDVR [ 49 ] , BasicVSR [ 4 ] , and BasicVSR++ [ 5 ] , introduced effective temporal alignment and propagation mechanisms that substantially improved reconstruction quality. Subsequently, GAN-based methods such as SRGAN [ 23 ] , ESRGAN [ 51 ] , and Real-ESRGAN [ 50 ] leveraged adversarial training [ 14 ] to produce realistic high-frequency details, achieving remarkable perceptual quality. More recently, diffusion models [ 16 , 35 , 42 ] have emerged as a powerful generative paradigm. Building upon large-scale text-to-image frameworks such as Stable Diffusion [ 38 ] and SDXL [ 37 ] , researchers have adapted these models to image super-resolution [ 39 , 25 , 48 , 53 ] and video restoration [ 65 , 56 , 8 ] , establishing diffusion-based methods as the new frontier in video super-resolution. While the generative strength of diffusion models is undeniable, it introduces a fundamental tension when applied to video super-resolution: the conflict between powerful detail synthesis and temporal consistency. Through large-scale pretraining, diffusion models acquire a rich understanding of visual content and can hallucinate plausible textures and fine-grained details absent in the low-quality input [ 20 , 10 ] , which is precisely what makes them superior to CNN-based and GAN-based methods in perceptual quality. However, in the video setting, this generative power becomes a double-edged sword. The model’s tendency toward over-hallucination causes it to synthesize pseudo-textures that do not correspond to the original content, producing visually distracting artifacts [ 48 , 59 ] . Meanwhile, the inherent stochasticity of the diffusion sampling process means that independently processed adjacent frames may receive inconsistent detail patterns, giving rise to temporal flickering [ 24 , 22 ] . Although several methods attempt to mitigate this through 3D attention mechanisms or inter-frame feature propagation, these solutions typically come at the cost of substantially increased computational overhead. Beyond temporal coherence, the multi-frame nature of video introduces a second critical bottleneck: computational efficiency. A typical video stream contains more than 20 frames per second, meaning that the total processing time can be tens of times greater than for a single image. While stacking multiple frames along the batch dimension enables parallel processing, this strategy incurs enormous GPU memory consumption. Diffusion models further exacerbate this issue, as their iterative denoising procedures typically require tens of sampling steps [ 42 , 33 ] , resulting in prohibitively long inference times that become even more severe in the multi-frame setting. In practice, many existing diffusion-based methods can only process short video clips at resolutions around 480p or below on datacenter-grade accelerators, severely limiting deployment on consumer-grade devices. Furthermore, training such models typically requires 8 or more high-end GPUs, placing them beyond the reach of some research groups. These constraints collectively represent a major barrier to translating the generative quality of diffusion models into practical video super-resolution systems. To address these two core challenges simultaneously, we propose InstaVSR , a novel and lightweight diffusion framework specifically designed for efficient video super-resolution. To overcome the efficiency bottleneck, our approach systematically removes redundant components from conventional diffusion-based VSR pipelines: we discard the 3D VAE encoder, eliminate 3D attention modules, and remove condition injection mechanisms, thereby substantially reducing both computational cost and memory footprint. To preserve and enhance the generative capability of the pruned model, we employ a hybrid adversarial training strategy that deploys discriminators at both feature and pixel levels, enabling the lightweight model to maintain the powerful detail synthesis capacity of the original diffusion model. To tackle the temporal consistency and over-hallucination problem, we introduce a recurrent consistency scheme that propagates hidden states across consecutive frames, explicitly enforcing temporal coherence while simultaneously suppressing pseudo-texture generation, further complemented by a temporal smoothness loss that directly penalizes inter-frame flicker. The resulting system is highly efficient in practice: on a single NVIDIA RTX 4090, InstaVSR processes a 30-frame sequence at 2K \times 2K resolution in 0.77 minute while consuming only approximately 7 GB of GPU memory. Extensive experiments on standard benchmarks demonstrate that our method achieves favorable perceptual quality with significantly smoother temporal transitions, striking an effective trade-off between generation quality and computational efficiency. Our contributions can be summarized as follows: • We propose InstaVSR, a lightweight diffusion-based video super-resolution framework that systematically removes redundant architectural components including the 3D VAE encoder, 3D attention, and condition injection mechanisms, enabling efficient inference on consumer-grade GPUs. • We propose a dual-space adversarial training approach coupled with a recurrent consistency scheme and temporal smoothness loss. The dual-space adversarial training preserves the generative capability of the pruned model, while the recurrent consistency and temporal regularization jointly enforce temporal stability and suppress over-hallucination. • We show on standard VSR benchmarks that InstaVSR achieves competitive perceptual quality and temporal consistency while using substantially less memory and inference time than prior diffusion-based VSR baselines. Figure 2 : The main pipeline of GaussianSR. GaussianSR begins with an encoder that extracts feature representations from the input image, followed by Selective Gaussian Splatting which assigns a learnable Gaussian kernel to each pixel, converting dicrete feature points into

Read the complete text