Progressive Image Restoration via Text-Conditioned Video Generation

Peng Kang 1 1 , Xijun Wang 2 1 , Yu Yuan 2 · 2025

Read Progressive Image Restoration via Text-Conditioned Video Generation by Peng Kang 1 1 , Xijun Wang 2 1 , Yu Yuan 2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-02273.txt.

Introduction

Text-to-video models have learned to generate smooth, plausible motion, and that talent may double as a repair kit for damaged images. Peng Kang, Xijun Wang, and Yu Yuan repurpose the video model CogVideo in 2025 for progressive image restoration, fine-tuning it to produce restoration trajectories instead of natural motion.

The damaged image is treated as the first frame of a short film whose final frame is the restored picture. Generating the in-between steps gradually may recover detail more gracefully than a single leap. The study opens a new use for video generation's temporal strength, suggesting that restoration is really motion through quality space.

Catalog record

CREATORPeng Kang 1 1 , Xijun Wang 2 1 , Yu Yuan 2
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-02273

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Progressive Image Restoration via Text-Conditioned Video Generation 1 Both authors contributed equally to this work. Peng Kang is the corresponding author. Email: [email protected] Peng Kang 1 1 , Xijun Wang 2 1 , Yu Yuan 2 Abstract Recent text-to-video models have demonstrated strong temporal generation capabilities, yet their potential for image restoration remains underexplored. In this work, we repurpose CogVideo for progressive visual restoration tasks by fine-tuning it to generate restoration trajectories rather than natural video motion. Specifically, we construct synthetic datasets for super-resolution, deblurring, and low-light enhancement, where each sample depicts a gradual transition from degraded to clean frames. Two prompting strategies are compared: a uniform text prompt shared across all samples, and a scene-specific prompting scheme generated via LLaVA multi-modal LLM and refined with ChatGPT. Our fine-tuned model learns to associate temporal progression with restoration quality, producing sequences that improve perceptual metrics such as PSNR, SSIM, and LPIPS across frames. Extensive experiments show that CogVideo effectively restores spatial detail and illumination consistency while maintaining temporal coherence. Moreover, the model generalizes to real-world scenarios on the ReLoBlur dataset without additional training, demonstrating strong zero-shot robustness and interpretability through temporal restoration. I Introduction Recent advances in text-to-video diffusion models [ 18 , 12 , 5 , 19 , 3 , 15 , 17 ] have enabled the generation of visually coherent and temporally consistent video content directly from textual prompts. Among these models, CogVideo [ 16 ] has demonstrated remarkable capability in synthesizing high-quality videos that capture realistic motion dynamics and scene details. While such models have primarily been used for creative video generation, their potential for visual restoration and enhancement tasks remains largely unexplored. In this work, we propose a novel paradigm that leverages CogVideo as a generative bridge for visual enhancement tasks, including image super-resolution, deblurring, and low-light enhancement. Instead of treating these tasks as independent image-to-image translation problems, we view them through the lens of video progression learning: that is, generating a sequence of temporally ordered frames that evolve from degraded inputs (e.g., low-resolution, dark, or blurred) toward their enhanced versions. By fine-tuning CogVideo on specialized datasets that encode these degradations as temporal transitions, the model learns to capture the intrinsic relationships between visual degradation and recovery in a dynamic, context-aware manner. Our key insight is that temporal consistency and motion priors learned by CogVideo can be exploited to improve spatial detail recovery and stability in single-frame restoration tasks. Traditional restoration methods [ 20 , 4 , 13 , 8 , 6 ] , including CNN- and transformer-based architectures, often focus on spatial fidelity without modeling progressive transitions. In contrast, our approach harnesses the temporal diffusion process of CogVideo to implicitly model the gradual transformation from degraded to clean states, yielding results that are both perceptually smooth and detail-preserving. To evaluate this idea, we construct and fine-tune CogVideo on three newly formalized datasets: • Resolution Progression Dataset – videos simulating transitions from low to high resolution; • Illumination Progression Dataset – videos progressing from dark to well-lit conditions; • Blur-to-Sharp Dataset – videos depicting gradual recovery from motion or defocus blur. Comprehensive experiments show that our fine-tuned CogVideo not only produces visually appealing videos but also serves as a powerful generator for restoration tasks when sampled frame-by-frame. Furthermore, to assess the generalization of our approach, we apply the fine-tuned CogVideo model to the ReLoBlur dataset [ 7 ] , a challenging real-world video deblurring benchmark. Despite not being explicitly trained on real-world motion blur, our model demonstrates superior visual restoration and temporal stability, producing realistic and coherent enhancement sequences. In summary, the main contributions of this work are threefold: • We propose a novel framework that repurposes CogVideo for image enhancement tasks by modeling restoration as a video progression process. • We construct and release three benchmark datasets representing progressive visual degradation-to-restoration transitions. • We demonstrate that fine-tuned CogVideo achieves competitive performance in super-resolution, deblurring, and low-light enhancement, offering a new perspective on leveraging text-to-video diffusion models for image restoration. II Methodology Building upon the motivation introduced earlier, our goal is to repurpose CogVideo, a large-scale text-to-video diffusion model, for image restoration and enhancement tasks. Rather than designing a new network architecture from scratch, we exploit the generative prior and temporal reasoning capabilities already embedded in CogVideo. By fine-tuning it on carefully constructed progression datasets, we guide the model to interpret “visual enhancement” as a gradual temporal transformation. Our method proceeds in three stages. First, we formalize the restoration problem as a temporal video generation task, where a degraded image progressively evolves into its clean counterpart. Second, we construct specialized datasets for super-resolution, deblurring, and low-light correction, providing both uniform and scene-adaptive text conditions for comparison. Finally, we fine-tune CogVideo using a framework that conditions the diffusion process on textual descriptions of the restoration. The overall pipeline is designed to be modular and data-driven, allowing CogVideo to generalize across multiple degradation types. II-A Problem Definition Given a degraded image I_{d} (e.g., low-resolution, blurred, or low-light) and its corresponding clean target I_{c} , conventional restoration methods aim to learn a direct mapping f_{\theta}:I_{d}\rightarrow I_{c}, (1) where f_{\theta} is typically a CNN- or Transformer-based model optimized for pixel-level fidelity or perceptual similarity. In this work, we reinterpret image restoration as a temporal video generation problem. Instead of predicting a single restored frame, we model the progressive enhancement process as a temporally ordered sequence \mathcal{V}=\{I_{1},I_{2},\dots,I_{T}\}, (2) where I_{1}=I_{d} is the degraded input, I_{T}=I_{c} is the restored target, and the intermediate frames \{I_{t}\}_{t=2}^{T-1} represent gradual transitions from degraded to enhanced states. Let G_{\phi} denote the CogVideo generator parameterized by \phi . During inference, the model takes as input both a textual prompt p that semantically describes the desired restoration process and the initial degraded frame I_{d} . The generation process can thus be formulated as \mathcal{V}=G_{\phi}(p,I_{d}), (3) where \mathcal{V} is a short video depicting a smooth enhancement trajectory from I_{d} to I_{T} . The final frame I_{T} is extracted as the restored image, while preceding frames visualize interpretable intermediate restoration stages. This formulation offers two key advantages. First, by leveraging the temporal diffusion dynamics of CogVideo, the model captures the gradual, continuous nature of visual enhancement. Second, the text prompt p provides an additional conditioning signal that aligns the generation process. Through this joint conditioning, the model learns to couple textual guidance with spatiotemporal visual transformations, enabling a unified framework for super-resolution, deblurring, and low-light enhancement. Fig. 1: Illustration of the CogVideo fine-tuning process for image restoration. The model receives a video showing progressive visual enhancement (e.g., from low-resolution to super-resolution) and a corresponding textual prompt describing the scene and restoration dynamics. Using LoRA fine-tuning, CogVideo learns to align temporal visual improvements with text semantics, enabling it to generate restoration sequences conditioned on both the input frame and the prompt. II-B Dataset Construction To enable CogVideo to learn visual restoration as a progressive video generation task, we construct three domain-specific datasets derived from the DIV2K high-resolution image corpus [ 14 ] . Each dataset contains short video clips, each composed of T=9 frames showing a smooth transition from a degraded state to a clean target. For every image in the DIV2K dataset, we generate two distinct videos using random degradation parameters

Read the complete text