All-in-One Video Restoration under Smoothly Evolving Unknown Weather Degradations Thanks: This work was supported in part by the National Key R&D Program of China (2023YFA1008501) and the National Natural Science Foundation of China (NSFC) under grant 624B2049 and U22B2035. (Corresponding author: Xiaopeng Fan.) Thanks: Wenrui Li, Hongtao Chen, Yao Xiao, Wangmeng Zuo and Xiaopeng Fan are with the Department of Computer Science and Technology, Harbin Institute of Technology, Harbin 150001, China. (e-mail: liwr618@163.com; ht166chen@163.com; 25S003001@stu.hit.edu.cn; wmzuo@hit.edu.cn; fxp@hit.edu.cn). Thanks: Jiantao Zhou is with the State Key Laboratory of Internet of Things for Smart City and the Department of Computer and Information Science, University of Macau, Macau, China (e-mail: jtzhou@um.edu.mo). Thanks: Yonghong Tian is with the School of AI for Science, the Shenzhen Graduate School, Peking University, Shenzhen, China, the Peng Cheng Laboratory, Shenzhen, China, and also with the School of Computer Science, Peking University, Beijing, China (e-mail: yhtian@pku.edu.cn). Wenrui Li, Hongtao Chen, Yao Xiao, Wangmeng Zuo Abstract All-in-one image restoration aims to recover clean images from diverse unknown degradations using a single model. But extending this task to videos faces unique challenges. Existing approaches primarily focus on frame-wise degradation variation, overlooking the temporal continuity that naturally exists in real-world degradation processes. In real world, degradation types and intensities evolve smoothly over time, and multiple degradations may coexist or transition gradually. In this paper, we introduce the Smoothly Evolving Unknown Degradations (SEUD) scenario to simulate such real-world processes, where both the active degradation set and degradation intensity change continuously over time. To support this scenario, we design a flexible synthesis pipeline that generates temporally coherent videos with single, compound, and evolving degradations. To address the challenges in the SEUD scenario, we propose an all-in-One Recurrent Conditional and Adaptive prompting Network (ORCANet). First, a Coarse Intensity Estimation Dehazing (CIED) module estimates haze intensity using physical priors and provides coarse dehazed features as initialization. Second, a Flow Prompt Generation (FPG) module extracts degradation features. FPG generates both static prompts that capture segment-level degradation types and dynamic prompts that adapt to frame-level intensity variations. Furthermore, a label-aware supervision mechanism improves the discriminability of static prompt representations under different degradations. Extensive experiments show that ORCANet achieves superior restoration quality, temporal consistency, and robustness over image and video-based baselines. Code is available at https://github.com/Friskknight/ORCANet-SEUD. I Introduction All-in-one image restoration (AiOIR) aims to recover clean images with a single model under diverse and unknown degradations [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 ] . Unlike task-specific methods [ 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 ] , which require separate models for denoising, deblurring, or dehazing, all-in-one approaches handle multiple degradation types in a single framework. This generality is important for real-world scenarios, where degradations are often diverse, composite, and unpredictable. Fig. 1: Examples of SEUD scenario. (a) is a real-world SEUD scenario captured by smartphone. (b) and (c) are synthetically generated sequences. In (c), the set of present degradation types changes over time in addition to intensity variation. The numerical values below each frame represent the estimated normalized degradation intensities used in synthesis process. With the increasing demand for video processing, AiOIR is gradually extending from images to videos. Video restoration is more challenging than image restoration because it must capture both spatial and temporal dependencies [ 29 , 30 ] . Moreover, in video all-in-one restoration, degradations are not only spatially complex but also temporally variant. These temporal variations introduce additional challenges. Existing studies have made preliminary research in this direction. Zhao et al. introduce the Time-varying Unknown Degradation (TUD) scenario [ 31 ] , which extends the all-in-one restoration problem from static images to temporally dynamic videos. In this setting, both the degradation type and its severity can change across frames, reflecting more realistic and challenging conditions. To construct such a dataset, the TUD framework independently apply degradations to each frame of a clean video. The degradations are randomly sampled from a set including noise, blur, and compression. This design enables comprehensive evaluation of all-in-one models under dynamic and uncertain degradation conditions. However, there are three limitations in previous work on all-in-one restoration for videos. (1) Limited modeling of temporal correlation of degradation in TUD. In practice, factors such as rain density, haze thickness, and snow motion change smoothly over time, and different weather types may coexist or transition gradually. In contrast, the TUD setting [ 31 ] applies degradations to each frame independently without temporal linkage. This design neglects the physical continuity of real-world degradations. This design breaks the natural continuity of degradation and produces unrealistic temporal changes. (2) Limited modeling of cross-frame dependencies in prompt design. Prompt-based AiOIR methods mainly focus on single-image restoration. Textual and multimodal prompts require precise external inputs and face cross-modal alignment issues and model complexity [ 32 , 33 , 34 , 35 , 36 ] . Visual prompts extract embeddings from the degraded image; but they are usually generated per frame and remain independent across time [ 4 , 37 , 38 ] . As a result, they fail to capture how degradations evolve. (3) Limited modeling of compound degradations. Many all-in-one restoration methods struggle to handle co-occurring degradations. For example, PromptIR [ 37 ] and PIP [ 4 ] do not include mechanisms to prevent similarity or conflict among task prompts. T3DiffWeather [ 36 ] uses a predefined prompt pool, which limits coverage of unseen or mixed degradations. TAP [ 39 ] models task relations with low-rank and contrastive constraints but relies on predefined degradation correlations, which limits flexibility in complex or dynamic scenarios. Fig. 2: Comparison of recovery results in SEUD scenarios. Existing restoration methods show artifacts and instability under SEUD, while our approach maintains more consistent recovery. To address the aforementioned limitations, we introduce a new Smoothly Evolving Unknown Degradations (SEUD) scenario. In SEUD, the active compound degradation types and severity both evolve smoothly over time. In this work, we focus on weather degradations within this scenario. Figure 1 shows three cases: (a) a real case with a fixed type set and varying intensity (rain+haze), (b) a synthetic case with a fixed type set (snow+haze) and (c) a synthetic case where both the type set and the intensity vary (rain+snow+haze). In every frame, parameters such as rain density, transparency, angle, and streak dimensions change smoothly. This setting better matches real processes and provides a more challenging benchmark for all-in-one video restoration. Figure 2 further shows that representative restoration models struggle under SEUD, exhibiting residual artifacts and inconsistent recovery when degradation type and intensity evolve over time. Because real SEUD data are difficult to collect, we design a flexible weather synthesis pipeline that takes a clean video and a depth map as inputs. For rain and snow, we generalize static image overlay models to a dynamic particle model. Each particle has physical attributes, including depth position, fall speed, transparency, and other relevant properties. The attribute of depth would control occlusion between foreground and background. Other attributes, such as length, thickness, speed, and transparency, are also correlated with depth. Specifically, distant particles are thinner, shorter, slower, and more transparent, and they are more easily occluded by the foreground. We then define several smooth intensity trajectories over time to modulate each degradation. A video may include segments with no degradation, a single degradation, or compound degradations. The pipeline enforces frame-to-frame coherence while maintaining diversity. In this paper, we propose a novel ORCANet for the SEUD scenario. As shown in Fig. 2 , our method produces more uniform and stable restoration across evolving degradations compared with existing approaches. Specifically, ORCANet consists of two key components: a Coarse Intensity Estimation Dehazing (CIED) module and a Flow Prompt Generation (FPG) module. CIED first estimates haze intensity coarsely and applies a physics-inspired dehazing before feature propagation. This step mitigates the multiplicative attenuation of haze that masks other degradations. FPG extends static visual prompts from AiOIR to temporal prompts for video. It generates static and dynamic prompts. Specifically, the static prompt is shared across multiple frames. It is computed on key frames every p frames and reused for intermediate frames. The static prompt serves as a segment-level condition that separates degradation types and remains stable within short temporal windows. We supervise static prompt with a label-aware metric loss, which pulls descriptors of similar types together and pushes dissimilar ones apart. This supervision mitigates prompt similarity or conflict and improves perception of compound degradations. The dynamic prompt varies with each frame. It propagates with features and fuses with the newly generated dynamic prompt at the current frame to maintain temporal linkage. Finally, the static and dynamic prompts are multiplied, and the fused prompt guides degradation perception and adaptive restoration. Extensive experiments on synthetic and real data show that ORCANet achieves strong performance on SEUD and outperforms existing methods. The main contributions of this work are as follows: • We introduce Smoothly Evolving Unknown Degradations (SEUD) scenario, which incorporats temporal continuity in both composition and intensity variation, better matching real-world conditions. • We propose a simple yet flexible pipeline for generating SEUD video sequences, capable of modeling the realistic temporal evolution of multiple degradations. We release the synthetic code and dataset to facilitate benchmarking. • We develop ORCANet, an all-in-one video restoration model that extends prompt-based degradation modeling from images to videos, enabling adaptive restoration under smoothly evolving degradations for SEUD scenario. • Extensive experiments demonstrate that our approach achieves superior performance on both synthetic and real-world SEUD videos. II Related work II-A Weather Degradation Models Early studies in adverse weather restoration often rely on physical models that describe how degradations form under different weather conditions. These formulations provide interpretable priors for image restoration. For hazy scenes, the Koschmieder’s atmospheric scattering model [ 40 ] is widely used. It assumes that the observed image L(x) is a combination of clear image H(x) attenuated by the transmission map t(x) and the global atmospheric light A_{\infty} : L(x)=H(x)\cdot t(x)+A_{\infty}\cdot(1-t(x)), (1) where the transmission t(x)=e^{-\beta d(x)} depends on the scattering coefficient \beta and the scene depth d(x) . Eq. ( 1 ) explains two key haze effects: contrast loss through multiplicative attenuation t(x) and color shifting through the additive airlight term. Many classical and modern dehazing methods estimate t(x) and A_{\infty} from a single image, sometimes with priors, learning-based estimators, or depth surrogates [ 41 , 10 , 11 , 12 , 13 ] . Rain and snow introduce a semi-transparent precipitation layer that occludes or blends with the scene. A unified and practical model uses per-pixel alpha compositing: L(x)=H(x)\cdot(1-\alpha(x))+R(x)\cdot\alpha(x), (2) where R(x) denotes the rain or snow layer, and \alpha(x)\!\in\![0,1] is a transparency map that controls the visibility of degradations. In rain, R(x) often contains thin, oriented, motion-blurred streaks. In snow, R(x) contains larger, blob-like, or out-of-focus particles. Many restoration methods exploit this layered view and design priors or networks for R(x) and \alpha [ 19 , 20 , 21 ] . These degradation models offer simplified yet effective representations of adverse weather phenomena. They provide strong physical priors that guide the design of weather-specific restoration methods, such as dehazing [ 10 , 11 , 12 , 13 , 14 ] , deraining [ 19 , 15 , 16 , 17 ] , and desnowing [ 18 , 20 , 21 , 22 , 23 ] . While these models enable effective restoration under static and single-frame assumptions, they do not explicitly model the temporal continuity of weather degradations. In practice, both degradation composition and intensity evolve smoothly over time, which limits the applicability of these formulations in dynamic video scenarios. Fig. 3: SEUD Weather Synthesis Pipeline. Left: The input consists of clear video frames and their corresponding depth maps. Middle: Each particle is assigned a random depth and a set of physical attributes which correlate with depth. Distant particles are thinner, shorter, slower, and more transparent, and are more likely to be occluded by the foreground visibility mask. Their motion follows gravity \mathbf{g} and the time-varying wind field \mathbf{w}(t) . Right: Example intensity functions \beta(t) and f(t) control the temporal evolution of haze and precipitation, producing synthetic videos with continuous, mixed, and time-varying degradations. II-B All-in-one Restoration The goal of all-in-one image restoration is to recover high-quality clean images from low-quality degraded inputs using a single unified network. In the field of weather degradation removal, the pioneering work All-in-One [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 ] inspires a series of unified models. TransWeather [ 2 ] and TKL [ 42 ] design specific encoder-decoder structures to handle different weather degradations. AWRCP [ 43 ] introduces a high-quality codebook prior and employs a pretrained VQGAN to recover fine texture details under adverse weather. Zhu et al. [ 44 ] present a two-stage training strategy to improve adaptability across various weather conditions. More recently, T3DiffWeather [ 36 ] proposes a prompt pool mechanism, where the network selects the most relevant prompts based on image features for targeted restoration. Research further extends from image-level restoration to video-level scenarios. ViWS-Net [ 45 ] introduces temporally-active messenger tokens to collect weather-related information during video encoding. Diff-TTA [ 46 ] combines diffusion models with test-time adaptation to remove adverse weather degradations under both known and unknown conditions. Xu et al. [ 47 ] provide pseudo-labels for real photos using various large visual language models and restore images with a semi-supervised framework. CUDN [ 48 ] proposes a sequence-adaptive degradation estimator for predicting the degradation matrix of the entire video sequence. However, video all-in-one restoration is still in an early stage. Compared with image tasks, video restoration faces the key challenge that degradations often change dynamically over time. Zhao et al. [ 31 ] discuss time-varying unknown degradations (TUD) in earlier all-in-one work. Yet, their method assumes that degradations are independent across frames. In real-world scenarios, degradations are usually continuous and correlated between adjacent frames. This work therefore focuses on the problem of smoothly evolving weather degradations in videos and explores effective modeling. II-C Prompt Learning for Restoration Prompt learning has recently gained popularity in all-in-one image restoration, as it provides additional guidance for handling diverse degradations. Prompts can be categorized into visual prompts [ 4 , 37 , 38 , 49 , 36 , 39 ] , textual prompts [ 32 , 33 , 34 ] , and multimodal prompts [ 35 , 36 ] . Textual and multimodal prompts require precise text inputs as external guidance, but face challenges of cross-modal alignment and large-scale language model complexity. Visual prompts instead extract embeddings from degraded images to guide the network. However, they may suffer from parameter conflicts across tasks. Among visual prompt methods, PromptIR [ 37 ] introduces a lightweight prompt block to enhance all-in-one image restoration. ProRes [ 38 ] integrates target visual prompts into input images and employs weighted combinations for customized restoration. PIP [ 4 ] proposes a Prompt-In-Prompt framework that restores clean images relying only on the input without prior knowledge of degradation types. More recently, T3DiffWeather [ 36 ] develops a prompt pool mechanism that adaptively selects the most relevant prompts according to image features at inference time. TAP [ 39 ] proposes a parameter-efficient task-aware prompting framework that models inter-task relationships through low-rank decomposition and contrastive constraints to mitigate task conflicts in all-in-one adverse weather image restoration. Most existing methods design prompts for image restoration. Although these approaches can be applied to frame-wise video restoration, the generated prompts are usually independent across frames. They fail to capture temporal dependencies in video degradations. In the task of SEUD, degradations across frames exhibit both shared patterns (e.g., consistent weather type within a short period) and unique variations (e.g., intensity or wind direction). To address this problem, we propose to divide prompts into two components: The first component captures temporally invariant degradation cues that are shared among nearby frames, while the second models frame-specific variations that evolve smoothly over time. This design allows the model to exploit both temporal consistency and local dynamic changes. By combining the two, the network learns temporally coherent yet adaptive representations for video restoration under SEUD conditions. III The proposed method In this section, we first present the physical models of time-varying weather degradation for video in Section III-A . Sections III-B – III-D then describe the overall framework of our method, which integrates an intensity-aware mechanism, an all-in-one visual prompting design for video, and a unified restoration network. Finally, Section III-E details the design of loss functions and training strategies. III-A Time-Varying Weather Degradation for Video The adverse weather in the video exhibits continuous and time-varying behavior. The intensity evolves smoothly or in segments over time. Different weather types may overlap on the temporal axis and transition into each other. To address the difficulty of obtaining real-world data, we propose a practical data synthesis method. We adopt a unified view that combines volumetric scattering and foreground particle occlusion. We model haze, rain, and snow in space and time, and provide a matching synthesis pipeline. It explicitly encodes the temporal evolution of intensity, which supports robust generalization under continuous, blind, and mixed degradations. The simple synthesis process is illustrated in Fig. 3 . Video Haze Model Video haze follows a volumetric scattering formulation. Let H(x,t)\in\mathbb{R}^{H\times W\times 3} denote the clean video, where x is the pixel location and t is time. We extend Eq. ( 1 ) to the video domain and adopt a time-varying transmission T(x,t) in the [ 40 ] : L(x,t)=H(x,t)\,T(x,t)+A_{\infty}\!\left(1-T(x,t)\right), (3) where A_{\infty} is the atmospheric light, which is approximately constant within a short time window. The transmission term T(x,t) represents the proportion of scene radiance that reaches the camera after scattering attenuation, and it depends jointly on the scene depth D(x) and a scattering coefficient \beta(t) : T(x,t)=\exp\!\big(-\beta(t)\,D(x)\big), (4) where the coefficient \beta(t) describes the temporal variation of haze density, which can evolve smoothly or in segments over time. Eq. ( 3 ) captures both the depth-dependent contrast attenuation and the color shift due to airlight. Camera motion and slight illumination changes do not affect the physical relation but only affect H(x,t) . Video Rain and Snow Model Precipitation introduces a semi-transparent foreground layer between the camera and the scene. We model it at the particle level and use alpha compositing for occlusion and blending. Let Z_{\rm scene}(x) be the scene depth at pixel x . For the k -th particle at time t , let Z_{k}(x,t) be its depth along the viewing ray and P_{k}(x,t) its radiance in the image. The visibility mask is formulated as: M_{k}(x,t)=\mathbf{1}\!\left\{\,Z_{k}(x,t)