Frames2Residual: Spatiotemporal Decoupling for Self-Supervised Video Denoising
Mingjie Ji, Zhan Shi, Kailai Zhou, Zixuan Fu, Xun Cao · 2026
Read Frames2Residual: Spatiotemporal Decoupling for Self-Supervised Video Denoising by Mingjie Ji, Zhan Shi, Kailai Zhou, Zixuan Fu, Xun Cao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-10417.txt.
Introduction
Mingjie Ji and co-authors present Frames2Residual, a self-supervised method for cleaning noisy video that splits time and space into two stages. Blind-spot networks must hide the center pixel, which blocks spatial clues and costs texture. Stage one learns inter-frame consistency with a blind strategy, giving a steady temporal anchor. Stage two, a non-blind spatial refiner, uses that anchor to bring back the center frame and recover fine detail while keeping time stability. Tests on sRGB and raw video beat prior self-supervised methods.
Catalog record
| CREATOR | Mingjie Ji, Zhan Shi, Kailai Zhou, Zixuan Fu, Xun Cao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37495 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.10417
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Frames2Residual: Spatiotemporal Decoupling for Self-Supervised Video Denoising Mingjie Ji, Zhan Shi, Kailai Zhou, Zixuan Fu, Xun Cao Abstract Self-supervised video denoising methods typically extend image-based frameworks into the temporal dimension, yet they often struggle to integrate inter-frame temporal consistency with intra-frame spatial specificity. Existing Video Blind-Spot Networks (BSNs) require noise independence by masking the center pixel, this constraint prevents the use of spatial evidence for texture recovery, thereby severing spatiotemporal correlations and causing texture loss. To address this, we propose Frames2Residual (F2R), a spatiotemporal decoupling framework that explicitly divides self-supervised training into two distinct stages: blind temporal consistency modeling and non-blind spatial texture recovery. In Stage 1, a blind temporal estimator learns inter-frame consistency using a frame-wise blind strategy, producing a temporally consistent anchor. In Stage 2, a non-blind spatial refiner leverages this anchor to safely reintroduce the center frame and recover intra-frame high-frequency spatial residuals while preserving temporal stability. Extensive experiments demonstrate that our decoupling strategy allows F2R to outperform existing self-supervised methods on both sRGB and raw video benchmarks. 1 Introduction Figure 1 : Comparison of self-supervised video denoising paradigms during the inference phase. (a) Video Noise2Noise. Warping-based supervision violates noise independence and causes warping artifacts, leading to blurred details. (b) Video Blind-Spot Network. The inherent imposition of pixel discontinuities severs vital spatiotemporal correlations , resulting the loss in texture. (c) The proposed F2R framework. F2R employs joint spatial inputs \{z_{i}\} , where z_{i}=\mathrm{cat}(\hat{x}_{i},r_{i}) , r_{i}=y_{i}-\hat{x}_{i} , and \hat{x}_{i} is derived from a pre-trained image denoiser \mathcal{D} . (d) Visualization of F2R framework. The gray dashed line illustrates the temporal consistency established by Stage 1. The final output acquires spatial specificity while maintaining temporal consistency, thereby effectively restoring spatiotemporal correlations . Deep learning has achieved remarkable progress in image and video restoration. While image denoising leverages spatial self-similarity to suppress noise [ 42 , 11 ] , its efficacy is fundamentally constrained by the spatial entanglement of signal and noise [ 18 ] . To address this, video denoising introduces temporal dimension, utilizing correlations across frames to robustly identify coherent structures against random fluctuations [ 28 , 33 , 31 ] . However, the success of these methods predominantly hinges on supervised learning with paired datasets. Ground Truth (GT) is rarely available in practical applications like live-cell fluorescence microscopy or ultrafast transient imaging [ 41 , 26 ] . This limitation necessitates self-supervised approaches that operate directly on raw noisy videos. Most existing self-supervised video methods extend image-based frameworks, primarily Noise2Noise (N2N) [ 16 ] and Blind-Spot Networks (BSNs) [ 13 , 38 ] to the temporal dimension. Early Video N2N approaches [ 5 , 6 , 8 ] typically rely on motion compensation to align adjacent frames as supervision targets. Unfortunately, the interpolation inherent in warping inevitably disrupts pixel continuity and noise statistics [ 4 ] , thereby fundamentally violating the independence assumption critical for self-supervision. Consequently, these methods often suffer from severe ghosting artifacts or over-smoothing in dynamic scenes (see Fig. 1 (a)). To avoid this issue, Video BSNs frameworks [ 27 ] provide a statistically grounded alternative. By approximating the conditional expectation \mathbb{E}[x_{t,i,j}|\Omega] solely from the spatiotemporal neighborhood \Omega (see Fig. 1 (b)), they circumvent the ill-posed motion alignment problem, ensuring that the training objective remains pristine and free from warping-induced corruptions. However, this strict exclusion strategy entails a specific trade-off. While this formulation strictly satisfies the noise independence assumption, the exclusion of direct spatial evidence leads to inherent pixel discontinuities, thereby disrupting continuous local variations [ 35 , 15 ] . Consequently, this severs vital spatiotemporal correlations , defined as the structural continuity between a pixel and its local spatiotemporal neighborhood, ultimately leading to texture loss. Simultaneously enforcing a blind-spot constraint for noise independence and utilizing direct spatial evidence for texture recovery are fundamentally conflicting objectives. To resolve this inherent conflict, we propose Frames2Residual (F2R) , a two-stage framework that fundamentally decouples temporal consistency modeling from spatial texture restoration. To enable this spatiotemporal decoupling and strictly focus the network on recovering high-frequency spatial textures lost by BSNs, we introduce a residual-domain learning formulation. Specifically, F2R leverages pre-computed image denoising outputs as a structural baseline (illustrated in Fig. 1 (c)). By explicitly offloading static structural modeling to this prior, the model can concentrate exclusively on recovering spatial residuals. Similarly, to relieve the network from complex motion modeling during temporal aggregation, we introduce pre-computed optical flow. Directly applying pre-computed flow fields for explicit image warping inevitably corrupts vital details. Instead, we embed them into flow-guided implicit alignment modules, empowering the network to actively suppress resampling artifacts. Under this residual-domain formulation, F2R utilizes the blindly estimated temporal consistency as a supervision target, creating a safe pathway to reintroduce direct spatial evidence for explicit texture recovery (illustrated in Fig. 1 (d)). Specifically, Stage 1 trains a Blind Estimator (BE) via a frame-wise blind strategy. Because the target frame is unobservable, a Flow-Guided Attention Alignment Module (FAAM) replaces standard U-Net skip connections to stabilize cross-frame correlations and extract a temporally consistent anchor. Guided by this anchor, Stage 2 trains a non-blind Spatial Refiner (SR) via a recorruption strategy (substituting the original noisy center frame with a noise-injected anchor) to force active spatial texture recovery. With the target visible, a Flow-Guided Deformable Alignment Module (FDAM) leverages this spatial evidence to correct flow misalignments. By synthesizing complete spatiotemporal features, the SR precisely restores the missing spatial residuals absent in the structural baseline. Ultimately, this decoupled design unifies the inter-frame temporal consistency established in Stage 1 with the intra-frame spatial specificity recovered in Stage 2, successfully restoring the spatiotemporal correlations . Extensive evaluations on both sRGB and raw video benchmarks demonstrate that our proposed framework achieves state-of-the-art video denoising performance against current unsupervised methods. In summary, our main contributions are: • We identify an inherent conflict in self-supervised video denoising: enforcing noise independence through blind-spot prediction prevents the model from exploiting direct spatial evidence for texture recovery. To resolve this issue, we propose F2R, a two-stage framework that decouples temporal consistency modeling from spatial texture restoration. • To enable this decoupling, we introduce a residual-domain learning formulation that leverages image denoising priors as a structural baseline, allowing the model to focus on recovering high-frequency spatial residuals while effectively aggregating temporal information. • Our approach surpasses current unsupervised methods on both raw and sRGB video denoising benchmarks, demonstrating its superior effectiveness. 2 Related Work 2.1 Supervised Video Denoising Supervised video denoising typically relies on GT sequences to learn spatiotemporal features [ 1 , 2 , 17 , 19 , 20 , 23 , 31 , 33 , 34 , 41 , 39 ] . TOFlow [ 39 ] and DVDNet [ 30 ] explicitly computed optical flow to align neighboring frames. To avoid artifacts from flow estimation errors, FastDVDNet [ 31 ] proposed a cascade of U-Nets without explicit motion compensation. EDVR [ 34 ] and RViDeNet [ 41 ] utilize Deformable Convolution Networks (DCN) [ 34 ] for feature-level alignment, while Transformer-based models like VRT [ 19 ] , RVRT [ 20 ] , and Tempformer [ 28 ] leverage attention mechanisms for long-range temporal modeling. Despite their high performance, the heavy reliance on paired clean-noisy data severely restricts their applicability in