Stabilizing Streaming Video Geometry via Dynamic Feature Normalization
Xiaoyang Lyu ∗1 Muxin Liu ∗1 Xiaoshan Wu 1 Ruicheng Wang 2 Yi-Hua Huang 1 Yang-Tian Sun 1 Shaoshuai Shi 3 Xiaojuan Qi 1⋄ 1 The University of Hong Kong 2 USTC 3 Voyager Research, Didi Chuxing ∗ Equal Contribution: {shawlyu, mxliu}@connect.hku.hk ⋄ Corresponding Author: [email protected] · 2026
Read Stabilizing Streaming Video Geometry via Dynamic Feature Normalization by Xiaoyang Lyu ∗1 Muxin Liu ∗1 Xiaoshan Wu 1 Ruicheng Wang 2 Yi-Hua Huang 1 Yang-Tian Sun 1 Shaoshuai Shi 3 Xiaojuan Qi 1⋄ 1 The University of Hong Kong 2 USTC 3 Voyager Research, Didi Chuxing ∗ Equal Contribution: {shawlyu, mxliu}@connect.hku.hk ⋄ Corresponding Author: [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-25308.txt.
Introduction
Xiaoyang Lyu and colleagues steady video models' frame-by-frame 3D geometry from one moving camera. Strong single-image models drift badly over time, as their depth scale and shift wander with internal feature statistics. The team traces the fault to those statistics and adds Dynamic Feature Normalization, a small module that steadies them in streaming video. Tuning only that module, about 2 percent of extra weights, with the backbone frozen preserves single-image accuracy. Across four benchmarks it removes temporal artifacts and beats prior streaming methods by up to 14 percent.
Catalog record
| CREATOR | Xiaoyang Lyu ∗1 Muxin Liu ∗1 Xiaoshan Wu 1 Ruicheng Wang 2 Yi-Hua Huang 1 Yang-Tian Sun 1 Shaoshuai Shi 3 Xiaojuan Qi 1⋄ 1 The University of Hong Kong 2 USTC 3 Voyager Research, Didi Chuxing ∗ Equal Contribution: {shawlyu, mxliu}@connect.hku.hk ⋄ Corresponding Author: [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55316 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.25308
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Stabilizing Streaming Video Geometry via Dynamic Feature Normalization Xiaoyang Lyu ∗1 Muxin Liu ∗1 Xiaoshan Wu 1 Ruicheng Wang 2 Yi-Hua Huang 1 Yang-Tian Sun 1 Shaoshuai Shi 3 Xiaojuan Qi 1⋄ 1 The University of Hong Kong 2 USTC 3 Voyager Research, Didi Chuxing ∗ Equal Contribution: {shawlyu, mxliu}@connect.hku.hk ⋄ Corresponding Author: [email protected] Abstract Consistent 3D geometry estimation from streaming RGB input is crucial for real-world applications such as autonomous driving, embodied AI, and large-scale reconstruction. While modern monocular geometry foundation models achieve strong single-image accuracy, they exhibit severe temporal inconsistency on continuous input, notably dominated by scale–shift drifting. Through targeted empirical analysis, we trace this instability to its root cause: fluctuations in latent feature statistics, whose mean and variance directly determine the predicted depth’s scale and shift. Building on this insight, we introduce Dynamic Feature Normalization (DyFN), a lightweight, causal recurrent module that dynamically and robustly modulates feature statistics to maintain stable geometry over time. We adapt powerful pretrained monocular geometry models for streaming by finetuning only DyFN, a mere 2% additional parameters, while keeping the backbone frozen, thereby achieving temporal consistency without compromising single-image accuracy. Extensive experiments across four benchmarks show that DyFN effectively eliminates temporal artifacts such as disjointed layering and positional jitter, and achieves state-of-the-art temporal stability, improving over prior streaming methods by up to 14% and even outperforming heavier non-causal video baselines. Project page: https://shawlyu.github.io/DyFN 1 Introduction 3D geometry estimation is fundamental to many real-world applications, such as robotics, autonomous driving, and augmented reality. Recently, Monocular Geometry Estimation (MGE) and Monocular Depth Estimation (MDE) 46 ; 47 ; 54 ; 55 ; 2 ; 36 ; 56 ; 33 has progressed rapidly with the rise of large-scale foundation models, significantly narrowing the gap between single-image prediction and sensor-based measurements. Models such as MoGe 46 further exhibit remarkable zero-shot generalization across diverse scenes by learning geometry-aware priors from massive image collections. Despite these advances, most MGE and MDE models are designed for image inference, limiting their applicability in dynamic environments where input naturally arrives as continuous video streams. When applied to continuous video streams, image-based MGE models exhibit pronounced temporal inconsistency: geometry predictions fluctuate across frames, causing distortions such as layering breaks and positional jitter in reconstructed scenes (Fig. 2 c). Existing methods mitigate this issue using temporal attention 22 ; 38 or recurrent memory modules 11 ; 45 to enforce inter-frame coherence. However, these solutions come with notable limitations: they typically require full-network finetuning on large-scale annotated video datasets, which is computationally expensive and data-intensive; and such finetuning often degrades the per-frame accuracy and zero-shot generalization of pretrained MGE models by overfitting the backbone to specific video domains. Figure 2 : Reconstruction comparison. We align the predicted depth to metric scale using an affine transformation. Per-frame Aligned involves calculating the scale and shift for each frame independently. Sequence-aligned involves calculating a single, consistent scale and shift for the entire sequence. The point clouds are then fused using ground truth poses.( \delta_{1} means \delta<1.25 ) In this paper, we argue that existing foundation MGE models are already well-equipped to be repurposed for streaming video depth estimation without retraining the entire network. We posit that temporal inconsistency does not primarily arise from per-frame perceptual errors, but from inherent scale ambiguity: without a mechanism to maintain consistent scale and shift over time, the model effectively predicts each frame in an independently drifting coordinate system. Notably, pretrained models such as MoGe 46 already encode strong geometric structure—when each frame is individually aligned with a simple per-frame scale and shift , the reconstructed 3D geometry becomes accurate and temporally coherent (Fig. 2 b). This implies that temporal inconsistency is largely driven by frame-to-frame scale–shift instability rather than deficient geometry. To further investigate this phenomenon, we conduct an empirical analysis of how global latent feature statistics influence predicted scale and shift (Sec. 3 ). Our results reveal that scale/shift variations are tightly coupled with the mean and variance of latent features extracted by the pretrained encoder (Fig. 3 ). This finding suggests that rather than retraining the entire model, temporal stability can be achieved by directly regulating these latent feature statistics of MGE models, specifically, their mean and variance. Motivated by this observation, we propose Dynamic Feature Normalization ( DyFN ), a lightweight, learnable module that predicts and dynamically modulates the mean and variance of latent features over time to enforce consistent depth across frames, without compromising the fidelity or generalization of pretrained MGE models. For online streaming depth estimation , DyFN incorporates a recurrent ConvGRU-based module that updates normalization parameters based on aggregated historical context. By finetuning only this DyFN , while keeping the pretrained encoder and decoder frozen, our method efficiently adapts existing MGE models to continuous inputs, achieving temporal coherence without sacrificing geometric accuracy. Extensive experiments across diverse benchmarks demonstrate that DyFN achieves state-of-the-art temporal stability in streaming scenarios (See Figure 5 and Table 1 ). It not only surpasses strong video-based methods, but also preserves the strong single-frame accuracy of pretrained MGE models. In summary, our main contributions are threefold: • We identify the principal source of temporal instability in pretrained monocular depth models: frame-to-frame scale–shift inconsistency arising from fluctuations in latent feature statistics (mean and variance), thereby establishing a direct connection between feature distribution and temporal stability. • We propose Dynamic Feature Normalization (DyFN) , a lightweight and general stabilization module that dynamically modulates latent feature statistics to maintain consistent scale and shift over time. • We show that finetuning only this small stabilizer while freezing the pretrained MGE backbone achieves state-of-the-art temporal stability across diverse video benchmarks without degrading single-frame accuracy or generalization, surpassing fully trained video-based models. 2 Related Work Relative Depth Estimation Relative depth estimation has demonstrated robust generalization across diverse domains by predicting depth up to an unknown scale and shift 3 ; 36 ; 26 ; 35 ; 13 ; 54 ; 55 . Foundational works like MiDaS 36 established this paradigm using multi-dataset training combined with scale-invariant losses. Subsequent research has increasingly integrated large-scale self-supervised pretraining 30 ; 19 ; 50 ; 51 ; 52 to enhance feature representation. Notably, DPT 35 successfully adapted transformers for dense prediction, a strategy further scaled by Depth Anything 54 ; 55 utilizing over 60 million unlabeled images to achieve superior zero-shot performance. More recently, diffusion-based methods such as Marigold 25 and GeoWizard 15 have leveraged generative priors 37 ; 34 to push the boundaries of detail recovery. Despite these advancements, these frame-centric approaches process images independently, inherently failing to maintain temporal consistency when applied to video streams. Metric Depth Estimation. To resolve scale-shift ambiguity, metric depth estimation aims to recover absolute depth from monocular inputs, a task that remains fundamentally ill-posed 57 ; 56 ; 21 ; 33 ; 32 ; 47 ; 18 ; 2 . Recent methods tackle this by incorporating strong geometric priors or optimizing for camera intrinsics. For instance, LeReS 57 leverages scene statistics to align predictions, while ZoeDepth 2 extends relative depth networks with adaptive metric bins to handle scene variability. Addressing the dependency on camera parameters, Metric3D 56 and its successor 21 propose zero-shot inference within a canonical camera space, whereas UniDepth 33 employs spherical parameterization