How Much 3D Do Video Foundation Models Encode?

Zixuan Huang, Xiang Li, Zhaoyang Lv, James M. Rehg · 2025

Read How Much 3D Do Video Foundation Models Encode? by Zixuan Huang, Xiang Li, Zhaoyang Lv, James M. Rehg (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-19949.txt.

Introduction

Zixuan Huang, Xiang Li, Zhaoyang Lv, and James Rehg probe how much three-dimensional understanding hides inside video models. A video is just a flat projection of a three-dimensional world, frame after frame, so a model trained on enough video might internalize depth, shape, and layout without ever being taught them. The team built the first model-agnostic framework for measuring that awareness, reading several three-dimensional properties out of model features with shallow probes.

Benchmarking major video foundation models across multiple axes produced a headline surprise: state-of-the-art video generation models understand objects and scenes three-dimensionally with striking strength, despite no three-dimensional training data. On some measures they surpassed large expert models built specifically for three-dimensional tasks. The authors offer the benchmark as guidance for building scalable 3D models from video.

Catalog record

CREATORZixuan Huang, Xiang Li, Zhaoyang Lv, James M. Rehg
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-19949

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

How Much 3D Do Video Foundation Models Encode? Zixuan Huang, Xiang Li, Zhaoyang Lv, James M. Rehg Abstract Videos are continuous 2D projections of 3D worlds. After training on large video data, will global 3D understanding naturally emerge? We study this by quantifying the 3D understanding of existing Video Foundation Models (VidFMs) pretrained on vast video data. We propose the first model-agnostic framework that measures the 3D awareness of various VidFMs by estimating multiple 3D properties from their features via shallow read-outs. Our study presents meaningful findings regarding the 3D awareness of VidFMs on multiple axes. In particular, we show that state-of-the-art video generation models exhibit a strong understanding of 3D objects and scenes, despite not being trained on any 3D data. Such understanding can even surpass that of large expert models specifically trained for 3D tasks. Our findings, together with the 3D benchmarking of major VidFMs, provide valuable observations for building scalable 3D models. https://vidfm-3d-probe.github.io/ Figure 1 : We study the emergence of 3D in video foundation models by probing their features with 3D reconstruction tasks. Our study reveals state-of-the-art video generators develop strong 3D understanding even compared to 3D experts, despite only trained on 2D data. * * footnotetext: Both authors contributed equally to this work. 1 Introduction Recovering 3D structure from 2D visual observations is a long-standing research problem in computer vision, with broad applications in AR/VR and embodied AI. Despite significant progress, the availability of high-quality 3D data at scale remains the bottleneck for current data-driven approaches. This fundamentally limits the scaling of 3D foundation models and makes it questionable whether we can learn truly generalizable models primarily from 3D data. Compared to native 3D assets, videos are much easier to acquire at scale, with multiple large curated datasets already available Bain et al. 2021 ; Abu-El-Haija et al. 2016 ; Miech et al. 2019 ; Chen et al. 2024a . The diversity and complexity of video data, with the fact that videos are 2D projections of 3D worlds, lead to a promising pathway for scalable 3D learning. Recent works study how to utilize video models for 3D, either by adding 3D control He et al. 2024 ; He et al. 2025 ; Bahmani et al. 2025 ; Xu et al. 2024 ; Sun et al. 2024 or by producing 3D caches/estimations Team et al. 2025 ; Lu et al. 2025 ; Zhang et al. 2025a ; Huang et al. 2025a ; Mai et al. 2025 ; Jiang et al. 2025 ; Yu et al. 2024 ; Gu et al. 2025 ; Ren et al. 2025 ; Huang et al. 2025b ; Wu et al. 2025 ; Liang et al. 2024 ; Zhang et al. 2025b along with the original frame synthesis target. These works suggest that video priors are useful for 3D, but 3D-inconsistency artifacts, the requirement of 3D fine-tuning, and task-specific engineering leave it unclear whether video data alone can induce strong 3D awareness in a general-purpose setting. These confounds motivate a direct, model-agnostic evaluation. In this paper, we present the first model-agnostic framework to probe the 3D awareness of video foundation models (VidFMs) pretrained on large-scale video data. We ask whether VidFMs develop internal representations of 3D structure and ego-motion and, if so, how strong and practically useful these representations are. We operationalize this question along four axes: 1) Extent : how does the 3D awareness of VidFMs compare to that of image models or specialized 3D models? 2) Factor : Which factors impact 3D awareness? Here, we focus on the effects of temporal reasoning, 3D finetuning and model scaling. 3) Localization : In which network layers, and at which timesteps in diffusion models, is this 3D information most concentrated? 4) Implication : Under limited resources (3D data and compute), can VidFM features be practically useful for 3D reconstruction tasks? We posit that if a video model understands 3D worlds, it should be feasible to extract accurate 3D properties using shallow readout modules in a feedforward manner, without any post-optimization or fine-tuning of the base model. Unlike prior works that evaluate image models using depth and cross-view consistency El Banani et al. 2024 , or per-scene optimization with off-the-shelf initialization Chen et al. 2025 , our proposed shallow feedforward readouts that estimate different 3D attributes from VidFMs’ feature space are a more direct probe of globally consistent 3D properties from pretrained video models. Specifically, we extract frozen spatialtemporal features from VidFMs, and design a probe model that predicts 3D points, camera poses and depth maps from these features. The probe model is a shallow VGGT Wang et al. 2025 -like transformer, consisting of four alternating-attention layers and three read-out heads: two dense prediction heads for 3D points and depth maps, and one camera head. We train the probe model on top of various video features, including features extracted from self-supervised video models and video generation models of different performance and sizes. We measure the performance of point, camera and depth reconstruction as indicators of 3D awareness. Our study leads to the following novel findings: • Extent : Frontier video generation models exhibit great understanding of 3D objects and scenes, and can be close to or better than models trained with 3D data; • Factor #1: Temporal reasoning is critical to the formation of global 3D understanding; • Factor #2: Finetuning video generation models with 3D objectives improves 3D awareness on in-domain data, but may hurt generalization beyond data domains; • Factor #3: Model scaling leads to mixed impact on 3D awareness, with WAN2.1-14B performing significantly better than WAN2.1-1.3B, while CogVideoX-5B is slightly worse than CogVideoX-2B. • Localization: The best layer and timestep to extract 3D-aware features are surprisingly consistent across all tested video diffusion models: mid-layer features with early-but-not-first timesteps lead to the highest 3D awareness. • Implication: We implement and train a VGGT model using frozen VidFM (WAN2.1-14B) features. On CO3Dv2 and DL3DV, it consistently outperforms the standard DINO-based VGGT, indicating VidFM features may be better suited for 3D reconstruction under limited 3D data. In summary, we conduct the first systematic model-agnostic evaluation on the 3D awareness of VidFMs and conclude with meaningful findings across multiple axes that the prior work has not surfaced well. Our findings are based on a comprehensive benchmark that compares 3D awareness of various video models, which can benefit the development of VidFMs by enabling the evaluation of their emerging 3D properties in a general-purpose way. 2 Related Works Figure 2 : Overview of the Probe. We extract video features using various video foundation models and keep the features frozen. We sample four frames from the original video clip and fetch the corresponding feature maps from the video features. We train the probe by taking these spatial features as input, and task the probe to estimate point maps, depth maps and camera poses. Our probe model consists of a shallow transformer and three readout heads. We measure the estimation errors as the main indicators of 3D awareness. Video foundation models (VidFMs) are deep models trained on massive video data that achieve strong performance across various downstream tasks. Early works in the field employ self-supervised learning Bardes et al. 2023 ; Tong et al. 2022 ; Wang et

Read the complete text