You See it, You Got it: Learning 3D Creation on Pose-Free Videos at Scale

Baorui Ma, Huachen Gao, Haoge Deng, Zhengxiong Luo, Tiejun Huang, Lulu Tang, Xinlong Wang Beijing Academy of Artificial Intelligence (BAAI) · 2024

Read You See it, You Got it: Learning 3D Creation on Pose-Free Videos at Scale by Baorui Ma, Huachen Gao, Haoge Deng, Zhengxiong Luo, Tiejun Huang, Lulu Tang, Xinlong Wang Beijing Academy of Artificial Intelligence (BAAI) (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06699.txt.

Introduction

Baorui Ma and six co-authors teach a model to create 3D scenes by watching ordinary online videos, no camera poses required. Most 3D generators lean on scarce hand-labeled 3D data or on 2D image models, and both cap how much the model can learn. Their 2024 system, See3D, is a diffusion model that makes multiple views of a scene from a visual prompt. A curation pipeline filters the flood of internet video into training material at scale. The bet is that seeing enough video teaches real 3D structure.

Pose-free learning removes the labeling bottleneck that has held the field back. Open-world creation, any scene a user imagines, is the target. The authors scale both data and training to match. Their slogan sums up the method: you see it, you got it. The paper is a scale play in a field short on data.

Catalog record

CREATORBaorui Ma, Huachen Gao, Haoge Deng, Zhengxiong Luo, Tiejun Huang, Lulu Tang, Xinlong Wang Beijing Academy of Artificial Intelligence (BAAI)
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE76 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-06699

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

You See it, You Got it: Learning 3D Creation on Pose-Free Videos at Scale Baorui Ma, Huachen Gao, Haoge Deng, Zhengxiong Luo, Tiejun Huang, Lulu Tang, Xinlong Wang Beijing Academy of Artificial Intelligence (BAAI) Abstract Recent 3D generation models typically rely on limited-scale 3D ‘gold-labels’ or 2D diffusion priors for 3D content creation. However, their performance is upper-bounded by constrained 3D priors due to the lack of scalable learning paradigms. In this work, we present See3D , a visual-conditional multi-view diffusion model trained on large-scale Internet videos for open-world 3D creation. The model aims to Get 3D knowledge by solely See ing the visual contents from the vast and rapidly growing video data — You See it, You Got it. To achieve this, we first scale up the training data using a proposed data curation pipeline that automatically filters out multi-view inconsistencies and insufficient observations from source videos. This results in a high-quality, richly diverse, large-scale dataset of multi-view images, termed WebVi3D, containing 320M frames from 16M video clips. Nevertheless, learning generic 3D priors from videos without explicit 3D geometry or camera pose annotations is nontrivial, and annotating poses for web-scale videos is prohibitively expensive. To eliminate the need for pose conditions, we introduce an innovative visual-condition - a purely 2D-inductive visual signal generated by adding time-dependent noise to the masked video data. Finally, we introduce a novel visual-conditional 3D generation framework by integrating See3D into a warping-based pipeline for high-fidelity 3D generation. Our numerical and visual comparisons on single and sparse reconstruction benchmarks show that See3D , trained on cost-effective and scalable video data, achieves notable zero-shot and open-world generation capabilities, markedly outperforming models trained on costly and constrained 3D datasets. Additionally, our model naturally supports other image-conditioned 3D creation tasks, such as 3D editing, without further fine-tuning. Please refer to our project page at: https://vision.baai.ac.cn/see3d . 1 Introduction Recent advances in 3D generation are essential for fields like virtual reality, entertainment, and simulation, offering the potential not only to recreate intricate real-world structures but also to expand human imagination. Nevertheless, developing these models is constrained by the scarcity and high costs of accessible 3D datasets. Despite recent industry efforts Tochilkin et al. 2024 ; Zhang et al. 2024b ; Yang et al. 2024 create extensive proprietary 3D assets, these initiatives come with substantial financial and operational burdens. Currently, building such a large-scale 3D dataset for academia remains prohibitively expensive. This motivates us to pursue a scalable, accessible, and affordable data source that can compete with advanced closed-source solutions, thereby enabling the broader research community to train high-performance 3D generation models. Human perception of the 3D world does not rely on specific 3D representation (e.g., point clouds Fan et al. 2017 , voxel grids Kar et al. 2017 , meshes Wang et al. 2018 , or neural fields Mildenhall et al. 2021 ) or precise camera conditions. Instead, our 3D awareness is shaped by multi-view observations accumulated throughout our lives. This raises the question: Can models similarly learn universal 3D priors from large collections of multi-view images? Fortunately, Internet videos offer a rich source of multi-view images, captured from various locations with diverse sensors and complex camera trajectories, providing a scalable, accessible, and cost-effective data source. Thus, how can we effectively learn 3D knowledge from Internet videos? The core challenges in achieving this goal are twofold: 1) filtering relevant, 3D-aware video data from raw sources, specifically static scenes with varied camera viewpoints that provide sufficient multi-view observations; and 2) learning generic 3D priors from videos lacking explicit 3D geometry and camera pose annotations (i.e. pose-free videos). This work carefully addresses these challenges and introduces a pose-free, visual-conditional multi-view diffusion (MVD) model, See3D , for open-world 3D creation. Specifically, we establish a novel video data curation pipeline that automatically filters out data with dynamic content or restricted camera viewpoints from source videos. The resulting dataset, termed WebVi3D, comprises 15.99M video clips from 25.48M source videos, totaling 4.41 years in duration—orders of magnitude larger than previous 3D datasets, such as DLV3D (0.01M) Ling et al. 2024 , RealEstate10K (0.08M) Zhou et al. 2018 , MVImgNet (0.22M) Yu et al. 2023 and Objaverse (0.8M) Deitke et al. 2023 . MVD models have recently gained widespread attention due to their advantages of integrating the generative capabilities of 2D diffusion models while maintaining consistency across multiple views Shi et al. 2023b ; Liu et al. 2023c ; Liu et al. 2024a ; Sun et al. 2024 ; Zhao et al. 2024b . Typically, these models rely on precise camera poses Shi et al. 2023b ; Gao et al. 2024 ; Hong et al. 2023 ; Liu et al. 2023b ; Müller et al. 2024 ; Bahmani et al. 2024 ; Xu et al. 2024a ; Xu et al. 2024b ; He et al. 2024 ; Zhang et al. 2024a ; Li et al. 2024a or warped images rendered according to camera position Yu et al. 2024c ; Tung et al. 2024 as conditional inputs to guide 3D-consistent view generation. We refer to these conditions, derived from pose or 3D annotations, as 3D-inductive conditions. However, annotating web-scale videos is prohibitively costly, or even intractable in some cases, posing significant challenges for scaling. To address this, we propose a novel, pose-free visual-condition derived from pixel-space hints within videos. It is a purely 2D-inductive visual signal, created by introducing time-dependent noise to masked input videos, free from any 3D-inductive bias. This enables training MVD model at scale, without requiring pose annotations. Intuitively, the proposed visual-condition can generalize effectively to tasks that rely on pixel-space hints distinct from those in videos, such as warping-based 3D generation Chung et al. 2023 ; Shriram et al. 2024 and mask-based 3D editing Chen et al. 2024 , without requiring additional training, see . For instance, in warping-based 3D generation, pixels from a reference image are rearranged through warping operations, creating a specific visual-condition to indicate camera movement. However, these warped images often exhibit artifacts or distortions, causing a significant domain gap compared to video frames. Whereas, our visual-condition functions as a generic one, capable of accommodating such unnatural images. To further harness the potential of See3D , we introduce an innovative visual-conditional 3D generation framework utilizing a warping-based pipeline. This framework first constructs the visual-condition using See3D , then iteratively refines the geometry of novel views to build comprehensive scene observations. Finally, the generated images are used for Gaussian Splatting reconstruction Kerbl et al. 2023 ; Huang et al. 2024 , which can be rendered from arbitrary viewpoints or converted into meshes through post-processing Lorensen and Cline 1998 . In summary, our key contributions are as follows. • We present See3D , a scalable visual-conditional MVD model for open-world 3D creation, which can be trained on web-scale video collections without pose annotations. • We curate WebVi3D, a multi-view images dataset containing static scenes with sufficient multi-view observations, and establish an automated pipeline for video data curation to train the MVD model. • We introduce a novel warping-based 3D generation framework with See3D , which supports long-sequence generation with complex camera trajectories. • We achieve state-of-the-art results in single and sparse views reconstruction, demonstrating remarkable zero-shot and open-world generation capability, offering a

Read the complete text