Syn4D: A Multiview Synthetic 4D Dataset

Zeren Jiang *, Yushi Lan *, Yihang Luo, Yufan Deng, Zihang Lai, Edgar Sucar, Christian Rupprecht, Iro Laina, Diane Larlus, Chuanxia Zheng, Andrea Vedaldi · 2026

Read Syn4D: A Multiview Synthetic 4D Dataset by Zeren Jiang *, Yushi Lan *, Yihang Luo, Yufan Deng, Zihang Lai, Edgar Sucar, Christian Rupprecht, Iro Laina, Diane Larlus, Chuanxia Zheng, Andrea Vedaldi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-05207.txt.

Introduction

Zeren Jiang and ten co-authors release Syn4D, a synthetic multiview dataset of moving three-dimensional scenes for computer vision research. Reconstructing and tracking dynamic scenes from a single video remains an open challenge, held back by scarce datasets with dense, accurate geometric labels. Syn4D supplies ground-truth camera motion, depth maps, dense tracking, and parametric human pose for its scenes.

Any pixel can be projected into three dimensions at any time and from any camera, a key feature for training and checking models. The authors evaluate the dataset across four tasks, including scene reconstruction, point tracking, camera retargeting, and pose estimation. Results highlight its potential for dynamic scene understanding and space-time modelling.

Catalog record

CREATORZeren Jiang *, Yushi Lan *, Yihang Luo, Yufan Deng, Zihang Lai, Edgar Sucar, Christian Rupprecht, Iro Laina, Diane Larlus, Chuanxia Zheng, Andrea Vedaldi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.05207

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Syn4D: A Multiview Synthetic 4D Dataset Zeren Jiang *, Yushi Lan *, Yihang Luo, Yufan Deng, Zihang Lai, Edgar Sucar, Christian Rupprecht, Iro Laina, Diane Larlus, Chuanxia Zheng, Andrea Vedaldi Abstract Dense 3D reconstruction and tracking of dynamic scenes from monocular video remains an important open challenge in computer vision. Progress in this area has been constrained by the scarcity of high-quality datasets with dense, complete, and accurate geometric annotations. To address this limitation, we introduce Syn4D, a multiview synthetic dataset of dynamic scenes that includes ground-truth camera motion, depth maps, dense tracking, and parametric human pose annotations. A key feature of Syn4D is the ability to unproject any pixel into 3D to any time and to any camera. We conduct extensive evaluations across multiple downstream tasks to demonstrate the utility and effectiveness of the proposed dataset, including 4D scene reconstruction, 3D point tracking, geometry-aware camera retargeting, and human pose estimation. The experimental results highlight Syn4D’s potential to facilitate research in dynamic scene understanding and spatiotemporal modeling. * * footnotetext: Equal contribution. Figure 1 : Syn4D is a large-scale synthetic dataset designed for a set of 4D tasks, including camera pose estimation, depth estimation, dynamic 3D scene reconstruction, 2D/3D tracking, human pose estimation, and novel-view synthesis. On the left, we visualize two shots with two synchronized frames for each shot. On the right, we unproject depth maps and visualize the camera trajectories, geometry, as well as 3D tracking results for both shots. 1 Introduction For some time now, most computer vision problems have been reduced to training large, general-purpose neural networks, such as transformers, with research focusing on improved learning formulations and data engineering. 3D reconstruction has long resisted this trend, but recent works have shown that feed-forward networks [ 111 , 108 , 113 , 58 , 64 ] can reconstruct 3D scenes, from one, a few, or hundreds of views, with performance comparable to traditional optimization-based pipelines like structure-from-motion (SfM). Furthermore, learning-based approaches can dramatically outperform geometry-based counterparts on more ambiguous tasks, such as reconstructing 3D motion from videos (4D reconstruction), which have eluded traditional approaches for half a century. These problems, in fact, require the use of statistical priors [ 101 ] , which are better captured by learning-based approaches. So why has machine learning taken so long to become competitive with methods like SfM? We argue that a main reason has been the lack of large-scale datasets with reliable geometric annotations. Works like the aforementioned [ 111 , 108 , 113 , 58 , 64 ] dedicate significant effort to collecting, curating, and annotating synthetic and real data. However, these datasets and annotations have largely remained private, and they contain only static scenes, which are insufficient for learning 4D reconstruction. This is a significant bottleneck to further progress. In this paper, we address this gap and contribute Syn4D , a fully-synthetic dataset designed to advance learning-based 4D reconstruction, tracking, novel-view synthesis, and related tasks ( Fig. 1 ). We mitigate the limitations of existing datasets in scale, quality, annotation coverage, availability, and licensing. Synthetic data has proven highly effective in 2D and 3D geometry tasks such as 3D reconstruction [ 123 ] and generation [ 24 ] , human pose estimation [ 11 ] , and 2D/3D tracking [ 36 , 135 , 53 ] . Conversely, real datasets such as Stereo4D [ 48 ] , while valuable, typically provide only sparse and noisy geometric annotations. We construct Syn4D procedurally using Unreal Engine, leveraging the large catalog of high-quality 3D environments available through the Unreal Fab store. For dynamic content, we extract 1{,}674 animated 3D assets from Objaverse(-XL) [ 24 , 23 ] and 585 simulated dynamic humans from Bedlam2 [ 100 ] . All this data is licensed for AI training. We then place these assets at carefully curated locations within each 3D environment, and design diverse camera trajectories, ranging from simple motions to complex movements, to encourage better generalization of models trained on this data. This combination of professionally designed 3D environments and diverse animated 3D assets — humans, animals, humanoid robots, and other characters — enables us to produce high-quality videos of dynamic scenes, as shown in Fig. 2 . Our Syn4D also comes with comprehensive 3D annotations, including ground-truth camera motion, depth maps, point maps, dense 2D and 3D tracking, and parametric human pose annotations, all in a multiview setting. This makes it one of the few datasets that can support a wide variety of multi-view 3D and 4D tasks, and novel view synthesis (NVS). Some datasets like Kubric [ 36 ] , PointOdyssey [ 135 ] , and Bedlam2 [ 100 ] contain dynamic scenes, but they lack multiview information and dense tracking annotations, which are crucial for learning 4D reconstruction. In particular, this is the first dataset to provide dense and complete 3D tracking annotations in a multiview setting, enabling the recovery of the 3D position of any point in any image at any time (and thus its 2D projection in any other frame and camera), offering an advantage over datasets like PointOdyssey [ 135 ] . Because storing this information explicitly is infeasible, we also develop an efficient representation of these dense tracks based on pixel-aligned barycentric maps paired with the corresponding mesh sequences. This design enables efficient querying of dense tracks within data loaders for training or evaluation. Overall, Syn4D comprises 4.7K multiview video clips , totaling 1.4M frames , with dense geometry annotations . This is significantly larger than existing 3D datasets that offer dense geometry and motion annotations ( Table 1 ). Through extensive experiments, we show that training on Syn4D significantly improves the performance of off-the-shelf models in geometry-aware NVS, 4D reconstruction and tracking, and human pose estimation. Because our Syn4D contains multiview videos with corresponding ground-truth geometry, we first introduce a new task, benchmark, and evaluation metrics for geometry-aware dynamic NVS, considering both the visual quality and geometric consistency of the generated views. Another unique property of Syn4D is its dense tracking annotations. Thus, we also propose new benchmarks and evaluation metrics for 3D tracking. Finally, we evaluate the performance of state-of-the-art 4D reconstruction ( e.g . , camera pose and point map recovery) and human pose estimation models trained on Syn4D . To summarize our contributions, Syn4D is the first publicly available dataset containing multiview videos of dynamic scenes with dense 3D tracking annotations, along with camera images, depth maps, and human pose labels. We demonstrate that this data improves the performance of state-of-the-art 2D/3D tracking and 4D reconstruction models. We further show that the dataset enables training new models, including a geometry-aware multiview diffusion model that can generate novel-view video sequences together with consistent geometry. 2 Related Work Dataset Camera Track HP SF OF IS Capt. Engine # Dyn. Obj . # Clips # Frames Sintel [ 13 ] Stereo \times \times \times ✓ ✓ \times Blender Few others 35 1.6K Spring [ 73 ] Stereo \times \times ✓ ✓ ✓ \times Blender Few others 47 6K FlyingThings3D [ 71 ] Stereo \times \times ✓ ✓ ✓ \times Custom Many Rigid 2.2K 35K Kubric (MOVi-F) [ 36 ] Mono Dense \times ✓ ✓ ✓ \times Blender Many Rigid 4K 96K Falling Things

Read the complete text