Mesh4D: 4D Mesh Reconstruction and Tracking from Monocular Video

Zeren Jiang, Chuanxia Zheng, Iro Laina, Diane Larlus, Andrea Vedaldi · 2026

Read Mesh4D: 4D Mesh Reconstruction and Tracking from Monocular Video by Zeren Jiang, Chuanxia Zheng, Iro Laina, Diane Larlus, Andrea Vedaldi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-05251.txt.

Introduction

Zeren Jiang and 4 co-authors reconstruct a moving object's full three-dimensional shape from a single ordinary video in a 2026 paper. Their model, Mesh4D, takes monocular video of a dynamic object and rebuilds its complete shape and motion as a deformation field, the mapping that bends a template shape through every pose. One camera view hides the object's far side, so the reconstruction must infer what it cannot see.

A compact latent space encodes the whole animation in one pass. During training, the skeletal structure of training objects guides the autoencoder, but no skeleton is needed at use time. Single-video 4D reconstruction would help animators, biologists, and sports analysts turn casual footage into measurable moving models.

Catalog record

CREATORZeren Jiang, Chuanxia Zheng, Iro Laina, Diane Larlus, Andrea Vedaldi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-05251

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Mesh4D: 4D Mesh Reconstruction and Tracking from Monocular Video Zeren Jiang, Chuanxia Zheng, Iro Laina, Diane Larlus, Andrea Vedaldi Abstract We propose Mesh4D, a feed-forward model for monocular 4D mesh reconstruction. Given a monocular video of a dynamic object, our model reconstructs the object’s complete 3D shape and motion, represented as a deformation field. Our key contribution is a compact latent space that encodes the entire animation sequence in a single pass. This latent space is learned by an autoencoder that, during training, is guided by the skeletal structure of the training objects, providing strong priors on plausible deformations. Crucially, skeletal information is not required at inference time. The encoder employs spatio-temporal attention, yielding a more stable representation of the object’s overall deformation. Building on this representation, we train a latent diffusion model that, conditioned on the input video and the mesh reconstructed from the first frame, predicts the full animation in one shot. We evaluate Mesh4D on reconstruction and novel view synthesis benchmarks, outperforming prior methods in recovering accurate 3D shape and deformation. Figure 1 : Illustration of Mesh4D . Given a monocular RGB video as input, Mesh4D generates a complete animated 3D mesh and its deformation. Each 4D reconstruction is shown at several time steps, the top layer displaying normals and the bottom one textured meshes. 1 Introduction In monocular 4D mesh reconstruction , we are interested in reconstructing the complete 3D shape and motion of dynamic objects from monocular RGB videos. Solving this problem has many applications in computer vision, graphics, and robotics. Automating this pipeline is particularly appealing, as manual 3D modeling and animation are costly, time-consuming, and require domain expertise. However, this is also very challenging because monocular videos only show parts of the objects. Since the goal is to recover the objects in full, one must complete their 3D shape and track their deformation throughout the video. Often Zollhöfer et al. 2014 ; Newcombe et al. 2015 ; Innmann et al. 2016 this problem has been tackled using analysis by synthesis, where an animated 3D mesh is optimized iteratively to fit the input RGB(D) frames. However, these methods can only reconstruct the visible parts of the object and often produce incomplete or noisy results due to occlusions and sensor noise. Recent feed-forward methods Sucar et al. 2025 ; Feng et al. 2025 ; Jin et al. 2025 have emerged as promising alternatives. They learn to reconstruct dynamic 3D geometry from RGB videos in a single pass. Even so, these methods focus on reconstructing the visible geometry and only track dense correspondences across pairs of frames, rather than across the whole sequence. As a result, they often fail to capture the entire 4D structure of animated objects. Reconstructing and tracking the parts of the geometry that are not visible in the video requires strong 3D and physical priors that can only be learned from data. This calls for latent 3D generative models, which have been shown to capture rich priors for 3D shapes Team 2025a ; Xiang et al. 2025 ; Ke et al. 2024 , at least in the static case. In this paper, we thus ask whether such generative models can be extended to solve 4D reconstruction , implicitly capturing cues such as symmetry and smoothness that can help recover the 3D mesh and its motion beyond what is directly visible in the video. Recent contributions such as GVFD Zhang et al. 2025a have shown that this is a promising direction by introducing versions of 3D generators Xiang et al. 2025 ; Team 2025a that can handle dynamics. However, they focus on generating good-looking images of the object using a 3D Gaussian Splatting (3D-GS) representation Kerbl et al. 2023 , and are less concerned with recovering accurate 3D shape and motion. In this work, we introduce Mesh4D, a feed-forward model that learns to reconstruct accurate and complete dynamic 3D meshes from a monocular RGB video. We represent the object’s shape as a 3D mesh extracted from the first frame of the input video, and then generate a deformation field that displaces the surface of the mesh to represent the deformation of the object over time. This representation is intuitive as it factorizes the 3D shape and motion. By comparison, others Wang et al. 2025c ; Jiang et al. 2025 ; Yenphraphai et al. 2025 ; Ren et al. 2024 have approached 4D reconstruction by outputting independent 3D reconstructions of each frame and thus do not explicitly model the object’s motion. Explicitly modeling motion is often crucial, for example, to texture the 3D object consistently over time. We argue that, in order to accurately recover the object’s dynamics, they should be modeled holistically, from the beginning to the end of the video. To make this feasible, we introduce a new Variational Auto-Encoder (VAE) to encode the mesh deformation in a compact latent space. Prior work like Motion2VecSet Cao et al. 2024 has proposed analogous encodings, but only for two frames at once. In contrast, we show that a latent space representing the motion of the object throughout the whole sequence performs better. To this end, we build a transformer Vaswani et al. 2017 encoder that uses spatial, temporal, and global attention modules Geyer et al. 2024 ; Wang et al. 2025a in each block, capturing long-term correlations among points on the object. We also argue for using motion-related information to guide VAE training, which is a widely adopted strategy for static 3D latent spaces Li et al. 2025b ; Xiang et al. 2025 ; Team 2025b . In particular, we propose using the object’s skeleton , as this provides strong cues on the space of possible deformations of the object, explicitly coupling different physical points. As shown in Fig. 2 , we leverage both skinning weights and bone information as additional inputs to the VAE transformer, which significantly improves the quality of the learned latent space. Importantly, these are not needed at inference time. Another challenge in monocular 4D mesh reconstruction is the lack of suitable 4D datasets for training and benchmarking. Following recent works Ke et al. 2024 ; Jiang et al. 2025 ; Lu et al. 2025 ; Xu et al. 2025a , we do not train our deformation model from scratch but start from pretrained static 3D generators Team 2025a . These are trained on large datasets and help achieve robustness and generalization to a wide variety of object types. However, we still need a 4D dataset with annotations for the motion of the 3D points (i.e., 3D tracks) for training and evaluation. To this end, we build a synthetic dataset by filtering 3D animated assets from Objaverse Deitke et al. 2023 , which provides high-quality dynamic 3D meshes with skeletons. We further extract dense 3D point correspondences across all frames. For evaluation, we propose a new benchmark focusing on the quality of 3D shape and motion reconstruction rather than just visual quality. To do so, we collect 50 animated 3D assets with significant object motion, high-quality textures, and no overlap with our training data. To summarize, our contributions are as follows. First, we propose a new framework, Mesh4D, for monocular 4D mesh reconstruction. This model reconstructs the 3D shape and

Read the complete text