Generalizable Articulated Object Reconstruction from Casually Captured RGBD Videos

Weikun Peng, Jun Lv, Cewu Lu, Manolis Savva · 2025

Read Generalizable Articulated Object Reconstruction from Casually Captured RGBD Videos by Weikun Peng, Jun Lv, Cewu Lu, Manolis Savva (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-08334.txt.

Introduction

Weikun Peng, Jun Lv, Cewu Lu, and Manolis Savva present a way to build digital twins of jointed objects from casual phone-style video, in this 2025 paper. The head gives the system its name, iTACO. Jointed objects, like laptops and cabinets, fill daily life, and robots need models of how their parts move. Older methods demand careful, staged captures.

The system works from a hand-held depth video shot while a person uses the object, wobble, hidden parts, and all. It first makes a coarse guess of the joints, then refines part boundaries and motion in stages. To test it, the team built a set of 784 videos covering 284 objects in eleven classes, twenty times larger than prior sets. It beat rival video methods on both synthetic and real clips, which the authors say opens the door to scanning objects at scale.

Catalog record

CREATORWeikun Peng, Jun Lv, Cewu Lu, Manolis Savva
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-08334

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Generalizable Articulated Object Reconstruction from Casually Captured RGBD Videos Weikun Peng, Jun Lv, Cewu Lu, Manolis Savva Abstract Articulated objects are prevalent in daily life. Understanding their kinematic structure and reconstructing them have numerous applications in embodied AI and robotics. However, current methods require carefully captured data for training or inference, preventing practical, scalable, and generalizable reconstruction of articulated objects. We focus on reconstruction of an articulated object from a casually captured RGBD video shot with a hand-held camera. A casually captured video of an interaction with an articulated object is easy to acquire at scale using smartphones. However, this setting is quite challenging, as the object and camera move simultaneously and there are significant occlusions as the person interacts with the object. To tackle these challenges, we introduce a coarse-to-fine framework that infers joint parameters and segments movable parts of the object from a dynamic RGBD video. To evaluate our method under this new setting, we build a 20 × \times larger synthetic dataset of 784 videos containing 284 objects across 11 categories. We compare our approach with existing methods that also take video as input. Experiments show that our method can reconstruct synthetic and real articulated objects across different categories from dynamic RGBD videos, outperforming existing methods significantly. 1 Introduction Articulated objects are prevalent in daily life. Building digital representations of such objects and understanding their kinematic structures has many applications in computer animation, embodied AI, and robotics. For example, automatically building interactable digital twins of articulated objects can benefit robotics research, since researchers can train and test manipulation policies more conveniently and efficiently in simulation prior to real-world deployment. However, to the best of our knowledge, current approaches to articulated object reconstruction impose many constraints on the input data, preventing practical and efficient deployment for acquisition of articulated objects in realistic settings. Thus, researchers still largely rely on manual annotation of articulation joint parameters [ 39 ] . Recent approaches to reconstructing articulated objects can be categorized into three families. The first uses a deformation model for implicit functions to represent the dynamics of articulated objects [ 26 , 40 , 46 , 34 ] . However, the resulting implicit representation is not easy to use in simulation as we first need to reconstruct the articulated object explicitly and find rigid transformations of the different object parts. Thus, this is not a practical path for physically accurate animation or interaction. The second family reconstructs articulated objects by estimating joint parameters explicitly from two different states of the object [ 8 , 16 , 47 , 19 , 21 , 27 ] . However, this type of approach requires aligning observations of two states to the same coordinate frame. Prediction accuracy can drop drastically due to slight errors or perturbations in the coordinate frame alignment of observations at each state. The third family of approaches generates articulated objects leveraging large language models and vision foundation models to help predict joint parameters [ 22 , 13 ] . At present, these approaches either rely on an external library for object retrieval or require additional data for fine-tuning. Retrieving objects from a library is not easily scalable, while collecting and annotating data for fine-tuning is labor-intensive. Furthermore, these approaches do not easily generalize well to real settings after fine-tuning on limited data. By analyzing the issues of recent approaches, we summarize three desirable features for a practical articulated object reconstruction pipeline: 1) minimal constraints on input; 2) explicit joint parameter and geometry representation for easy integration into simulators; and 3) generalization across diverse categories of articulated objects. To address these desiderata, we propose to reconstruct articulated objects from casually captured RGBD videos of interaction with the object (i.e. a video shot from a hand-held RGBD camera while the same person recording interacts with the object). Casually captured videos, or dynamic videos, are easy to capture at scale and contain the necessary motion information to infer joint parameters. Compared to the problem setting in prior work, the input video data is far less constrained and easier to capture, the resulting explicit representation of articulated objects is easier to use for simulation, and the overall pipeline is scalable to real-world settings. Figure 1: We propose a coarse-to-fine framework that can reconstruct an articulated object and relevant articulation parameters from a casually captured RGBD video. Our pipeline first predicts coarse joint parameters and movable part segmentation. The initial estimates are then refined with gradient-based optimization. However, this kind of input also brings more challenges to this problem. Since both the camera and the scene are dynamic, and the person occludes the object as they are manipulating it, decoupling camera motion and object part motion is challenging. Consequently, accurate estimation of joint parameters and movable part segmentation is difficult. To address these challenges, we develop a coarse-to-fine framework to reconstruct articulated objects from dynamic RGBD videos, illustrated in Fig. 1 . Our pipeline first estimates joint parameters and a movable part segmentation via image feature matching. Then, a gradient-based optimization framework refines these initial estimates against a surface point cloud representation of the object acquired through 3D reconstruction. Our pipeline explicitly parameterizes the articulation joint parameters and other relevant parameters, satisfying the second feature. Moreover, to aid generalization and satisfy the third desideratum, our pipeline neither relies on an external library nor requires additional data for fine-tuning. Instead, it only uses pretrained models. Since the problem setting and task setup we described has not been previously addressed, we build a new dataset for evaluation. We select 284 synthetic objects from 11 categories in the PartNet-Mobility dataset [ 25 ] to generate dynamic videos for input and evaluate the performance of our method. Compared to datasets used in prior works such as PARIS [ 16 ] and ArtGS [ 19 ] , our dataset contains 20 \times more objects and thus provides a more robust evaluation of the performance of different methods for reconstructing articulated objects from dynamic videos. In summary, we make the following contributions: 1. We address the task of articulated object reconstruction from a casually captured RGBD video, a new problem setting that is more practical than prior work. 2. To tackle this new problem, we develop a coarse-to-fine framework that reconstructs articulated objects explicitly and does not rely on an external articulated object library or additional data for fine-tuning. 3. To evaluate our method robustly, we build a new synthetic dataset with 20 \times more objects than prior datasets. Experiments show that our method outperforms state of the art methods on reconstruction of both synthetic and real articulated objects across different categories. 2 Related work Our work lies at the intersection of articulated object reconstruction and dynamic scene understanding. We review recent work in both fields to provide a comprehensive background of our work. 2.1 Articulated object reconstruction Articulated object reconstruction is a long-standing task in computer vision and robotics. One line of research focuses on the visual fidelity of the dynamics of the articulated objects [ 26 , 40 , 46 , 34 ] . They usually apply implicit representations and deformation fields to model the articulated objects. This type of representation is not suitable for physically accurate animation and

Read the complete text