ArticulatedGS: Self-supervised Digital Twin Modeling of Articulated Objects using 3D Gaussian Splatting

Junfu Guo, Yu Xin, Gaoyi Liu, Kai Xu, Ligang Liu, Ruizhen Hu · 2025

Read ArticulatedGS: Self-supervised Digital Twin Modeling of Articulated Objects using 3D Gaussian Splatting by Junfu Guo, Yu Xin, Gaoyi Liu, Kai Xu, Ligang Liu, Ruizhen Hu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08135.txt.

Introduction

Junfu Guo and five co-authors build digital twins of jointed objects from photos, in a paper from 2025. Objects such as tools or doors move at joints, so a twin must learn each part shape and how the parts move.

Their method, ArticulatedGS, learns from two sets of views that show the object in two still poses. It needs no three dimensional ground truth, no motion cues, and no part labels. Steps split the linked unknowns so training stays steady. In tests it beat like methods on part splits, motion scores, and how true the views looked.

Catalog record

CREATORJunfu Guo, Yu Xin, Gaoyi Liu, Kai Xu, Ligang Liu, Ruizhen Hu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-08135

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ArticulatedGS: Self-supervised Digital Twin Modeling of Articulated Objects using 3D Gaussian Splatting Junfu Guo, Yu Xin, Gaoyi Liu, Kai Xu, Ligang Liu, Ruizhen Hu Abstract We tackle the challenge of concurrent reconstruction at the part level with the RGB appearance and estimation of motion parameters for building digital twins of articulated objects using the 3D Gaussian Splatting (3D-GS) method. With two distinct sets of multi-view imagery, each depicting an object in separate static articulation configurations, we reconstruct the articulated object in 3D Gaussian representations with both appearance and geometry information at the same time. Our approach decoupled multiple highly interdependent parameters through a multi-step optimization process, thereby achieving a stable optimization procedure and high-quality outcomes. We introduce ArticulatedGS, a self-supervised, comprehensive framework that autonomously learns to model shapes and appearances at the part level and synchronizes the optimization of motion parameters, all without reliance on 3D supervision, motion cues, or semantic labels. Our experimental results demonstrate that, among comparable methodologies, our approach has achieved optimal outcomes in terms of part segmentation accuracy, motion estimation accuracy, and visual quality. The code will be made publicly available at our website https://guojunfu-tech.github.io/articulatedGS-io/ Real-World Object Any-State Synthesis Motion Estimation & Part Segmentation 3D Gaussians Figure 1 : ArticulatedGS is a self-supervised modeling and reasoning pipeline to reconstruct the articulated objects using 3D Gaussian Splatting. 1 Introduction Articulated objects, such as laptops, ovens, and drawers, are a part of our daily life. Building the digital twin of an articulated object with accurate reconstruction of and fine-grained analysis often plays an important role in robotics [ 32 , 28 , 49 , 25 ] , animation [ 46 , 20 ] , and simulation [ 47 , 43 ] . Moreover, it is also desirable to equip the digital twins with color appearance to facilitate downstream applications such as training sim-to-real transferable embodied AI [ 37 , 1 , 35 ] . Consequently, the capturing and reconstruction of shape geometry , visual appearance , and articulation parameters is indispensable in building accurate and functional digital twins of articulated objects. The problem of modeling high-fidelity digital twins of articulated objects is receiving increasing attention in recent years [ 23 , 22 , 11 , 34 ] . Most of the previous works [ 11 , 22 , 17 ] are concentrated on the regeneration or construction of 3D geometrical representation of articulated objects, which may involve the estimation of articulation parameters or functionalities to animate the objects into different articulation states. The result is usually a mesh-based representation of articulated objects with relatively low geometric fidelity and visual realism. Moreover, the reliance on supervised learning, which typically consumes a large 3D dataset with articulation annotation, also limits the generalizability to unseen object categories. To our knowledge, the only self-supervised work that takes RGB observations as input is PARIS [ 22 ] . PARIS adopts the neural radiance field (NeRF) as an intermediate representation and outputs a mesh with compromised accuracy of geometry and appearance. Additionally, it exhibits instability and suboptimality issues in the optimization process. Recently, there has also been a massive success of 3D Gaussian Splatting (3D-GS) [ 13 ] in 3D scene learning and novel view synthesis as well as extensions to various applications such as 3D reconstruction [ 3 , 18 , 33 ] and dynamic scene rendering [ 4 , 7 ] . 3D-GS requires only RGB images as input and enables rapid scene reconstruction with real-time and realistic rendering. The reconstructed scene is composed of numerous Gaussians densely distributed around the object’s surface, thereby allowing for modeling rich geometric details. Recent works [ 45 , 44 , 26 ] make it possible to interact with digital twins represented with 3D-GS for physically meaningful simulation and manipulation. The fast and realistic rendering of 3D-GS also benefits learning in simulation. For those purposes, what has been missing is to reconstruct digital twins of articulated objects with the representation of 3D Gaussians. In this work, we aim to take the RGB observations of two states of a given articulated object as an input and reconstruct a 3D-GS model of the object with geometry , appearance and mobility . Optimizing for object geometry (masks of movable parts), appearance, and motion parameters simultaneously from only two-view inputs is a highly ill-posed problem. This is because the three terms are deeply intertwined. The object’s movable parts, undergoing rigid body transformations based on motion parameters, can greatly impact on the learning of part geometry and visual appearance. The same goes for the opposite way; the optimization of the movable mask and the motion parameters can be significantly affected by rendering quality. To that end, we propose to decouple the optimization process into multiple sub-processes. We first reconstruct the 3D-GS model of the start state. We then train a network to estimate a deformation flow for moving the movable Gaussians and initialize the geometry structure (movable mask) for the end state. Based on the initial movable mask, we leverage the geometry structure to supervise the learning of the motion parameters. At last, we jointly update the appearance, motion parameters, and movable mask to obtain the final reconstruction of the articulated object. Evaluation on an extended dataset from the previous works [ 8 , 22 ] demonstrates that our method achieves the state-of-the-art performance of visual quality, motion parameter estimation, and part segmentation. Our method also shows enhanced efficiency in model training and more stability of the optimization process compared to the previous self-supervision methods. Our work makes the following contributions: • A 3D Gaussian model for building digital twins of articulated objects in a self-supervised fashion; • A progressive optimization scheme to the challenging optimization task with efficiency and stability; • New state-of-the-art results of reconstruction quality, motion estimation, and visual quality for articulated objects. Figure 2 : A pipeline overview of our self-supervised 3D Articulated Objects modeling method. (a) Given the sets of RGB observation sequence of two states I_{0} and I_{1} of an arbitrary articulated object, we first reconstruct the start state using 3D Gaussian Splatting method into the initial Gaussians \mathcal{G}=\{g_{i}\}_{i=1}^{N} . (b) Each Gaussian g_{i} is then fed into DeformNet, where its Gaussian kernel’s 3D coordinates \boldsymbol{x}_{i} is input to predict its individual rigid motion \{\delta\boldsymbol{x}_{i},\delta\boldsymbol{r}_{i}\} within a local coordinate system, with the end state images I_{1} serving as visual supervision. (c) Based on the transformation predicted by DeformNet, we classify the displacements after regularization to obtain an initial moving Gaussian, and then apply the motion parameters to this moving Gaussian to perform a global rigid transformation, which is supervised by I_{1} , ultimately resulting in a motion category and converged motion parameters. (d) Subsequently, we optimize the preprocessed Gaussians, motion parameters, and the movable mask in tandem to achieve the final results. 2 Related work Articulated Modeling. The primary objective of articulated object reconstruction is to accurately recover the three-dimensional (3D) geometry of articulated entities. Pekelny and Gotsman [ 30 ] introduced one of the pioneering works in this area, which focuses on reconstructing the surface from a sequence of point clouds. A-SDF [ 29 ] represents an innovative approach, the inaugural study to reconstruct articulated objects from signed distance fields (SDFs) derived from a

Read the complete text