GaussTwin: Unified Simulation and Correction with Gaussian Splatting for Robotic Digital Twins
Yichen Cai, Paul Jansonnie, Cristiana de Farias, Oleg Arenz, Jan Peters · 2026
Read GaussTwin: Unified Simulation and Correction with Gaussian Splatting for Robotic Digital Twins by Yichen Cai, Paul Jansonnie, Cristiana de Farias, Oleg Arenz, Jan Peters (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-05108.txt.
Introduction
Yichen Cai and team offer GaussTwin, a real-time digital twin mixing Cosserat rod physics with position-based dynamics for simulation. It adds Gaussian splatting for rendering and visual fix. Gaussians anchored to real primitives update coherently under photometric error and splitting masks, giving stable guess-fix while keeping real trueness. Tests in simulation and with a Franka Research 3 robot show raised tracking accuracy and toughness over shape-matching and rigid-only older models. The twin also lets push-based planning happen.
Catalog record
| CREATOR | Yichen Cai, Paul Jansonnie, Cristiana de Farias, Oleg Arenz, Jan Peters |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40222 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.05108
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GaussTwin: Unified Simulation and Correction with Gaussian Splatting for Robotic Digital Twins Yichen Cai, Paul Jansonnie, Cristiana de Farias, Oleg Arenz, Jan Peters Abstract Digital twins promise to enhance robotic manipulation by maintaining a consistent link between real-world perception and simulation. However, most existing systems struggle with the lack of a unified model, complex dynamic interactions, and the real-to-sim gap, which limits downstream applications such as model predictive control. Thus, we propose GaussTwin, a real-time digital twin that combines position-based dynamics with discrete Cosserat rod formulations for physically grounded simulation, and Gaussian splatting for efficient rendering and visual correction. By anchoring Gaussians to physical primitives and enforcing coherent SE(3) updates driven by photometric error and segmentation masks, GaussTwin achieves stable prediction–correction while preserving physical fidelity. Through experiments in both simulation and on a Franka Research 3 platform, we show that GaussTwin consistently improves tracking accuracy and robustness compared to shape-matching and rigid-only baselines, while also enabling downstream tasks such as push-based planning. These results highlight GaussTwin as a step toward unified, physically meaningful digital twins that can support closed-loop robotic interaction and learning. Code and videos are available at https://6cyc6.github.io/gstwin/ . I Introduction Building a real-time, dynamic digital twin offers significant benefits for robotic manipulation. Unlike traditional simulation, which creates a virtual environment separate from the real world, a dynamic digital twin predicts future states, continuously corrects and synchronizes the robot with reality, and generates visual representations of changing environment conditions. This capability helps bridge the real-to-simulation gap, enabling more effective planning, tracking, and the generation of robot–object interaction videos for policy learning [ 42 , 3 ] . However, developing digital twins that rely on a unified physics model capable of handling a wide range of properties and behaviors, such as rigid and deformable bodies, contact interactions, and different material types, remains a significant challenge. In practice, digital twins have typically been constructed using point clouds [ 33 , 36 , 21 , 20 ] , meshes [ 30 ] , or NeRF [ 1 ] to represent the 3D environment, but these representations face trade-offs among efficiency, differentiability, and fidelity. Recently, 3DGS ( 3DGS ) [ 15 ] has emerged as an alternative for 3D scene representation. By modeling a scene as a dense set of 3D Gaussians, 3DGS enables efficient and differentiable novel-view image synthesis with low memory cost and high visual fidelity. Building on this idea, works such as, [ 44 , 14 , 43 ] combine 3DGS -based reconstruction with neural dynamic models to construct more visually accurate digital twins that capture dynamics directly from videos. Here, by integrating Graph Neural Networks with a particle-grid representation, [ 43 ] demonstrated strong capabilities in modeling deformable object dynamics, achieving accurate long-horizon predictions while leveraging 3DGS for rendering. However, these learning-based approaches often struggle to generalize beyond their training distribution and demand extensive, labor-intensive data collection and processing. Fig. 1: Tracking performance of GaussTwin on rigid objects and DLO. The system takes multi-view camera observations as input. First, the objects in the scene are masked, represented by particles, and bonded to 3D Gaussians. The motion of these particles is then predicted at each time step using PBD, and subsequently refined through Gaussian splatting optimization. The bottom two rows illustrate the prediction–correction process carried out by GaussTwin. Concurrently with learning-based methods, another line of work has explored hybrid approaches that integrate physics-based simulation with visual or data-driven corrections. As was often the case in earlier digital twin implementations, point clouds or signed distance fields were commonly used for the visual correction step [ 33 , 36 , 21 , 38 , 20 ] . However, these representations are typically tailored to specific object types such as ropes or soft tissues, which restricts their general applicability. In addition, both point clouds and SDFs are prone to occlusion and sensor noise. In parallel with the advances described above, 3DGS has also emerged as a promising alternative for the visual correction component of prediction-correction models. For instance, [ 2 , 3 ] introduced a 3DGS -based visual corrective world model, which uses a simulator for prediction and then refines the results through visual feedback. Both methods are built upon PBD ( PBD ) [ 26 ] , a simulation framework that enforces constraints on positions of particles to achieve fast and stable real-time dynamics. Specifically, [ 2 ] proposed a hybrid Particle-Gaussians representation, where particles evolve in a PBD simulator while bonded 3D Gaussians are used for image rendering and visual feedback. Their approach further employed a shape-matching algorithm with oriented particles [ 27 , 25 ] to simulate both rigid and deformable objects within the PBD framework. However, the shape-matching algorithm lacks physically meaningful properties, leading to inaccurate simulation predictions, especially for DLO ( DLO ), e.g., a rope. Moreover, it requires a high visual correction gain because the Gaussians are optimized independently, which often leads to strong oscillations. In follow-up work, [ 3 ] adopted rigid-body dynamics to achieve more accurate predictions and applied their improved corrective model to a manipulation task, where they used it to push a T-shaped object. While this enhanced rigid-body tracking, it came at the cost of losing the ability to simulate and track deformable objects. In this paper, we present GT ( GT ), a unified hybrid framework that overcomes the limitations of prior 3DGS -based corrective models (See Fig. 1 ). Instead of relying on shape matching alone or restricting to rigid-body dynamics, we extend PBD with the discrete Cosserat rod model [ 17 ] , which provides a physically meaningful formulation capable of accurately modeling both rigid objects and DLO within the same framework. To further improve stability, we introduce a joint optimization scheme for 3D Gaussians that constrains them to move coherently with their associated rigid bodies or rod segments, preventing independent drift that previously caused oscillations and required high correction gains. Together, these contributions enable GaussTwin to unify the prediction and correction of rigid and deformable objects, while ensuring more stable, efficient, and physically accurate digital twin simulations. The key contributions of this paper are summarized as follows: • We introduce GT , a hybrid framework that combines PBD with 3D Gaussians to jointly predict and correct the state of both rigid bodies and DLO , thereby bridging the real-to-sim gap. • By leveraging segmentation masks, enforcing coherent motion of 3D Gaussians with their corresponding rigid bodies or rod segments, and incorporating physically plausible constraints in PBD , our method achieves stable and precise prediction–correction without sacrificing real-time performance. • Through simulation and real-world experiments, we demonstrate that GT achieves more accurate and robust tracking compared to prior 3DGS -based corrective models [ 2 ] . • We show the effectiveness of our method in a downstream robotic planning task, highlighting its potential to support closed-loop interaction and control in real-world environments. II Related Works Building a model that can simulate and track both rigid and linear deformable objects is crucial for complex robotic manipulation tasks. Many physics simulators are available to simulate rigid bodies [ 37 , 29 ] and DLOs [ 29 , 18 , 41 , 23 ] . However, they fail to predict and track