\papertitle
A High-Fidelity Digital Twin for Robotic Manipulation Based on 3D Gaussian Splatting · 2026
Read \papertitle by A High-Fidelity Digital Twin for Robotic Manipulation Based on 3D Gaussian Splatting (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-03200.txt.
Introduction
Ziyang Sun and four co-authors build high-fidelity digital twins for robot manipulation using 3D Gaussian splatting, a fast photorealistic scene representation. Good twins matter because they let robots plan motions in simulation and execute them reliably in the real world. Older approaches suffer from slow reconstruction, limited visual quality, and trouble turning pretty models into collision geometry a planner can use.
Their framework constructs a twin within minutes from sparse ordinary photos. Visibility-aware semantic fusion adds accurate three-dimensional labels, and a filter-based conversion produces collision-ready geometry that plugs into a Unity, ROS2, and MoveIt physics setup. In trials, a Franka Emika Panda arm performed pick-and-place tasks against the twin, and the improved geometric accuracy supported robust manipulation in the real world.
Catalog record
| CREATOR | A High-Fidelity Digital Twin for Robotic Manipulation Based on 3D Gaussian Splatting |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-03200
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
\papertitle A High-Fidelity Digital Twin for Robotic Manipulation Based on 3D Gaussian Splatting Abstract † † footnotetext: Copyright©Year by the authors. Published by ELSP. This work is licensed under a Creative Commons Attribution 4.0 International License, which permits unrestricted use, distribution, and reproduction in any medium provided the original work is properly cited. Highlights: • Proposes a novel, end-to-end pipeline leveraging 3DGS to create photorealistic, interactive digital twins from sparse RGB views in minutes. The system integrates perception, planning, real-robot execution ,and tested on real robot. • Introduces a robust method for semantic understanding by lifting 2D masks from foundation models like SAM into the 3D scene using multi-view consensus. It also contributes an efficient conversion of the 3DGS model into planning-ready collision geometry for physics-based simulation. • Demonstrates outstanding real-world performance and closes the sim-to-real gap. The framework is validated through physical experiments on a robotic manipulator, confirming that motion plans generated in the digital twin are effective for real-world execution. Abstract: Developing high-fidelity, interactive digital twins is crucial for enabling closed-loop motion planning and reliable real-world robot execution, which are essential to advancing sim-to-real transfer. However, existing approaches often suffer from slow reconstruction, limited visual fidelity, and difficulties in converting photorealistic models into planning-ready collision geometry. We present a practical framework that constructs high-quality digital twins within minutes from sparse RGB inputs. Our system employs 3D Gaussian Splatting (3DGS) for fast, photorealistic reconstruction as a unified scene representation. We enhance 3DGS with visibility-aware semantic fusion for accurate 3D labelling and introduce an efficient, filter-based geometry conversion method to produce collision-ready models seamlessly integrated with a Unity–ROS2–MoveIt physics engine. In experiments with a Franka Emika Panda robot performing pick-and-place tasks, we demonstrate that this enhanced geometric accuracy effectively supports robust manipulation in real-world trials. These results demonstrate that 3DGS-based digital twins, enriched with semantic and geometric consistency, offer a fast, reliable, and scalable path from perception to manipulation in unstructured environments. Keywords: 3D Gaussian Splatting, Digital Twin, Robotic Manipulation, Real2Sim2Real 1 Introduction The field of robotics is rapidly moving towards full autonomy in complex and unstructured environments. While, effective autonomous manipulation in unstructured environments fundamentally relies on the robot’s ability to rapidly construct a high-fidelity, actionable understanding of its surroundings, which is a core requirement for achieving advanced tasks such as fine-grained manipulation. This actionable understanding relies heavily on the construction of an accurate virtual replica, commonly known as a digital twin [ 14 ] . Digital twins are essential tools that enable safe, repeatable validation, closed-loop motion planning, and reliable sim-to-real transfer, which is important for advancing the deployment of robotic systems in the real world. However, existing reconstruction pipelines [ 25 ] introduce significant bottlenecks that impede their seamless integration into real-time robotic workflows. The pursuit of visual fidelity often conflicts with the need for computational efficiency and physical utility. Specifically, Neural Radiance Fields (NeRFs) [ 17 , 1 ] offer high photorealism but are computationally expensive, often requiring minutes to hours of time for optimization, which severely limits rapid deployment. Conversely, traditional methods based on point clouds or mesh reconstruction [ 16 ] are faster but often suffer from insufficient fidelity, noise sensitivity, and the difficulty of projecting reliable, consistent semantic labels from sparse multi-view inputs. In this context, 3DGS [ 11 ] , a novel explicit radiance field reconstruction method, has shown outstanding performance in both reconstruction quality and speed, emerged as a potential breakthrough representation. 3DGS successfully balances the trade-off between speed and fidelity, achieving photorealistic rendering quality within minutes, which is highly promising for rapid robotic scene capture. However, the core representation used by 3DGS, anisotropic Gaussian splats (or "balls"), defines each scene primitive not merely by a point but by a position, a covariance matrix, a colour, and an opacity. Although this representation excels at combining colour and alpha (opacity) components to create a visually convincing and continuous surface view, the underlying explicit geometry remains inherently ambiguous and problematic for physical interaction. Specifically, the resulting cloud of stretched Gaussian primitives is not a clean, watertight surface, but is riddled with reconstruction artefacts. These issues include floaters (isolated clusters derived from optimisation residuals), ghost artefacts (semi-transparent, low-opacity points near reflective or occluded boundaries), and overall surface fuzziness. These geometric imperfections, while visually hidden in the rendered image, render the raw 3DGS output unsuitable for precise robotic tasks such as collision checking and motion planning. The fragmentation between these approaches highlights a critical, unsolved challenge: generating a digital twin that is simultaneously photorealistic, rapidly reconstructed, and equipped with planning-ready collision geometry and consistent semantic structures. Addressing this triple constraint, speed, fidelity, and actionability, is the central challenge for next-generation robotic perception systems. To bridge the gap between visual fidelity, reconstruction efficiency, and physical utility, and thereby address the digital twins construction problem, we present an end-to-end framework that constructs interactive, semantically structured digital twins directly from sparse RGB inputs. This framework utilises 3DGS [ 11 ] as a unified representation to deliver a rapid real-to-sim-to-real pipeline. This leverages 3DGS for rapid, photorealistic reconstruction (addressing speed and fidelity) that provides the high-fidelity visual experience. Furthermore, we introduce two key components to guarantee a geometrically interactive environment: a visibility-aware semantic fusion module that aggregates multi-view cues [ 12 , 22 ] to ensure consistent 3D labelling (addressing semantic structure), which is then applied to the 3D Gaussian primitives. Following this, a geometric refinement process converts the semantically-labelled, but still noisy, Gaussian primitives into precise collision meshes (addressing planning-ready geometry). Finally, we deploy the generated assets into a specially designed Unity environment where and then robot can simulate with the digital twins, evaluating the manipulation process. Therefore, our framework proposes a unified closed-loop Real-to-simulated-to-real pipeline [ 15 ] illustrated in Fig. 1 for complex robotic manipulation. It seamlessly integrates photorealistic 3DGS reconstruction with our refined collision geometry, semantic understanding, and a Unity based Robot simulation/manipulation interface. This complete, validated sim-to-real workflow is a key advancement, ensuring that motion plans generated against the high-fidelity digital twin are reliable for real-world execution on the real robot. The main contributions of this work are summarized as follows: 1. Unified Real-to-Sim-to-Real Framework: This work proposes a unified real-to-sim-to-real framework that synergizes 3DGS with robust point cloud processing. This approach generates actionable digital twins within minutes in a desk environment, effectively bridging the gap between neural rendering and robotic manipulation. 2. Visibility-Aware Semantic Fusion: This work introduces a view-dependent semantic aggregation with occlusion-aware confidence weighting strategy. This method distils 2D segmentation cues from vision foundation models [ 12 , 21 ] into consistent 3D attributes, resolving projection ambiguities to ensure accurate labelling under occlusion and providing a semantic, manipulatable environment for the robot. 3. Planning-Ready Geometry Conversion: This work implements a multi-scale geometric filtering process with statistical outlier removal and adaptive mesh decimation. By employing alpha-based threshold and robust mesh generation algorithms, this step converts raw Gaussian splat point clouds into precise, planning-ready collision meshes. 4. Experimental Demonstration: This work demonstrates the system’s practical efficacy through a complete perception-planning-execution loop. The framework verifies reliable performance on both standard benchmarks and complex real-world pick-and-place tasks using a Franka Emika robot [ 5 ] . The remainder of this paper is organised as follows.