DynamicGSG: Dynamic 3D Gaussian Scene Graphs for Environment Adaptation
Luzhou Ge, Xiangyu Zhu, Zhuo Yang, Xuesong Li · 2025
Read DynamicGSG: Dynamic 3D Gaussian Scene Graphs for Environment Adaptation by Luzhou Ge, Xiangyu Zhu, Zhuo Yang, Xuesong Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-15309.txt.
Introduction
Luzhou Ge and co-authors built a robot memory that updates itself as rooms change, in a 2025 paper. Long-term robot work fails when people move things and the robot's map goes stale. Their system, DynamicGSG, renders the scene with Gaussian splatting and layers a scene graph over it: objects, places, and how they relate, all named in open vocabulary by vision-language models.
When the real room changes, only the affected patch of the graph gets rebuilt. Tests covered labeling, finding objects by description, and rebuild quality, each beating prior methods. A live lab trial showed the updates working in practice. The authors pitch it as memory for robots that stay on the job for months.
Catalog record
| CREATOR | Luzhou Ge, Xiangyu Zhu, Zhuo Yang, Xuesong Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-15309
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DynamicGSG: Dynamic 3D Gaussian Scene Graphs for Environment Adaptation Luzhou Ge, Xiangyu Zhu, Zhuo Yang, Xuesong Li Abstract In real-world scenarios, environment changes caused by human or agent activities make it extremely challenging for robots to perform various long-term tasks. Recent works typically struggle to effectively understand and adapt to dynamic environments due to the inability to update their environment representations in memory according to environment changes and lack of fine-grained reconstruction of the environments. To address these challenges, we propose DynamicGSG , a dynamic, high-fidelity, open-vocabulary scene graph construction system leveraging Gaussian splatting. DynamicGSG builds hierarchical scene graphs using advanced vision language models to represent the spatial and semantic relationships between objects in the environments, utilizes a joint feature loss we designed to supervise Gaussian instance grouping while optimizing the Gaussian maps, and locally updates the Gaussian scene graphs according to real environment changes for long-term environment adaptation. Experiments and ablation studies demonstrate the performance and efficacy of our proposed method in terms of semantic segmentation, language-guided object retrieval, and reconstruction quality. Furthermore, we validate the dynamic updating capabilities of our system in real laboratory environments. The source code and supplementary experimental materials will be released at: https://github.com/GeLuzhou/Dynamic-GSG . I Introduction Future intelligent robots are supposed to execute diverse long-term complex tasks in dynamic environments based on intricate natural language instructions from humans. To achieve this, agents must possess dynamic environment perception and comprehension capabilities. Prior studies [ 1 , 2 , 3 , 4 ] construct static open-vocabulary scene graphs from sensor data to capture the topological structures of environments at specific moments, which enhances robots’ understanding of complex instructions and facilitates task completion. However, static scene graphs are of limited use in real-world scenarios, as robot workspaces typically change due to human activities or other agents’ operations. Additionally, the inherent latency between the scene graph stored in memory (which guides task planning and execution) and the actual environment state significantly hinders successful task completion. Recent advancements in 3D Gaussian splatting [ 5 , 6 , 7 , 8 ] have attracted significant attention within the robotics community. These developments have applications in various domains, including high-quality reconstruction [ 5 , 6 , 9 , 10 ] robotic manipulation [ 11 ] , semantic embedding [ 12 , 13 ] , and 3D environment understanding [ 14 , 15 ] . The explicit point-based representation of 3D Gaussians effectively facilitates semantic information integration from advanced vision language models, enabling the construction of topological scene representations. Based on fast differentiable rendering and explicit representation, we find 3D Gaussians are particularly suitable for locally rapid updates of reconstructed scenes. Moreover, the high-fidelity environment reconstruction provided by 3D Gaussians naturally supports the development of dynamic, high-quality scene graph construction systems. Fig. 1: The dynamic high-fidelity multi-layer Gaussian scene graphs we constructed can adapt to environment changes, represent the spatial and semantic relationships of the objects, and support various forms of language-guided object retrieval. Most previous works on scene graph construction primarily rely on point clouds [ 1 , 2 , 4 , 16 , 17 ] These methods often struggle to promptly respond to dynamic environment changes and fail to capture fine-grained details of 3D scenes due to the inherent limitations of traditional representation techniques. In this paper, we propose DynamicGSG , a framework that utilizes 3D Gaussian Splatting to construct dynamic, high-fidelity, open-vocabulary scene graphs. Advanced vision models such as Yolo-World [ 18 ] , Segment Anything [ 19 ] , CLIP [ 20 ] are employed to detect objects and extract their semantic features. Subsequently, we analyze the spatial and semantic relationships of objects to build hierarchical scene graphs using Large Vision Language Model (LVLM). Through incorporating additional semantic supervision, we improve the accuracy of instance-level Gaussian grouping and the overall reconstruction quality. Moreover, the rapid training and differentiable rendering of 3D Gaussians facilitate efficient scene updates to accommodate environment changes. In summary, our contributions are as follows: • We propose Dynamic 3D G aussian S cene G raphs, combining instance-level rendering with VLM semantic information to achieve 3D-2D object association and building multi-layer scene graphs with LVLM. • We design a joint loss function that ensures accurate intra-instance Gaussian grouping and high-fidelity scene reconstruction. • We utilize the fast differentiable rendering of Gaussians to update the scene graphs, enabling our system to adapt to dynamic environment changes. • We deployed DynamicGSG in real-lab environments, demonstrating its capability to construct 3D Gaussian scene graphs and perform dynamic updates for effective environment adaptation. II Related Work II-A 3D Scene Graphs 3D scene graph [ 21 , 17 ] offers a hierarchical graph-structured representation of the environment with nodes representing spatial concepts at multiple levels of abstraction (e.g., objects, places, rooms, buildings.) and edges preserving the spatial and semantic relationships between nodes. Taking advantage of the generalization abilities of vision foundation models [ 18 , 19 , 22 , 23 ] and cross-modal grounding capabilities of vision-language models [ 20 , 24 ] enables the construction of 3D scene graphs at the open-vocabulary level. Recent methods [ 1 , 2 , 4 , 3 , 25 ] construct 3D scene graphs embedded with VLM semantic features, facilitating object retrieval, robot manipulation and navigation. However, most of these methods [ 1 , 2 , 4 ] rely on static environment assumptions and typically lack mechanisms for handling dynamic updates. RoboEXP [ 25 ] conducts dynamic scene updates based on object spatial relationships and the scene modifications resulting from robot operations. DovSG [ 3 ] extends dynamic scene graphs construction to mobile agents and utilizes Large Language Models (LLMs) [ 26 ] for task decomposition and planning, enabling robots to accomplish complex tasks in dynamic environments over the long term. To enhance computational efficiency, these methods often employ aggressive point cloud downsampling strategies, which significantly limits their capacity for high-fidelity geometric reconstruction of scene details. II-B Gaussian-based Open-Vocabulary Scene Understanding Recent progress in Gaussian Splatting [ 5 , 6 , 7 , 27 , 8 ] demonstrates outstanding performance in photo-realistic reconstruction with promising efficiency. The gradient-based optimization has also been applied in an online setting to incrementally construct the Gaussian map [ 9 , 10 , 28 , 29 ] through differentiable rendering. Concurrently, advancements in vision foundation models (such as SAM [ 19 ] , CLIP [ 20 ] , DINO [ 30 ] ) have motivated the exploration of integrating 2D semantic features into 3D Gaussians. LangSplat [ 14 ] learns hierarchical semantics using SAM and trains an autoencoder to distill high-dimensional CLIP features into low-dimensional semantic attributes of Gaussians. GaussianGrouping [ 12 ] implements joint 3D Gaussian optimization based on 2D pre-matched SAM masks and 3D spatial consistency, enabling high-quality scene reconstruction and open-vocabulary object segmentation. OpenGaussian [ 15 ] augments 3D Gaussian Splatting with point-level open-vocabulary understanding through SAM-based instance feature training, a two-level codebook for discretization, and instance-level 3D-2D feature association for geometric-semantic alignment. However, Most of the above works require embedding semantic information into pre-trained Gaussian scenes or conducting data preprocess prior to gradient-based optimization. Such offline pipelines inherently conflict with robots’ operational paradigms in dynamic settings. To address these deficiencies, we incrementally construct the semantic