OnlinePG: Online Open-Vocabulary Panoptic Mapping with 3D Gaussian Splatting
Hongjia Zhai, Qi Zhang, Xiaokun Pan, Xiyu Zhang, Yitong Dong, Huaqi Zhang, Dan Xu, Guofeng Zhang · 2026
Read OnlinePG: Online Open-Vocabulary Panoptic Mapping with 3D Gaussian Splatting by Hongjia Zhai, Qi Zhang, Xiaokun Pan, Xiyu Zhang, Yitong Dong, Huaqi Zhang, Dan Xu, Guofeng Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-18510.txt.
Introduction
Zhai and co-authors present OnlinePG, an online open-vocabulary panoptic mapping system built on 3D Gaussian Splatting. Past methods are mostly offline or lack instance-level scene understanding. A local-to-global sliding window design keeps the mapping online and real-time. Segment clustering graphs fuse split pieces into full instances with geometric and semantic cues. Bidirectional matching then merges local maps into the global map for open-vocabulary queries.
Catalog record
| CREATOR | Hongjia Zhai, Qi Zhang, Xiaokun Pan, Xiyu Zhang, Yitong Dong, Huaqi Zhang, Dan Xu, Guofeng Zhang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44132 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.18510
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OnlinePG: Online Open-Vocabulary Panoptic Mapping with 3D Gaussian Splatting Hongjia Zhai, Qi Zhang, Xiaokun Pan, Xiyu Zhang, Yitong Dong, Huaqi Zhang, Dan Xu, Guofeng Zhang Abstract Open-vocabulary scene understanding with online panoptic mapping is essential for embodied applications to perceive and interact with environments. However, existing methods are predominantly offline or lack instance-level understanding, limiting their applicability to real-world robotic tasks. In this paper, we propose OnlinePG, a novel and effective system that integrates geometric reconstruction and open-vocabulary perception using 3D Gaussian Splatting in an online setting. Technically, to achieve online panoptic mapping, we employ an efficient local-to-global paradigm with a sliding window. To build local consistency map, we construct a 3D segment clustering graph that jointly leverages geometric and semantic cues, fusing inconsistent segments within sliding window into complete instances. Subsequently, to update the global map, we construct explicit grids with spatial attributes for the local 3D Gaussian map and fuse them into the global map via robust bidirectional bipartite 3D Gaussian instance matching. Finally, we utilize the fused VLM features inside the 3D spatial attribute grids to achieve open-vocabulary scene understanding. Extensive experiments on widely used datasets demonstrate that our method achieves better performance among online approaches, while maintaining real-time efficiency. Figure 1 : Illustration of OnlinePG , which integrates geometric reconstruction and open-vocabulary panoptic perception built upon 3D Gaussian Splatting. Given the posed video stream and 2D noise priors (mask and feature) generated by vision-language models (VLMs), OnlinePG can reconstruct a consistent 3D language feature field and perform online open-vocabulary panoptic mapping. 1 Introduction Open-vocabulary 3D scene understanding is fundamental for embodied tasks, enabling robots to perceive, reason about, and interact with complex environments using natural language and instruction 9 ; 31 ; 38 ; 33 . While significant progress has been made in offline settings, real-world robotic applications demand online perception and mapping capabilities 27 ; 61 ; 23 that can process streaming data and support decision-making in real time. To overcome the limited capability of annotated 3D data, recent works 49 ; 28 ; 16 ; 22 ; 48 leverage various 2D vision-language models (VLMs) 34 ; 17 ; 1 ; 35 ; 29 ; 10 to transfer open-vocabulary knowledge into 3D space. Building on differentiable rendering techniques such as Neural Radiance Fields (NeRF) 26 and 3D Gaussian Splatting (3DGS) 11 , several methods 57 ; 19 ; 50 ; 33 ; 38 ; 6 have achieved impressive results in fine-grained open-vocabulary scene understanding. However, these approaches are predominantly offline and lack support for online instance-level panoptic perception, hindering their applications in embodied tasks. Despite previous approaches that combine VLMs with 3DGS having yielded satisfactory performance, two critical limitations remain: 1) offline reconstruction and perception settings. Most existing reconstruction and understanding methods 38 ; 57 ; 50 ; 33 require pre-collected data. This offline setting prevents robots from simultaneously exploring and understanding unknown environments in real time. Online mapping and perception are indispensable for robots. 2) missing or inaccurate 3D instance-level understanding. Current online open-vocabulary scene understanding approaches 42 ; 52 cannot distinguish individual 3D instances based on text queries, while offline instance-aware approaches 19 ; 50 ; 57 ; 39 rely on time-consuming clustering or multi-view guidance that is unsuitable for online systems. However, equipping the online mapping system with capabilities of open-vocabulary panoptic understanding is highly meaningful and needed for embodied applications. Addressing these challenges is crucial for enabling real-time, open-vocabulary panoptic mapping and understanding in embodied applications. To this end, we present OnlinePG, an efficient online open-vocabulary panoptic mapping system based on 3D Gaussian Splatting that integrates geometric reconstruction with semantic understanding. Our key insight is to employ an efficient local-to-global paradigm: we first build local consistent maps within a sliding window to mitigate noisy 2D VLM priors, then incrementally fuse them into a global representation. Specifically, given noisy 2D masks and feature maps from VLMs, we initialize 3D Gaussian segments from keyframes and construct a multi-cue clustering graph that leverages geometric overlap, semantic similarity, and view consensus to merge inconsistent segments into complete instances. We then voxelize the local map to build explicit spatial attribute grids for storing semantic features and instance labels, which are robustly fused into the global map via our bidirectional bipartite matching scheme. This design enables accurate instance-level open-vocabulary understanding while maintaining real-time performance. Overall, the technical contributions of our approach are summarized as follows: • We propose an online open-vocabulary panoptic mapping framework that unifies geometric reconstruction and semantic understanding in a local-to-global paradigm. • We develop a multi-cue segment clustering algorithm that synergistically leverages geometric, semantic, and view consensus information to obtain local consistent 3D instances from noisy 2D VLM priors. • We design a bidirectional bipartite matching scheme with explicit spatial attributes for robust incremental fusion of local 3D instances into the global map. • Extensive experiments demonstrating our superior performance among online methods, with significant improvements in both semantic and panoptic metrics. 2 Related Work Panoptic Segmentation. 2D panoptic segmentation task was first proposed by Kirillov et al . 13 , which aims to understand objects with different semantics and instances within images. Follow-up works 3 ; 4 ; 32 have attempted to advance the CNN model’s ability to reason and understand images from a human perspective. To explore spatial semantics and understand 3D scenes, some works 28 ; 57 ; 39 ; 7 ; 16 ; 64 integrate and lift 2D panoptic predictions of 2D models 4 ; 21 into 3D space. Inspired by 25 , PanopticFusion 28 fuses the 2D panoptic information into 3D volume and performs map regularization. In addition, some works 16 ; 57 ; 19 have exploited recent optimization and rendering capabilities of differentiable representations 26 ; 11 to understand 3D scenes. However, limited by the offline or closed-set setting, these works cannot simultaneously perform online 3D panoptic reconstruction or open vocabulary 3D scene understanding. Online 3D Scene Mapping and Perception. For real-world applications of embodied intelligence 27 ; 61 ; 56 ; 9 , intelligent robots typically perceive and understand the environment with an online data stream. Previous online 3D scene mapping and perception methods 51 ; 36 ; 22 ; 41 ; 52 ; 28 were generally implemented by projecting 2D semantic predictions back into 3D scene maps. Accurately reconstructing and understanding the surrounding 3D scene from RGB-D video streams is the visual foundation for these robotic tasks. Benefiting from NeRF/3DGS-based reconstruction 59 ; 18 ; 62 ; 60 , recent works 45 ; 56 ; 54 ; 58 ; 42 ; 30 ; 65 have incorporated NeRF and 3DGS for fine-grained, better online mapping and perception performance. However, these online approaches cannot achieve instance-aware panoptic mapping and perception for open-vocabulary scene understanding. Figure 2 : Overview of Our Approach . Our system performs online open-vocabulary panoptic mapping from RGB-D streams using a local-to-global paradigm. (a) Maintaining a sliding window \mathcal{W} to build locally consistent instance and fuse them globally. (b) Using noisy 2D masks and features from 21 ; 17 to initialize 3D Gaussian segments, which are clustered via multi-cues to form consistent 3D instances. (c) Bidirectional bipartite matching incrementally fuses local instances and spatial attributes into the global map. Open Vocabulary 3D Scene Understanding.