RayFronts: Open-Set Semantic Ray Frontiers for Online Scene Understanding and Exploration
Omar Alama, Avigyan Bhattacharya, Haoyang He, Seungchan Kim, Yuheng Qiu · 2025
Read RayFronts: Open-Set Semantic Ray Frontiers for Online Scene Understanding and Exploration by Omar Alama, Avigyan Bhattacharya, Haoyang He, Seungchan Kim, Yuheng Qiu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-06994.txt.
Introduction
Omar Alama and co-authors present RayFronts in 2025, a system that lets robots map and explore open worlds by meaning. A robot entering a new building must map what is around it and guess usefully about what lies beyond sensor range. Prior mapping methods either stopped at depth range or handled far entities only in narrow settings.
RayFronts casts semantic rays from the robot's position, marking frontiers where known space ends, labeled by the kinds of things likely beyond. It fuses near and far observations into one map and guides exploration toward informative frontiers. The open-set design handles object types never trained on. Service and rescue robots get a way to explore unknown spaces with judgment, not just geometry.
Catalog record
| CREATOR | Omar Alama, Avigyan Bhattacharya, Haoyang He, Seungchan Kim, Yuheng Qiu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-06994
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RayFronts: Open-Set Semantic Ray Frontiers for Online Scene Understanding and Exploration rayfronts.github.io Thanks: Authors are with Carnegie Mellon University, Pittsburgh, PA, USA. Emails: {oalama, avigyanb, hhe2, seungch2, yuhengq, wenshanw, cherieh, nkeetha, basti}@andrew.cmu.edu Omar Alama, Avigyan Bhattacharya, Haoyang He, Seungchan Kim, Yuheng Qiu Abstract Open-set semantic mapping is crucial for open-world robots. Current mapping approaches either are limited by the depth range or only map beyond-range entities in constrained settings, where overall they fail to combine within-range and beyond-range observations. Furthermore, these methods make a trade-off between fine-grained semantics and efficiency. We introduce RayFronts , a unified representation that enables both dense and beyond-range efficient semantic mapping. RayFronts encodes task-agnostic open-set semantics to both in-range voxels and beyond-range rays encoded at map boundaries, empowering the robot to reduce search volumes significantly and make informed decisions both within & beyond sensory range, while running at 8.84 8.84 Hz on an Orin AGX. Benchmarking the within-range semantics shows that RayFronts ’s fine-grained image encoding provides 1.34 × 1.34\times zero-shot 3D semantic segmentation performance while improving throughput by 16.5 × 16.5\times . Traditionally, online mapping performance is entangled with other system components, complicating evaluation. We propose a planner-agnostic evaluation framework that captures the utility for online beyond-range search and exploration, and show RayFronts reduces search volume 2.2 × 2.2\times more efficiently than the closest online baselines. Fig. 1: RayFronts is a real-time semantic mapping system that enables fine-grained scene understanding both within and beyond the depth perception range. Given an example mission through multi-modal queries to locate red buildings & a water tower, RayFronts enables: (1) Significant search volume reduction for online exploration (as shown by the red and blue cones at the top) and localization of far-away entities (e.g., the water & radio tower). (2) Online semantic mapping, where prior semantic ray frontiers evolve into semantic voxels as entities enter the depth perception range (e.g., the red buildings query on the right side). (3) Multi-objective fine-grained open-set querying supporting various open-set prompts such as “Road Cracks”, “Metal Stairs”, and “Green Dense Canopy”. I Introduction Open-set semantic mapping is essential for robotic systems to reason, search, and navigate in open-world environments. The task requires capturing both fine-grained local details and distant beyond-range semantic cues in real-time. For instance, as shown in Fig. 1 , an aerial or ground robot may need to localize the water or radio towers over 100 meters beyond its depth perception capability, as well as locate any hazards (road cracks) or interesting structures along the way (red building). This work explores what the most effective semantic mapping system would be to capture this information to imbue the robot with the ability to reason within and beyond depth sensing limitations. Although there is a growing body of literature on open-set metric semantic mapping [ 1 , 2 , 3 , 4 , 5 ] , these methods focus primarily on offline mapping for downstream usage in limited environments ignoring efficiency and depth-sensing limitations. Such representations cannot guide the robot in search and exploration tasks as they provide no information about the unmapped region. Other works change the way semantics are typically encoded in a map (point clouds, voxels, and bounding boxes) to representations that can guide exploration (i.e semantic frontiers [ 6 , 7 ] , and semantic poses [ 8 ] ). However, existing semantic frontier maps are limited to 2D indoor environments and have limited semantics due to whole-image encoding [ 6 ] or using closed-set models [ 7 ] , whereas semantic poses [ 8 ] lack fine-grained reconstructions and can only recognize prominent objects in an image. In this context, we explore the question, “How to design an efficient online mapping representation that facilitates fine-grained scene understanding, and be aware of beyond-range semantic entities?” We introduce RayFronts , a semantic map representation which seamlessly integrates traditional within-depth mapping with ray-based representations, facilitating both dense mapping within observed depth ranges and perception beyond them. Unlike conventional representations truncated at the depth range, multi-directional semantic ray frontiers retain coarse-grained far-range information, enabling downstream planners (e.g., object-search) to reduce their search volume significantly. Additionally, to assess the utility of the proposed representation, we construct a planner-agnostic benchmark and propose a new metric to measure how effectively an online mapping strategy reduces the search space for fast object localization and exploration. Finally, to avoid image level encoding and expensive pipelines, we introduce a novel image encoder that achieves state-of-the-art performance on zero-shot 3D semantic segmentation enabling a computationally efficient, open-world, and deployable 3D online mapping system. Our key contributions are as follows: C1: Unified 3D Map Representation for Within-Depth and Beyond-Depth Perception: We develop the first-of-its-kind open-set semantic ray frontier 3D map, which enables robots to reason in open environments achieving up to 1.85x mIoU in offline zero-shot performance, and are 2.2x more efficient in reducing search volume in online mapping than the closest offline & online baselines respectively. C2: Planner-Agnostic Online Semantic Mapping Evaluation Framework: We showcase that online semantic mapping systems can be evaluated on their fundamental utility for exploration, without being tightly coupled with a planner, by developing a metric that assesses “correctly reduced search volume”. C3: Efficient real-time open-set online mapping system: can run end to end at 8.84 Hz on an ORIN AGX and our efficient dense vision-language encoder is 16.5x faster than the closest baseline and achieves state-of-the-art on open-vocab zero-shot 3D semantic segmentation mIoU. II Related Work II-A Dense 2D Open-Set Semantics The rapid rise of foundation models [ 9 ] has spearheaded progress in tasks requiring fine-grained open-set concepts which are hard to capture with a fixed taxonomy of semantic classes [ 10 , 11 ] . CLIP [ 12 ] and its subsequent variants such as SIGLIP [ 13 ] have shown impressive alignment between abstract textual concepts and images. These Visual Language Models (VLMs) initially aligned textual descriptions and images as a whole and not to particular regions or pixels. Subsequently, follow-up work based on supervised and unsupervised regimes has attempted to address this issue [ 14 ] . A recurring theme in these methods is the trade-off between efficiency and accuracy, with the most performant approaches often using multiple foundation models like DINOv2 [ 10 ] , Grounding DINO [ 15 ] , and SAM [ 11 ] . This is not optimal for online real-world deployment and hence we explore the applicability of RADIO [ 16 ] , a foundation model aligned with various dense visual foundation models. While RADIO’s language alignment is to the image as a whole, we find that employing a simple attention trick [ 17 ] with its SIGLIP adapter enables us to achieve state-of-the-art pixel-level language alignment and real-time performance on embedded hardware. II-B Offline & Bounded Open-Set Semantic Mapping Traditional semantic mapping systems have relied on learning-based methods to detect and segment a fixed set of concepts, with performance limited by vocabulary size and training distribution [ 18 , 19 , 20 , 21 , 22 , 23 , 24 ] . With the rise of dense 2D open-set semantics, interest has grown in open-vocab semantic mapping systems using representations like point clouds, voxels, and scene graphs [ 25 ] . These systems