OmniVLN: Omnidirectional 3D Perception and Token-Efficient LLM Reasoning for Visual-Language Navigation across Air and Ground Platforms

Zhongyuang Liu, Min He, Shaonan Yu, Xinhang Xu, Muqing Cao, Jianping Li, Jianfei Yang, Lihua Xie · 2026

Read OmniVLN: Omnidirectional 3D Perception and Token-Efficient LLM Reasoning for Visual-Language Navigation across Air and Ground Platforms by Zhongyuang Liu, Min He, Shaonan Yu, Xinhang Xu, Muqing Cao, Jianping Li, Jianfei Yang, Lihua Xie (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-17351.txt.

Introduction

Liu and co-authors present OmniVLN, a navigation system for air and ground robots that follow language orders. It joins rotating LiDAR with panoramic vision to build a five-layer scene graph. A compact octant view then guides a language model to pick rooms, aim, and move. Spatial referring accuracy rose from about 77 to 93 percent in their tests. Prompt tokens fell by up to 61.7 percent, and success rose as well.

Catalog record

CREATORZhongyuang Liu, Min He, Shaonan Yu, Xinhang Xu, Muqing Cao, Jianping Li, Jianfei Yang, Lihua Xie
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35210
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.17351

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

OmniVLN: Omnidirectional 3D Perception and Token-Efficient LLM Reasoning for Visual-Language Navigation across Air and Ground Platforms Zhongyuang Liu, Min He, Shaonan Yu, Xinhang Xu, Muqing Cao, Jianping Li, Jianfei Yang, Lihua Xie Abstract Language-guided embodied navigation requires an agent to interpret object-referential instructions, search across multiple rooms, localize the referenced target, and execute reliable motion toward it. Existing systems remain limited in real indoor environments because narrow field-of-view sensing exposes only a partial local scene at each step, often forcing repeated rotations, delaying target discovery, and producing fragmented spatial understanding; meanwhile, directly prompting LLMs with dense 3D maps or exhaustive object lists quickly exceeds the context budget. We present OmniVLN, a zero-shot visual-language navigation framework that couples omnidirectional 3D perception with token-efficient hierarchical reasoning for both aerial and ground robots. OmniVLN fuses a rotating LiDAR and panoramic vision into a hardware-agnostic mapping stack, incrementally constructs a five-layer Dynamic Scene Graph (DSG) from mesh geometry to room- and building-level structure, and stabilizes high-level topology through persistent-homology-based room partitioning and hybrid geometric/VLM relation verification. For navigation, the global DSG is transformed into an agent-centric 3D octant representation with multi-resolution spatial attention prompting, enabling the LLM to progressively filter candidate rooms, infer egocentric orientation, localize target objects, and emit executable navigation primitives while preserving fine local detail and compact long-range memory. Experiments show that the proposed hierarchical interface improves spatial referring accuracy from 77.27% to 93.18%, reduces cumulative prompt tokens by up to 61.7% in cluttered multi-room settings, and improves navigation success by up to 11.68% over a flat-list baseline. We will release the code and an omnidirectional multimodal dataset to support reproducible research. I Introduction Executing object-referential instructions, such as “find the mug near the sink,” is a fundamental capability for embodied intelligence [ 1 ] . In visual-language navigation (VLN), this task requires an agent to interpret a language goal, search over a partially observed environment, disambiguate the referenced target from surrounding context, and finally convert that decision into executable motion. Recent advances in Large Language Models (LLMs) and Vision-Language Models (VLMs) have transformed navigation from a purely geometric control problem into a semantic reasoning process, where an agent must connect language, perception, memory, and action [ 8 , 31 ] . Despite this progress, reliable language-guided navigation in complex multi-room environments remains challenging, especially when long-horizon search and cross-room reasoning must be supported by real-world perception. A first challenge lies in the sensing front-end. Existing VLN systems are commonly built upon narrow field-of-view observations, so the agent sees only a partial local scene at each step and must infer the rest from sparse glimpses [ 5 , 28 ] . In practice, this limited-view setting often causes repeated rotations, delayed discovery of side- or rear-located targets, and fragmented topological understanding during exploration and search. These limitations become more severe in cluttered or occluded indoor environments, where small visibility gaps can accumulate into large navigation errors across rooms. In addition, current embodied navigation systems typically rely on platform-specific sensing layouts, making it difficult to transfer a unified perception pipeline across heterogeneous aerial and ground robots. As a result, the lack of a hardware-agnostic omnidirectional perception front-end remains a key obstacle to scalable cross-platform deployment. Figure 1: OmniVLN, a zero-shot visual-language navigation (VLN) framework coupling 360^{\circ} 3D perception with token-efficient reasoning across aerial & ground platforms. A second challenge arises once perception is expanded to omnidirectional 3D mapping. Although richer observations improve global situational awareness, they also produce denser semantic maps, more candidate objects, and more long-range spatial relations that must be reasoned over during navigation. Directly verbalizing such information for LLM-based planning quickly leads to a token explosion problem, where exhaustive object lists or dense semantic map descriptions exceed the prompt budget, reduce reasoning stability, and increase inference latency [ 29 , 31 ] . This issue is particularly harmful for long-horizon navigation, where the agent must maintain room-level context, egocentric orientation, and target-specific evidence over multiple decision steps. Therefore, an effective navigation system must not only perceive more, but also compress the world model into concise, decision-relevant representations suitable for scalable reasoning. To address these issues, we present OmniVLN, a framework that bridges 360^{\circ} spatio-temporally consistent perception with token-efficient hierarchical reasoning. OmniVLN integrates rotating LiDAR and panoramic vision into an omnidirectional sensing and mapping stack, producing a unified semantic 3D representation that can be deployed on both aerial and ground robotic platforms. On top of this perception front-end, we incrementally construct a five-layer Dynamic Scene Graph (DSG) [ 18 ] , which abstracts the environment from low-level mesh geometry to rooms and building-level structure. To support navigation-oriented reasoning, we further introduce an agent-centric 3D octant observation model that translates global graph information into an egocentric and compact spatial interface. Combined with a multi-resolution spatial attention prompting strategy, OmniVLN enables the agent to progressively select relevant rooms, orient itself with respect to the target, localize candidate objects, and emit executable navigation actions while preserving fine-grained local details and compressing distant regions into concise topological summaries. The main contribution of this paper is a unified system integration of omnidirectional 3D perception, hierarchical scene abstraction, and token-efficient LLM/VLM reasoning for language-guided navigation. Specifically, this work makes the following contributions: (1) We develop a cross-platform omnidirectional perception front-end that combines rotating LiDAR and panoramic cameras to provide consistent 360 ∘ semantic 3D mapping for both aerial and ground robots. (2) We build a hierarchical scene representation and reasoning pipeline that integrates Dynamic Scene Graph construction, adaptive room-level topology abstraction, and agent-centric 3D octant observations into a compact world model for cross-room embodied navigation. (3) We design a practical token-efficient prompting strategy that enables scalable long-horizon reasoning over omnidirectional 3D environments without naively verbalizing dense maps or exhaustive object lists, thereby supporting target search and action generation under strict prompt budgets. (4) We release an open-source multimodal dataset and deployable system framework to support reproducible evaluation of omnidirectional embodied perception and language-guided navigation. Figure 2: Overview of the proposed framework. The Multimodal Perception module (left) achieves 360^{\circ} spatio-temporal consistency by fusing data from a rotating LiDAR and panoramic fisheye cameras, enabling the generation of high-fidelity semantic point clouds across robotic platforms. The Hierarchical Representation module (center) incrementally constructs an online five-layer DSG, bridging the gap between low-level geometric places and high-level macro-spatial rooms, while employing a VLM-based hybrid pruning mechanism to refine physical veracity. The LLM-based Reasoning module (right) transforms global graph knowledge into an agent-centric 3D octant observation model, utilizing a closed-loop actor–critic framework and DSG-guided hierarchical prompting to translate natural language queries into executable navigation actions. II Related Work II-A Omnidirectional Perception and Embodied Datasets Omnidirectional sensing improves coverage and reduces rotational overhead in cluttered indoor environments [ 16 , 22 , 10 , 20 ] . Yet existing embodied systems are typically either vision-centric with limited metric fidelity or LiDAR-centric with directional blind spots [ 21 , 9 , 30 ] . Mainstream navigation benchmarks also largely assume narrow-FoV RGB-D sensing [ 19 , 7 , 6 , 4 , 24 ] , leaving limited support for synchronized omnidirectional geometry, semantics, and language tasks. Our work addresses this gap with a rotating-LiDAR and panoramic-vision pipeline and a corresponding dataset. II-B Hierarchical Representations

Read the complete text