Hierarchical Language Models for Semantic Navigation and Manipulation in an Aerial-Ground Robotic System

Haokun Liu, Zhaoqi Ma, Yunong Li, Junichiro Sugihara, Yicheng Chen, Jinjie Li, Moju Zhao · 2025

Read Hierarchical Language Models for Semantic Navigation and Manipulation in an Aerial-Ground Robotic System by Haokun Liu, Zhaoqi Ma, Yunong Li, Junichiro Sugihara, Yicheng Chen, Jinjie Li, Moju Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-05020.txt.

Introduction

Haokun Liu and co-authors present, in a 2025 paper, a two-brain control scheme for a drone-and-rover team that tidies objects over long tasks. Mixed robot teams promise much, but fixed task models break when scenes change. Their design splits thought by level.

A large language model breaks the job down and builds a global semantic map. A vision-language model, sharpened by a grid masking trick, spots and locates objects precisely for the rover's gripper. The drone scouts from above, drawing semantic paths that steer the rover's local moves, even when the target is missing or the scene is ambiguous. Trials in simulation and the real world showed the pair arranging objects over long horizons without retraining. The authors claim the first such system marrying vision-language perception to language-model planning.

Catalog record

CREATORHaokun Liu, Zhaoqi Ma, Yunong Li, Junichiro Sugihara, Yicheng Chen, Jinjie Li, Moju Zhao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-05020

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Hierarchical Language Models for Semantic Navigation and Manipulation in an Aerial-Ground Robotic System Haokun Liu, Zhaoqi Ma, Yunong Li, Junichiro Sugihara, Yicheng Chen, Jinjie Li, Moju Zhao Abstract Heterogeneous multi-robot systems show great potential in complex tasks requiring coordinated hybrid cooperation. However, traditional approaches relying on static models often struggle with task diversity and dynamic environments. This highlights the need for generalizable intelligence that can bridge high-level reasoning with low-level execution across heterogeneous agents. To address this, we propose a hierarchical framework integrating a prompted Large Language Model (LLM) and a GridMask-enhanced fine-tuned Vision Language Model (VLM). The LLM performs task decomposition and global semantic map construction, while the VLM extracts task-specified semantic labels and 2D spatial information from aerial images to support local planning. Within this framework, the aerial robot follows a globally optimized semantic path and continuously provides bird-view images, guiding the ground robot’s local semantic navigation and manipulation, including target-absent scenarios where implicit alignment is maintained. Experiments on real-world cubes or objects arrangement tasks demonstrate the framework’s adaptability and robustness in dynamic environments. To the best of our knowledge, this is the first demonstration of an aerial-ground heterogeneous system integrating VLM-based perception with LLM-driven task reasoning and motion planning. I Introduction Heterogeneous Multi-Robot Systems (HMRS), composed of agents with diverse capabilities—such as aerial, ground, or underwater robots—are well-suited for complex tasks that demand a combination of manipulation, navigation, and observation [ 1 ] . Compared to homogeneous systems [ 2 ] , HMRS provides greater flexibility and efficiency in dynamic environments. However, unlocking their full potential remains a significant challenge due to the inherent complexity of cross-modal coordination. Traditional approaches to robot control and coordination often rely on pre-programming and explicit communication, which struggle to adapt to unforeseen situations—a limitation commonly observed in homogeneous systems [ 3 ] , and further intensified in heterogeneous systems due to their cross-modal complexity and coordination requirements. In dynamic environments, where tasks and conditions rapidly change, static models and pre-defined behaviors often lead to suboptimal decisions and poor coordination [ 4 ] . Traditional control strategies struggle to generalize beyond fixed scenarios, particularly in task allocation, motion planning, and manipulation within dynamic environments. This limitation highlights the need for adaptable intelligence capable of decomposing high-level instructions for task allocation, performing motion planning, and executing manipulation tasks, without task-specific training. This need for adaptable intelligence has been catalyzed by recent advancements in large-scale multimodal models, such as GPT [ 5 ] and Gemini [ 6 ] , which have enabled Multi-Agent Language Model (MA-LLM) systems that combine high-level reasoning with visual perception. The MA-LLM frameworks utilize the semantic reasoning capabilities of LLMs and the perceptual grounding offered by VLMs, allowing robots to process both linguistic and visual information. While LLMs can decompose high-level tasks into executable motion functions [ 7 ] , and VLMs provide semantic and spatial information from images [ 8 ] , directly integrating reasoning and perception into a single-layer system often results in tight coupling, leading to brittleness against perception errors and difficulties in long-horizon task maintenance. To address this, we propose a hierarchical MA-LLM framework that modularly separates reasoning, perception, and execution. This hierarchical structure facilitates flexible task allocation, semantic-aware motion planning, and robust manipulation within dynamic, heterogeneous robotic systems. Fig. 1: Overview of the proposed hierarchical language model framework integrated with an aerial-ground robotic system. In the sub-task “move to (XX, XX)”, the aerial robot follows a globally optimized path while continuously capturing bird-view images. These images are processed into semantic observations that guide the ground robot’s real-time local navigation by implicitly following the aerial robot’s position. In our proposed hierarchical MA-LLM framework (Figure 1 ), task execution is structured into three functional layers: a reasoning layer that decomposes user commands into task flows, a perceptual layer that extracts semantic information from aerial imagery, and an execution layer that performs motion through aerial-ground collaboration. The execution process is guided by a leader-follower mechanism, where the aerial robot plans a globally optimized path and provides a continuous stream of bird-view images. The ground robot uses the center of these images—corresponding to the aerial robot’s projected ground position—as a dynamic navigation anchor, allowing it to follow the aerial path even when the target object is temporarily out of view. The main contributions of this paper are as follows: 1. A hierarchical multimodal MA-LLM framework for aerial-ground robotic systems, where the aerial robot provides global semantic guidance through bird-view images and natural language reasoning, and the ground robot performs local navigation and manipulation by implicitly following the aerial path. This system bridges high-level reasoning and low-level execution, enabling adaptive and robust task performance in dynamic environments. 2. An LLM-guided reasoning framework that performs two critical functions: hierarchical task decomposition and global semantic map construction. The LLM decomposes high-level user commands into executable sub-tasks and constructs a comprehensive global semantic map by integrating spatial-semantic information collected from multiple viewpoints, which serves as the basis for aerial path planning. 3. A VLM-driven perception strategy enhanced by GridMask-based fine-tuning is firstly proposed, enabling accurate 2D objects localization and semantic labeling from bird-view images without depth sensing. The VLM extracts task-relevant semantic representations in real time, providing essential input for both global map construction and robots’ motion planning. 4. Comprehensive evaluation of zero-shot transferability and generalizability through extensive simulation experiments. These experiments demonstrate the system’s capability to generalize semantic reasoning, navigation, and manipulation skills to novel object categories, ambiguous semantic configurations, and spatially complex environments without additional training or parameter adjustments. 5. Extensive real-world experiments validate the effectiveness and robustness of our hierarchical MA-LLM framework in dynamic, long-horizon tasks. Using a heterogeneous aerial-ground robotic system, we demonstrate robust semantic navigation, precise manipulation, and coordinated behavior through letter-block assembly tasks, highlighting the system’s adaptability and decision-making capabilities in complex real-world scenarios. II Related Works II-A Heterogeneous Multi-Robot Systems HMRS leverages diverse agent capabilities to address complex tasks. One of the earliest works in this area is Parker’s ALLIANCE architecture [ 9 ] , which introduced fault-tolerant behavior-based coordination. Follow-up studies examined distributed task allocation and coordination [ 10 , 11 , 12 ] . Recent methods also integrate deep Multi-Agent Reinforcement Learning (MARL) to improve scalability and adaptability in decentralized settings [ 13 ] . Beyond these approaches, mapping frameworks in HMRS (shared-SLAM [ 14 , 15 ] , also MARL [ 16 , 17 ] ) have enabled teams of robots to maintain consistent global spatial representations. However, such systems mainly facilitate spatial alignment and navigation, while high-level semantic reasoning and flexible multi-stage task planning typically require additional manual programming [ 18 , 19 ] . In contrast, our hierarchical MA-LLM framework directly bridges high-level task decomposition, semantic-aware navigation, and low-level manipulation. By leveraging LLM and VLM for reasoning and perception, our system interprets complex instructions, dynamically allocates tasks, and enables modular semantic navigation and manipulation among heterogeneous agents. This approach offers greater flexibility for multi-robot cooperation, efficiently addressing challenges that are cumbersome or impractical for previous methods, particularly in scenarios requiring complex reasoning and adaptive coordination. II-B Task Planning with Large Language Models LLMs have enabled robots to decompose natural language into action plans, as demonstrated in SayCan [ 20 ] ,

Read the complete text