Hierarchical Instruction-aware Embodied Visual Tracking
Kui Wu, Hao Chen, Churan Wang, Fakhri Karray, Zhoujun Li, Yizhou Wang, Fangwei Zhong · 2025
Read Hierarchical Instruction-aware Embodied Visual Tracking by Kui Wu, Hao Chen, Churan Wang, Fakhri Karray, Zhoujun Li, Yizhou Wang, Fangwei Zhong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20710.txt.
Introduction
Kui Wu and six co-authors build a robot tracker that takes orders in plain language in a 2025 paper. The task is user-centric tracking: a person tells a mobile agent whom or what to follow, and the agent must turn vague words into wheel-level moves. Big language models read orders well but answer slowly, while action-tuned models are fast yet brittle on new wording.
Their agent splits the job into layers. A language model first turns each order into a spatial goal marked on the scene. A lower layer then steers toward that goal mark by mark. Spatial goals form the bridge between understanding and action, letting each layer do what it does best. Tests check the stack on varied phrasings and moving targets.
Catalog record
| CREATOR | Kui Wu, Hao Chen, Churan Wang, Fakhri Karray, Zhoujun Li, Yizhou Wang, Fangwei Zhong |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 80 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-20710
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Hierarchical Instruction-aware Embodied Visual Tracking Kui Wu, Hao Chen, Churan Wang, Fakhri Karray, Zhoujun Li, Yizhou Wang, Fangwei Zhong Abstract User-Centric Embodied Visual Tracking (UC-EVT) presents a novel challenge for reinforcement learning-based models due to the substantial gap between high-level user instructions and low-level agent actions. While recent advancements in language models (e.g., LLMs, VLMs, VLAs) have improved instruction comprehension, these models face critical limitations in either inference speed (LLMs, VLMs) or generalizability (VLAs) for UC-EVT tasks. To address these challenges, we propose Hierarchical Instruction-aware Embodied Visual Tracking (HIEVT) agent, which bridges instruction comprehension and action generation using spatial goals as intermediaries. HIEVT first introduces LLM-based Semantic-Spatial Goal Aligner to translate diverse human instructions into spatial goals that directly annotate the desired spatial position. Then the RL-based Adaptive Goal-Aligned Policy , a general offline policy, enables the tracker to position the target as specified by the spatial goal. To benchmark UC-EVT tasks, we collect over ten million trajectories for training and evaluate across one seen environment and nine unseen challenging environments. Extensive experiments and real-world deployments demonstrate the robustness and generalizability of HIEVT across diverse environments, varying target dynamics, and complex instruction combinations. The complete project is available at https://sites.google.com/view/hievt . † † footnotetext: ✉ Corresponding authors. Figure 1: Examples of User-Centric embodied visual tracking with diverse instructions. 1 Introduction Providing User-Centric Embodied Visual Tracking (UC-EVT) has become increasingly appealing for modern intelligent agents, as users demand more dynamic and interactive systems beyond fixed targets and distance [ 1 , 2 ] . In particular, users expect systems that can quickly comprehend instructions and respond effectively to dynamic and complex tracking scenarios [ 3 , 4 , 5 ] . Motivated by the above needs, this paper proposes three core requirements for UC-EVT: 1) User Instructions Understanding. The UC-EVT agent must be capable of interpreting user instructions, such as natural language commands, to dynamically adjust the tracking agent, including targets, angles, and distances. 2) Real-Time Responsiveness. The agent must operate in real time, maintaining robust performance by efficiently tracking fast-moving or unseen targets in dynamic environments. 3) Flexibility and Generalizability. The agent should flexibly adapt to various targets, changing environments, diverse instructions, and evolving spatial relationships, all without requiring retraining. Existing methods for embodied visual tracking (EVT) [ 6 , 7 , 8 , 9 ] primarily focus on static or predefined goals, such as maintaining fixed distances or angles relative to a target. Reinforcement learning (RL)-based models, for example, utilize the distance from the target to the expected location (usually at the center of the view) as the reward to train tracking policy in an end-to-end manner [ 6 , 7 , 8 ] or based on vision foundation models [ 9 , 10 ] . Additionally, systems like Vision-Language-Action Models (VLA) and general-purpose large models (e.g., LLMs and VLMs) have demonstrated improved instruction comprehension capabilities due to their large-scale parameterization. Despite these advancements, existing EVT models face significant limitations when applied to UC-EVT tasks. Specifically: 1) Limited Comprehension and Flexibility in RL-based EVT: RL-based EVT models struggle to handle complex user instructions and exhibit poor transferability across different environments and embodiments. Their rigid tracking strategies (e.g., fixed distances or angles) hinder adaptability to user-centric instructions. 2) Limited Generalization of VLA Models: VLA models rely heavily on large-scale annotated data, making them ill-suited for unseen conditions such as novel environments, instructions, targets, or agents. 3) Inference Latency in Large Models: While large models exhibit strong capabilities, their inference speeds (typically 0.5–3 FPS) are insufficient for real-time tracking, resulting in frequent target loss during fast movements. To address these limitations, we propose a novel Hierarchical Instruction-aware Embodied Visual Tracking (HIEVT) agent that integrates instruction comprehension models with adaptive tracking policies and introduces intermediate spatial goals to bridge human instructions and agent behavior. Specifically, HIEVT introduces an LLM-based Semantic-Spatial Goal Aligner , which translates diverse human instructions into spatial goals. This is followed by the RL-Based Adaptive Goal-Aligned Policy , a general offline policy enabling the agent to efficiently approximate the spatial goal and achieve precise tracking. Our contributions are summarized as follows: 1) We introduce the User-Centric Embodied Visual Tracking (UC-EVT) task, which lays the foundation for user-centric human-robot interactions, enabling robots to follow users and provide personalized services. 2) We propose a novel Hierarchical Instruction-aware EVT (HIEVT) model that effectively addresses the limitations of state-of-the-art (SOTA) models while preserving their advantages. 3) We benchmark the UC-EVT task by preparing a large-scale dataset of 10 million annotated trajectories and conducting extensive evaluations across 10 diverse virtual environments. Additionally, we implement RL-based, VLA, and VLM baselines for the UC-EVT task, with all codes and datasets made publicly available. 4) Our extensive experiments across 10 virtual environments and different moving speeds demonstrate that HIEVT outperforms existing baselines significantly. Furthermore, real-world deployments in three different environments validate the robustness and effectiveness of our approach. 2 Preliminaries Problem Definition. User-Centric Embodied Visual Tracking (UC-EVT) starts with an initial user instruction and a target. The tracker then attempts to track the target immediately by following the user’s instructions. The main components of this problem are outlined as follows: • User Instruction. The user initially provides an instruction \mathcal{I}_{0} , which serves as the starting command for the tracking agent. At any subsequent time step t , the user can issue a new instruction \mathcal{I}_{t} to update the agent’s behavior. These instructions can be provided in natural language (e.g., “Move closer”) or by directly specifying a location, such as drawing a bounding box. • Tracking State. The tracking agent’s state at time t , denoted as s_{t} , is represented by the relative distance \rho_{t} and the relative angle \theta_{t} with respect to the target. Specifically, the tracking state is given as: \mathcal{S}_{t}=(\rho_{t},\theta_{t}) . • Agent Action. At each time step t , the tracking agent executes an action a_{t} to adjust its position and orientation in order to minimize the discrepancy between the current tracking state and the user’s instruction. • Environment Transferring. To ensure applicability to real-world scenarios, the tracker should be tested in unseen environments. This setup is designed to test the generalization and transferability of the tracking system. Objective Function. The goal of UC-EVT is to minimize the discrepancy between the user instruction \mathcal{I}_{t} and the tracking state s_{t} throughout the entire tracking period T . This discrepancy is measured using a distance function \mathcal{D}(\mathcal{I}_{t},s_{t}) . The objective function is defined as \min\sum_{t=1}^{T}\mathcal{D}(\mathcal{I}_{t},s_{t}) , where \mathcal{D}(\mathcal{I}_{t},s_{t}) quantifies the difference between the user instruction \mathcal{I}_{t} and the tracking state s_{t} at each time step t . The objective aims to ensure that the tracking agent follows the user’s instructions as closely as possible over the entire tracking period (episode). 3 Hierarchical Instruction-aware Embodied Visual Tracking In this section, we present the design insight of our proposed model, Hierarchical Instruction-aware Embodied Visual Tracking (HIEVT) . The core challenge of the User-Centric Embodied Visual Tracking (UC-EVT) is the giant gap from the user instruction {\mathcal{I}}_{t} and the actual state {\mathcal{S}}_{t} of the tracker. Moreover, the instruction {\mathcal{I}}_{t} is given by a user-friendly mode rather than an agent-friendly mode. To bridge the gap, it is necessary to import an intermediate goal {\mathcal{G}}_{inter}{(t)} to bridge the user’s