VLM Can Be a Good Assistant: Enhancing Embodied Visual Tracking with Self-Improving Vision-Language Models

Kui Wu, Shuhang Xu, Hao Chen, Churan Wang, Zhoujun Li, Yizhou Wang, Fangwei Zhong · 2025

Read VLM Can Be a Good Assistant: Enhancing Embodied Visual Tracking with Self-Improving Vision-Language Models by Kui Wu, Shuhang Xu, Hao Chen, Churan Wang, Zhoujun Li, Yizhou Wang, Fangwei Zhong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20718.txt.

Introduction

Kui Wu and co-authors wrote this 2025 paper on giving robot trackers a thoughtful assistant for when they lose their target. Active visual tracking fails in cluttered, shifting scenes, and recovery is its weak point. Their design pairs a fast, ordinary tracking policy for the good times with a vision-language model that wakes only when failure is detected.

The model reasons about where the target may have gone and coaches the search. A memory of past episodes lets it reflect and improve its own recovery advice over time, patching the shaky three-dimensional sense of such models. Success rates jumped 72 percent over a strong reinforcement-learning tracker and 220 percent over a PID-based one in hard environments. The authors present failure recovery, not raw tracking, as the place language smarts pay off first.

Catalog record

CREATORKui Wu, Shuhang Xu, Hao Chen, Churan Wang, Zhoujun Li, Yizhou Wang, Fangwei Zhong
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-20718

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VLM Can Be a Good Assistant: Enhancing Embodied Visual Tracking with Self-Improving Vision-Language Models Kui Wu, Shuhang Xu, Hao Chen, Churan Wang, Zhoujun Li, Yizhou Wang, Fangwei Zhong Abstract We introduce a novel self-improving framework that enhances Embodied Visual Tracking (EVT) with Vision-Language Models (VLMs) to address the limitations of current active visual tracking systems in recovering from tracking failure. Our approach combines the off-the-shelf active tracking methods with VLMs’ reasoning capabilities, deploying a fast visual policy for normal tracking and activating VLM reasoning only upon failure detection. The framework features a memory-augmented self-reflection mechanism that enables the VLM to progressively improve by learning from past experiences, effectively addressing VLMs’ limitations in 3D spatial reasoning. Experimental results demonstrate significant performance improvements, with our framework boosting success rates by 72 % 72\% with state-of-the-art RL-based approaches and 220 % 220\% with PID-based methods in challenging environments. This work represents the first integration of VLM-based reasoning to assist EVT agents in proactive failure recovery, offering substantial advances for real-world robotic applications that require continuous target monitoring in dynamic, unstructured environments. Project website: https://sites.google.com/view/evt-recovery-assistant . I INTRODUCTION Embodied Visual Tracking (EVT) is a critical task for embodied AI, requiring agents to track dynamic targets while navigating through unstructured environments. Unlike traditional visual tracking tasks [ 13 ] , EVT requires agents to not only understand their surroundings but also to control their movements and camera angles to continuously monitor a target in an ever-changing context. This capability forms the foundation for numerous real-world robotic applications, such as social navigation and person-following robots [ 5 , 22 , 11 ] , which must maintain awareness of a target human in dynamic environments, assistive robots that shadow users while avoiding obstacles. While recent advancements in EVT, primarily focused on reinforcement learning (RL) methods [ 6 , 31 , 7 , 8 ] , have achieved significant progress in maintaining stable tracking in dynamic scenarios, many existing models still face notable limitations. These models typically rely on temporal information and learned policies to handle short-term occlusions or brief losses of target visibility. However, they struggle with more complex situations, such as completely losing track of the target for extended periods, where existing strategies often prematurely declare failure. Furthermore, these methods lack the ability to proactively search for and recover targets after prolonged loss, severely limiting their applicability in complex real-world environments. Concurrently, the field has witnessed remarkable advances in Vision-Language Models (VLMs) [ 25 ] , which have demonstrated exceptional cross-domain generalization capabilities and powerful scene-reasoning abilities. However, these models have notable limitations in robotics applications. Most VLMs are primarily trained on static images or passive video understanding without action capabilities, resulting in limited spatial reasoning [ 2 ] and navigation planning [ 24 ] in complex 3D environments. Despite these constraints, recent VLMs like GPT-4V have shown impressive zero-shot performance and in-context learning ability in understanding complex visual scenes, which presents a promising direction for addressing the limitations of current EVT approaches. Fig. 1: Comparison of tracking capabilities between traditional tracking method and our proposed VLM-enhanced Embodied Visual Tracking approach. When the target is occluded by obstacles (boxes and pillars), the traditional method may lose track and fail ( Red ), while our VLM-enhanced approach analyzes the target’s trajectory, and surrounding environment, and actively attempts to recover the target by reasoning about its possible location behind the pillar ( Green ). To bridge this gap, we propose a novel self-improving framework that integrates a Vision-Language Model (VLM) to assist EVT agents in recovering from tracking failures. Our approach combines the off-the-shelf traditional tracking algorithms with the powerful reasoning capabilities of VLMs, creating a complementary system that leverages each component’s strengths while mitigating their individual weaknesses. Specifically, we employ an RL-based visual tracking policy [ 29 ] to follow the target in nominal conditions and only activate the more computationally intensive VLM reasoning when failures are detected. The system identifies target loss through a reliable segmentation-based detection mechanism that continuously monitors the target’s presence. Upon failure detection, the VLM analyzes the historical observation sequence leading to the failure, interprets the environmental context, and generates a structured sequence of recovery actions as movement suggestions. Most importantly, to overcome the inherent limitations of VLMs in 3D spatial reasoning and to optimize recovery strategies for unknown failure cases, we introduce a memory-augmented self-reflection mechanism that enables the VLM to progressively self-improve by analyzing, storing, and learning from its historical experiences. This approach creates a continuously evolving recovery capability that becomes increasingly effective at handling complex tracking failures over time. Our contributions can be summarized in four-fold: 1) We are the first to introduce a VLM-based reasoner to assist the embodied visual tracking agents to proactively recover from failure cases. 2) We propose a novel memory-augmented self-reflection mechanism that enables agents to improve through accumulated experience, effectively addressing VLMs’ inherent limitations in 3D spatial reasoning. 3) Our VLM-enhanced framework significantly boosts tracking performance across different base policies, increasing success rates by 72\% with state-of-the-art RL-based approaches and 220\% with PID-based methods in challenging environments. 4) We also conduct comprehensive experiments to analyze the effectiveness of each module in our proposed framework, providing insights into the complementary nature of our memory management and reflection mechanisms and their collective impact on performance. II Related works II-A Embodied Visual Tracking in Complex Environments Visual tracking is a fundamental capability for mobile robots operating in dynamic environments, requiring systems to actively control camera views and maintain awareness of surroundings while following targets [ 6 , 7 ] . Previous research in this domain has approached the challenge from several perspectives. Traditional approaches relied primarily on geometric methods [ 4 , 18 ] and filter-based tracking techniques [ 21 ] to maintain target awareness. While effective in controlled environments, these methods often struggled with occlusions and dynamic scenes. Building on these foundations, specialized person-following robots [ 14 , 19 ] emerged, predictive modeling to maintain line-of-sight tracking. More recently, reinforcement learning (RL) based approaches have made substantial advances in tracking robustness [ 17 , 31 ] . Researchers further employ spatial-temporal neural networks and adversarial mechanisms to handle short-term occlusions and distractions [ 28 , 27 ] . Advanced state representation and RL strategies [ 29 ] have further improved training efficiency and enabled short-term trajectory prediction for brief occlusion handling. Despite these improvements, existing robotic systems still struggle to proactively recover from significant tracking failures, particularly in scenarios involving complex environmental structures. In this work, we propose a self-improving framework that integrates a Vision-Language Model (VLM) to assist visual tracking agents in recovering from such failures. II-B Vision-Language Model for Embodied Vision Vision-Language Models (VLMs) [ 25 ] have advanced embodied AI through their extensive domain knowledge and multi-modal understanding. Researchers have leveraged VLMs in several ways to enhance robots. Zero-shot approaches [ 1 , 16 , 10 ] apply VLMs directly to tasks like visual question answering (VQA), utilizing the intrinsic image-text relations for more detailed caption. However, VLMs often lack accurate spatial reasoning abilities [ 2 ] , which limits their effectiveness in tasks requiring precise 3D awareness. To address this, researchers have explored several ways to integrate VLM into the robotic

Read the complete text