CoMaTrack: Competitive Multi-Agent Game-Theoretic Tracking with Vision-Language-Action Models
Youzhi Liu, Li Gao, Liu Liu, Mingyang Lv, Yang Cai · 2026
Read CoMaTrack: Competitive Multi-Agent Game-Theoretic Tracking with Vision-Language-Action Models by Youzhi Liu, Li Gao, Liu Liu, Mingyang Lv, Yang Cai (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-22846.txt.
Introduction
Liu and co-authors present CoMaTrack, a framework for embodied visual tracking with language goals. One agent must follow a target named in words. Single-agent imitation needs costly expert data and adapts poorly. CoMaTrack trains agents in a competitive multi-agent game instead. A tracker faces adaptive opponents across varied scenes and instructions. The team also built a Habitat-based benchmark for such duels. Their tracker beats prior methods on standard tests and the new benchmark. A small three-billion-parameter model beats larger imitation-based rivals.
Catalog record
| CREATOR | Youzhi Liu, Li Gao, Liu Liu, Mingyang Lv, Yang Cai |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37355 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.22846
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CoMaTrack: Competitive Multi-Agent Game-Theoretic Tracking with Vision-Language-Action Models Youzhi Liu, Li Gao, Liu Liu, Mingyang Lv, Yang Cai Abstract Embodied Visual Tracking (EVT), a core dynamic task in embodied intelligence, requires an agent to precisely follow a language-specified target. Yet most existing methods rely on single-agent imitation learning, suffering from costly expert data and limited generalization due to static training environments. Inspired by competition-driven capability evolution, we propose CoMaTrack, a competitive game-theoretic multi-agent reinforcement learning framework that trains agents in a dynamic adversarial setting with competitive subtasks, yielding stronger adaptive planning and interference-resilient strategies. We further introduce CoMaTrack-Bench, the first open-source Habitat-based benchmark protocol and episode set for language-conditioned competitive EVT featuring dynamic dueling, featuring game scenarios between a tracker and adaptive opponents across diverse environments and instructions, enabling standardized robustness evaluation under active adversarial interactions. Experiments show that CoMaTrack achieves state-of-the-art results on both standard benchmarks and CoMaTrack-Bench. Notably, a 3B VLM trained with our framework surpasses previous single-agent imitation learning methods based on 7B models on the challenging EVT-Bench, achieving 92.1% in STT, 74.2% in DT, and 57.5% in AT. The benchmark code will be available at https://github.com/wlqcode/CoMaTrack-Bench Figure 1 : CoMaTrack frames EVT as a competitive multi-agent game rather than a single-agent pursuit in a static enviroment. (a) IL: the agent learns from offline demonstrations in static scenes, with limited exposure to rare failures. (b) Single-Agent RL: the agent improves through interaction, but the target and environment remain largely fixed, leading to slow exploration and overfitting to predefined behaviors. (c) Multi-Agent RL: the agent trains against adaptive opponents that evade or block on purpose, dynamically increasing difficulty and producing diverse adversarial trajectories, encouraging anticipation, relocalization, and robustness under interference. 1 Introduction In recent years, embodied agents driven by Large Language Models (LLMs) [ 1 , 29 , 6 , 52 , 22 , 24 ] have made remarkable progress, allowing perception, reasoning, and decision-making in complex 3D environments and moving from merely understanding instructions to executing tasks. Among them, Embodied Visual Tracking (EVT) [ 70 , 39 ] is a key embodied task that requires an agent to continuously follow a dynamic target from egocentric observations, while operating under environmental uncertainty, changing target behaviors, frequent occlusions, and partial observability over long horizons. Unlike static recognition or short-range navigation, EVT emphasizes persistent closed-loop control and online error correction: the agent must not only see the target, but also maintain identity consistency when the target moves, becomes occluded, is interrupted by distractors, or is confused with similar instances, and meanwhile take proper actions for pursuit and re-localization. As such, the task inherently demands target understanding, spatiotemporal reasoning, motion planning, memory maintenance, and robustness to interference—making it a critical stepping stone towards general embodied intelligence and real-world robotic applications. However, existing research remains largely in a single-agent setting and is predominantly trained through Imitation Learning (IL) [ 66 , 14 , 55 , 17 , 37 ] , leading to limited generalization and robustness in out-of-distribution scenes. A central limitation of single-agent IL is its strong dependence on high-quality expert trajectories: data collection and annotation are expensive, and the task distribution is manually designed, making it difficult to cover the rich variations encountered in the real world [ 40 , 8 , 15 ] .As a result, agents trained on such a finite problem set often learn to fit the training distribution rather than acquiring transferable strategies [ 71 ] . More importantly, although reinforcement learning (RL) [ 11 , 53 , 30 , 73 ] provides closed-loop interaction and exploration, and in principle can yield more general policies through trial-and-error [ 67 ] , EVT still lacks systematic RL studies and effective training paradigms. Conventional RL training [ 50 , 51 , 23 ] typically relies on relatively static environments and fixed target generators, which struggle to continuously produce hard enough and progressively more challenging supervision signals. Without an automatically escalating learning pressure, agents are prone to inefficient exploration and overfitting, resulting in limited sample efficiency and generalization gains. In other words, even with RL, if the environment itself does not become stronger, the agent is unlikely to be forced to develop robust long-horizon planning, counter-planning behaviors. We observe that in humans and animals, capabilities often evolve rapidly through sustained competition [ 74 ] : changes in opponents’ strategies continuously raise task difficulty, forcing individuals to improve anticipation, planning, and counteraction [ 41 ] . Inspired by this, we introduce CoMaTrack, a competitive multi-agent game-theoretic learning mechanism into EVT and build a co-evolving training loop among agent–opponent–environment. The agent no longer faces a static environment and a fixed data distribution; instead, it repeatedly encounters stronger, more cunning, and more diverse behaviors through adversarial interactions, yielding an automatic curriculum with progressive difficulty. Under this framework, the policies of the other agents directly reshape the training distribution—effectively making opponents the environment. Any improvement by one agent becomes a new challenge that others must overcome, ultimately forming a self-reinforcing arms race. Different from conventional multi-agent cooperation that primarily optimizes system-level efficiency, we focus on sharpening the core competence of a single tracking agent via competitive games, enabling robust behaviors under typical tracking difficulties, and improving transferability to unseen environments and target behaviors. As shown in Fig. 1 , we shift the paradigm from static data-driven training to adversarial interaction-driven learning. Critically, existing EVT benchmarks ( e.g . , EVT-Bench [ 55 ] , TPT-Bench [ 60 ] , Gym-UnrealCV [ 47 ] ) exacerbate this limitation: they evaluate tracking under idealized or passively disturbed conditions, lacking scenarios with active adversarial competition that mirror real-world challenges like intentional evasion, strategic deception, or contested pursuit. To bridge this gap, we introduce CoMaTrack-Bench, the first open-source benchmark for competitive EVT, featuring dynamic dueling scenarios between a tracker agent and adaptive opponent across diverse environments and language instructions. This benchmark establishes a standardized protocol for rigorously assessing robustness under intentional adversarial interactions—moving beyond the static obstacle paradigm of prior evaluations. Our main contributions are as follows: • First integration of competitive multi-agent game-theoretic RL with modern VLA architectures for language-conditioned embodied visual tracking: We devise a co-evolving adversarial training loop where opponent strategies dynamically escalate task difficulty. Crucially, a compact 3B-parameter VLM trained under CoMaTrack surpasses all existing single-agent methods leveraging 7B-parameter models, confirming that strategic competition—not model scale—drives robust generalization. • Open-sourcing CoMaTrack-Bench: We release the first open-source benchmark suite that standardizes competitive EVT evaluation in Habitat with language instructions, featuring dynamic dueling scenarios with adaptive opponents to enable rigorous, real-world-aligned evaluation of tracking robustness—addressing the idealized limitations of prior benchmarks. • Extensibility to broader VLA embodied tasks: While this paper focuses on tracking, our approach offers a general paradigm of opponent-driven difficulty generation and competitive-game-driven generalization, and can be naturally extended to other VLA embodied tasks to address out-of-distribution generalization. 2 Related Work 2.1 Visual Language Navigation Recent VLN [ 3 , 12 , 2 , 37 ] research has increasingly shifted from modular pipelines to Vision-Language-Action (VLA) [ 8 , 25 , 42 ] that map egocentric observations and language instructions to actions in continuous environments. Uni-NaVid [ 66 ] exemplifies this trend by extending video VLMs with an online token merging mechanism