CSAOT: Cooperative Multi-Agent System for Active Object Tracking Conference: Proc. of the 24th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2025); May 19 – 23, 2025; Detroit, Michigan, USA 1294
Hy Nguyen*, Bao Pham*, Hung Du, Srikanth Thudumu, Rajesh Vasa, Kon Mouzakis · 2025
Read CSAOT: Cooperative Multi-Agent System for Active Object Tracking Conference: Proc. of the 24th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2025); May 19 – 23, 2025; Detroit, Michigan, USA 1294 by Hy Nguyen*, Bao Pham*, Hung Du, Srikanth Thudumu, Rajesh Vasa, Kon Mouzakis (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-13994.txt.
Introduction
Hy Nguyen and five co-authors build a team of cooperating agents, running on one device, that keeps a moving target in view, in a conference paper from twenty twenty-five. Active tracking differs from passive watching: the camera itself must move to follow the target through cluttered scenes. Single agents gather too little information and falter when targets dart or hide.
Their system trains several agents together with reinforcement learning under a mixture-of-experts design. Each agent takes a distinct role, and the team coordinates viewpoints without extra hardware. Tests on interactive maps with moving and fixed obstacles showed stronger resistance to occlusion and rapid motion, plus longer tracking runs. The design aims at cheaper, sturdier tracking for navigation, surveillance, and robots.
Catalog record
| CREATOR | Hy Nguyen*, Bao Pham*, Hung Du, Srikanth Thudumu, Rajesh Vasa, Kon Mouzakis |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-13994
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CSAOT: Cooperative Multi-Agent System for Active Object Tracking Conference: Proc. of the 24th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2025); May 19 – 23, 2025; Detroit, Michigan, USA 1294 Hy Nguyen*, Bao Pham*, Hung Du, Srikanth Thudumu, Rajesh Vasa, Kon Mouzakis Abstract . Object Tracking is essential for many computer vision applications, such as autonomous navigation, surveillance, and robotics. Unlike Passive Object Tracking (POT), which relies on static camera viewpoints to detect and track objects across consecutive frames, Active Object Tracking (AOT) requires a controller agent to actively adjust its viewpoint to maintain visual contact with a moving target in complex environments. Existing AOT solutions are predominantly single-agent-based, which struggle in dynamic and complex scenarios due to limited information gathering and processing capabilities, often resulting in suboptimal decision-making. Alleviating these limitations necessitates the development of a multi-agent system where different agents perform distinct roles and collaborate to enhance learning and robustness in dynamic and complex environments. Although some multi-agent approaches exist for AOT, they typically rely on external auxiliary agents, which require additional devices, making them costly. In contrast, we introduce the Collaborative System for Active Object Tracking (CSAOT), a method that leverages multi-agent deep reinforcement learning (MADRL) and a Mixture of Experts (MoE) framework to enable multiple agents to operate on a single device, thereby improving tracking performance and reducing costs. Our approach enhances robustness against occlusions and rapid motion while optimizing camera movements to extend tracking duration. We validated the effectiveness of CSAOT on various interactive maps with dynamic and stationary obstacles. A. El Fallah Seghrouchni, Y. Vorobeychik, S. Das, A. Nowe (eds.) 1. Introduction Object tracking is a fundamental task in computer vision, broadly categorized into passive and active tracking ( Soleimanitaleb and Keyvanrad 2022 ) . Passive Object Tracking (POT) identifies a target object from an input video sequence recorded by a static camera. POT operates without influencing or interacting with the environment or the target object; its sole task is to detect the target object across consecutive video frames. In contrast, Active Object Tracking (AOT) involves an interactive agent that actively adjusts its viewpoint to maintain continuous visual contact with the target object as it moves through the environment. This makes AOT inherently more dynamic than POT, requiring real-time decision-making capabilities to adapt to rapid and often unpredictable changes in the target’s movement, environmental conditions, and potential obstacles. Deep Reinforcement Learning (DRL) is a subset of machine learning that combines reinforcement learning with deep learning techniques to enable agents to make decisions in complex environments. DRL’s strengths lie in its ability to learn optimal policies through trial and error, leveraging large amounts of data and powerful neural networks to approximate value functions ( Nguyen et al. 2023 ) . This adaptability allows DRL to handle high-dimensional state spaces and learn from delayed rewards, making it particularly effective for tasks requiring continuous interaction and decision-making. In the context of Active Object Tracking (AOT), DRL is the most suitable method because it enables agents to dynamically adjust their tracking strategies in real-time, responding to the unpredictable movements of target objects while optimizing tracking accuracy and efficiency. Existing DRL solutions to AOT primarily rely on a single-agent approach, where one agent performs all actions for the tracking task (e.g., detection, navigation, obstacle avoidance) ( Jiang et al. 2019 ) . While these solutions can be effective in static environments, they often limit the agent’s ability to adapt in dynamic environments ( Padakandla 2021 ) . This can be due to challenges, such as occlusions, varying speeds, and uncertain changes in the target object’s trajectory. To alleviate these challenges, distributing actions into the multi-agent system where each agent performs a single action can be a potential solution ( Jiang et al. 2019 ) . In addition, Multi-Agent Deep Reinforcement Learning (MADRL) can be employed to facilitate agent interaction for AOT in dynamic environments ( Padakandla 2021 ) . In MADRL, agents communicate, coordinate and learn from their experiences, improving their decision-making processes and adaptability to the unpredictable behavior of target objects ( Silver et al. 2016 ; Mnih et al. 2015 ) . Note that, there exists both individual goals and collective goals in MADRL. This leads to the challenge of designing a reward function that support the optimal performance of the entire system ( Wang et al. 2023 ) . Current MADRL methods for AOT mainly focus on designing auxiliary agents that operate externally to the main agent. In this setup, the master agent is responsible for the tracking task, while the auxiliary agents gather and transmit supplementary information to enhance the master agent’s decision-making capabilities ( Ning and Xie 2024 ) . Although this approach can improve tracking accuracy, it necessitates the use of additional devices and resources, leading to increased costs and complexity in system deployment. To address these challenges, we propose a role-based approach that enables multiple agents to collaborate within a single device. This innovative design not only reduces costs associated with additional hardware but also minimizes communication overhead. Additionally, we propose a novel mechanism called Mixture of Policy (MoP) to learn a policy for each agent. The MoP employs a gating mechanism to coordinate multiple smaller policy networks, with each policy acting as an expert tailored to handle specific scenarios encountered during the tracking task. This approach significantly reduces inference time, which is crucial given the multiple agents involved in the system, enabling faster decision-making without sacrificing performance. Additionally, the use of expert policies enhances accuracy, as each agent can select the most relevant policy based on the current context. By combining the advantages of the MoP mechanism with a role-based collaboration framework, our method aims to optimize both efficiency and effectiveness in real-time AOT applications. In summary, our contribution to this work includes: • Proposed a novel framework CSAOT for cooperative multi-agent deep reinforcement learning for AOT task on a single device. • Introduced MoP, a novel mechanism to learn policy efficiently. • Adapt the framework to AOT task by applying subtask-based reward functions. • Evaluate the proposed framework on a simulated environment, empirically proving the system’s performance. 2. Related work 2.1. Object tracking task and existing solutions With AOT, its application in real-time environment is broader, hence different approaches have been developed to solve this task. Several studies have leveraged reinforcement learning and multi-camera collaboration to improve active object-tracking performance. Luo et al. ( Luo et al. 2019 ) introduced an end-to-end tracking system that integrates object tracking and camera control into a single reinforcement learning framework. The system demonstrates adaptability in real-world scenarios by training in virtual environments and deploying on physical robots. Similarly, Li et al. ( Li et al. 2020 ) tackled object-tracking challenges using pose-assisted multi-camera collaboration. Their approach allows cameras to share positional data, enabling more accurate tracking in environments with multiple obstructions and complex dynamics. These methods highlight the power of collaborative and learning-based strategies to optimize tracking performance across diverse environments. Another previous sub-field in object tracking focuses on active feature selection and motion detection to enhance the robustness and efficiency of object tracking. Zhang et al. ( Zhang et al. 2013 ) applied active learning techniques to feature selection, ensuring