X-Ego: Acquiring Team-Level Tactical Situational Awareness via Cross-Egocentric Contrastive Video Representation Learning 1103

Yunzhe Wang, Soham Hans, Volkan Ustun · 2025

Read X-Ego: Acquiring Team-Level Tactical Situational Awareness via Cross-Egocentric Contrastive Video Representation Learning 1103 by Yunzhe Wang, Soham Hans, Volkan Ustun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19150.txt.

Introduction

Yunzhe Wang, Soham Hans, and Volkan Ustun build tools for team-level tactical awareness from first-person game video. Good team play grows from each player's own view plus the ability to guess what teammates intend, yet most video research uses third-person broadcast views and misses that shared first-person structure.

Their benchmark holds many hours of Counter-Strike 2 footage from professional-level matches, with synchronized first-person streams for every player plus records of states and actions. A cross-egocentric learning method then builds representations across those views. The dataset gives multi-agent research a realistic test bed in a complex three-dimensional setting.

Catalog record

CREATORYunzhe Wang, Soham Hans, Volkan Ustun
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-19150

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

X-Ego: Acquiring Team-Level Tactical Situational Awareness via Cross-Egocentric Contrastive Video Representation Learning 1103 Yunzhe Wang, Soham Hans, Volkan Ustun Abstract . Human team tactics emerge from each player’s individual perspective and their ability to anticipate, interpret, and adapt to teammates’s intentions. While advances in video understanding have improved the modeling of team interactions in sports, most existing work relies on third-person broadcast views and overlooks the synchronous, egocentric nature of multi-agent learning. We introduce X-Ego-CS , a benchmark dataset consisting of 124 hours of gameplay footage from 45 professional-level matches of the popular e-sports game Counter-Strike 2 , designed to facilitate research on multi-agent decision-making in complex 3D environments. X-Ego-CS provides cross-egocentric video streams that synchronously capture all players’ first-person perspectives, along with state-action trajectories. Building on this resource, we propose Cross-Ego Contrastive Learning (CECL) , which aligns teammates’ egocentric visual streams to foster team-level tactical situational awareness from an individual’s perspective. We evaluate CECL on a teammate-opponent location prediction task, demonstrating its effectiveness in enhancing agent’s ability to infer both teammate and opponent positions from a single first-person view on state-of-the-art video encoders. Together, X-Ego-CS and CECL establish a foundation for cross-egocentric multi-agent benchmarking in esports. More broadly, our work positions gameplay understanding as a testbed for multi-agent modeling and tactical learning, with implications for spatiotemporal reasoning and human-AI teaming in both virtual and real-world domains. Code and dataset are available at https://github.com/HATS-ICT/x-ego 1. Introduction † † Preprint. Under review. Figure 1 . Illustration of Cross-Ego Contrastive Learning (CECL), where teammates’ egocentric video representations are aligned across teams. Team tactics are a defining feature of many human activities, from competitive sports to defense operations, emergency response, and cooperative games. Understanding and modeling such tactics requires reasoning not only about high-level joint actions and spatiotemporal coordination, but also about each agent’s capacity to infer and anticipate the beliefs, intentions, and reactions of both teammates and opponents—a capability often referred to as Computational Theory of Mind ( Rabinowitz et al. 2018 ; Baker et al. 2011 ; Jara-Ettinger et al. 2016 ; Gurney et al. 2021 ) . Early computational approaches to teamwork primarily focused on learning joint policies within simulated Multi-Agent Reinforcement Learning (MARL) frameworks ( Lowe et al. 2017 ; Leibo et al. 2017 ; Yang et al. 2018 ; Rashid et al. 2020 ; Vinyals et al. 2019 ; Berner et al. 2019 ) . However, these methods face two major limitations: (1) many simulation environments are grid-based or two-dimensional, simplifying real-world complexity; even in large-scale domains such as StarCraft II ( Vinyals et al. 2019 ) and Dota 2 ( Berner et al. 2019 ) , the worlds are presented through isometric or “pseudo-3D” perspectives rather than fully three-dimensional environments; (2) learning architectures seldom include explicit models of other agents’ internal states. Consequently, while such systems can coordinate actions, they lack the ability to interpret or anticipate teammates’ and opponents’ mental states. This gap limits their scalability and transferability to real-world scenarios, which is essential for advancing the computational modeling of human team tactics. In recent years, there has been growing interest in extending team-level behavior modeling to real-world competitive sports domains, particularly in soccer and basketball ( Wang et al. 2024 ; Deliege et al. 2021 ; Rao et al. 2024 ; Gu et al. 2020 ; Decroos et al. 2018 ) for tasks such as action recognition, event detection, tactical planning and discovery, and automated commentary generation. This trend is driven both by the complexity of these sports and by the economic and analytical value of understanding team strategies and player interactions. However, existing datasets and models still fall short of enabling true multi-agent understanding since most datasets rely on third-person broadcast footage from fixed angles with global visibility, which limits the study of individual perception, uncertainty, and coordination from a first-person perspective. In contrast, First-Person Shooter (FPS) competitive video games such as Counter-Strike provide a natural testbed and an effective middle ground that balances game-state richness and decision-making complexity, while offering reliable ground-truth data for studying team behavior under partial observability. The game provides a cooperative yet adversarial environment on fixed 3D maps, where players must continuously balance tactical risk, spatiotemporal dynamics, and coordination. While relatively underexplored, prior work has begun to investigate this space: ( Xenopoulos and Silva 2022 ) introduced the ESTA dataset along with the AWPY parser for extracting ground-truth state-action trajectories and game events, ( Wang et al. 2025 ) developed the DECOY simulator to support multi-agent reinforcement learning, ( Durst et al. 2024 ) trained a transformer-based control model to mimic professional player movement, and ( Fanourakis and Chanel 2025 ) studies players’ emotional dynamics with synchronized physiological, behavioral, and gameplay signals. While these efforts highlight the promise of the domain, they largely focus on individual agents or global game states, lacking a systematic approach to modeling the synchronous, egocentric nature of multi-agent team interactions. To address this gap, we introduce X-Ego-CS , a benchmark dataset comprising 124 hours of gameplay footage from 45 professional-level Counter-Strike 2 matches, containing synchronized egocentric video streams from all players. Unlike prior datasets focused on single perspectives or third-person views, X-Ego-CS captures simultaneous first-person recordings paired with state-action trajectories sampled at 64 ticks per second. We define this multi-perspective video paradigm as cross-egocentric , combining the egocentric focus of individual perception with cross-view synchronization across teammates. Building upon X-Ego-CS, we propose Cross-Ego Contrastive Learning (CECL) , a method designed to learn shared world-state representations by aligning teammates’ egocentric visual streams at corresponding timesteps. This alignment encourages models to internalize a collective understanding of team context, which is an essential capability for agents aiming to exhibit theory-of-mind -like reasoning and collaborative tactical awareness. We evaluate CECL on a downstream teammate-opponent location prediction task, demonstrating its effectiveness in enhancing agents’ ability to infer both teammate and opponent positions from a single first-person view when applied to state-of-the-art video encoders. These results highlight CECL’s potential to foster team-level situational awareness and establish cross-egocentric multi-agent video understanding as a promising direction for advancing tactical reasoning and human-AI teaming. Our contributions are twofold: • We introduce X-Ego-CS, the first benchmark dataset for cross-egocentric multi-agent video understanding in esports, and established teammate-opponent location prediction task for teammate and opponent modeling. • We propose CECL, a visual contrastive learning method that learns shared team-state representations by aligning entire teams’ first-person views at the same timestep, boosting individual agents’ team-level situational awareness. 2. Related Work Sports Understanding Computational analysis of team sports has traditionally focused on third-person broadcast views, particularly in soccer and basketball, addressing tasks such as action recognition ( Giancola et al. 2018 ; Deliege et al. 2021 ) , tactical planning ( Wang et al. 2024 ) , and automated commentary ( Mkhallati et al. 2023 ; Rao et al. 2024 ; Rao et al. 2025 ) . While these approaches have advanced strategic analysis and performance analytics, they rely on omniscient overhead perspectives that do not capture the egocentric, partially observable nature of real-time decision-making that players experience. Gameplay Understanding Research in gameplay understanding within AI can be broadly organized into two categories: (1) enabling AI agents to play games autonomously, and (2) facilitating effective collaboration between

Read the complete text