F.A.C.U.L.: Language-Based Interaction with AI Companions in Gaming
Wenya Wei, Sipeng Yang, Qixian Zhou, Ruochen Liu, Xuelei Zhang, Yifu Yuan, Yan Jiang, Yongle Luo, Hailong Wang, Tianzhou Wang, Peipei Jin, Wangtong Liu, Zhou Zhao, Xiaogang Jin, Elvis S. Liu · 2025
Read F.A.C.U.L.: Language-Based Interaction with AI Companions in Gaming by Wenya Wei, Sipeng Yang, Qixian Zhou, Ruochen Liu, Xuelei Zhang, Yifu Yuan, Yan Jiang, Yongle Luo, Hailong Wang, Tianzhou Wang, Peipei Jin, Wangtong Liu, Zhou Zhao, Xiaogang Jin, Elvis S. Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-13112.txt.
Introduction
Wenya Wei and fourteen co-authors let players talk to their AI teammates. In cooperative shooters, companions are usually steered by hotkeys or command wheels offering a few canned orders, with no way to name a target or a place. F.A.C.U.L. is, its authors say, the first real-time system where a player instructs an FPS companion in natural language. It decomposes complex commands, gauges its confidence in the interpretation, and retrieves the relevant entities from the environment so spoken orders can be carried out.
The companion answers with behavior and voice feedback, keeping the collaboration flowing. Using the commercial game Arena Breakout: Infinite as the testbed, the team compares the system against conventional control and reports user feedback on its strengths and limits. It targets immersion: a teammate you can brief like a person.
Catalog record
| CREATOR | Wenya Wei, Sipeng Yang, Qixian Zhou, Ruochen Liu, Xuelei Zhang, Yifu Yuan, Yan Jiang, Yongle Luo, Hailong Wang, Tianzhou Wang, Peipei Jin, Wangtong Liu, Zhou Zhao, Xiaogang Jin, Elvis S. Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-13112
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
F.A.C.U.L.: Language-Based Interaction with AI Companions in Gaming Wenya Wei, Sipeng Yang, Qixian Zhou, Ruochen Liu, Xuelei Zhang, Yifu Yuan, Yan Jiang, Yongle Luo, Hailong Wang, Tianzhou Wang, Peipei Jin, Wangtong Liu, Zhou Zhao, Xiaogang Jin, Elvis S. Liu Abstract In cooperative video games, traditional AI companions are deployed to assist players, who control them using hotkeys or command wheels to issue predefined commands such as “attack”, “defend”, or “retreat”. Despite their simplicity, these methods, which lack target specificity, limit players’ ability to give complex tactical instructions and hinder immersive gameplay experiences. To address this problem, we propose the FPS AI Companion who Understands Language (F.A.C.U.L.), the first real-time AI system that enables players to communicate and collaborate with AI companions using natural language. By integrating natural language processing with a confidence-based framework, F.A.C.U.L. efficiently decomposes complex commands and interprets player intent. It also employs a dynamic entity retrieval method for environmental awareness, aligning human intentions with decision-making. Unlike traditional rule-based systems, our method supports real-time language interactions, enabling players to issue complex commands such as “clear the second floor”, “take cover behind that tree”, or “retreat to the river”. The system provides real-time behavioral responses and vocal feedback, ensuring seamless tactical collaboration. Using the popular FPS game Arena Breakout: Infinite as a case study, we present comparisons demonstrating the efficacy of our approach and discuss the advantages and limitations of AI companions based on real-world user feedback. Homepage — https://sites.google.com/view/FACUL-demo Introduction Over the past few decades, video games have attracted billions of players, with new techniques continuously introduced to enhance their appeal. In particular, recent breakthroughs in AI have enabled the integration of advanced capabilities in games, such as those seen in Go ( Silver et al. 2016 ) and real-time strategy games like StarCraft II ( Vinyals et al. 2019 ) , where AI has reached expert-level performance. However, most AI research in gaming focuses on competition with players, while the exploration of AI companions for cooperative gameplay has been largely overlooked ( Gao et al. 2023 ) . In First-Person Shooter (FPS) games, players’ enjoyment often depends on effective teamwork, making cooperation crucial for gameplay enhancement and immersion. When it is challenging to find human teammates online, or when players prefer to avoid teaming up with others, the development of AI companions becomes crucial. Such companions must be capable of comprehending player instructions and delivering reliable, real-time assistance. Figure 1: F.A.C.U.L. is the first open-interaction and real-time language-operated AI companion system for commercial video games. For AI companions, automated goal execution is relatively straightforward to implement, requiring only predefined objectives, paths, and action sequences. However, the methods for players to communicate with and command AI companions remain limited and simplistic. Existing interaction approaches, such as hotkeys and command wheels in FPS games, allow players to issue basic commands like “attack” and “retreat”. These commands are overly simplistic and lack target specificity, which prevents players from issuing more complex tactical instructions, such as “clear the second floor” and “take cover behind that tree”, hindering human-AI collaboration and reducing immersion. Intuitively, language is the most natural and immersive form of interaction, making it highly appealing to collaborate with AI companions in games. Although recent advances in powerful large language models (LLMs) ( Yang et al. 2024 ; Zhang et al. 2024a ) have enabled this capability, integrating them into real-world games for effective use presents several challenges. First, the framework must operate with minimal latency to deliver real-time feedback to players, while ensuring high concurrency and low operational costs on the server side. Second, the language model must accurately understand player commands within the specific game context, translating and decomposing commands into actions, objects, and other relevant elements. Finally, AI companions must precisely identify the objects in the game environment that correspond to the player’s commands to execute them effectively. In this paper, we propose F.A.C.U.L., the first language-based AI companion framework for FPS video games. F.A.C.U.L. enables natural language interaction with players, supports tactical collaboration, and understands the game environment. The framework comprises three key components: instruction reasoning, environmental perception, and decision execution. To achieve high-throughput language instruction processing, we introduce a confidence-based method that combines a lightweight model, BERT-FID (BERT for Fuzzy Intent Detection), with a larger LLM. BERT-FID quickly decomposes commands into action–target templates and provides both reasoning results and confidence scores. High-confidence (simple) commands are executed directly, while low-confidence (complex) ones are routed to the LLM for deeper interpretation. This design reduces computational overhead for routine commands while preserving accuracy for ambiguous instructions, balancing efficiency and comprehension. Next, a multimodal entity retrieval module matches player commands to real-time game assets, ensuring robust environmental awareness. Finally, commands are translated into atomic actions and merged with environmental data to generate the AI companion’s responses, which are executed using behavior trees with real-time feedback to the player. Using the popular FPS game Arena Breakout: Infinite ( Tencent 2024 ) as a case study, we demonstrate the application of F.A.C.U.L., offering an immersive human-AI collaboration experience and supporting a wide range of open-ended commands within the game. Fig. 1 illustrates the performance of our algorithm in task completion. A full match visualization video is available on our project page. We summarize the contributions of our work as follows: • F.A.C.U.L., the first FPS AI companion that understands human language, offering natural language interaction and contextual environment recognition for commercial video games. • BERT-FID, a command decomposition and confidence-based model for fuzzy intent detection, enhancing both the speed and accuracy of command processing. • Developed and evaluated in a real-world FPS game, showing high reliability and strong player adoption, with a comprehensive analysis of AI companions through player feedback. Figure 2: Overview of F.A.C.U.L.. Players can cooperate with the agent in combat and freely speak into the microphone and get behavior response with voice feedback. Related Work LLMs in gaming have recently garnered significant attention ( Gallotta et al. 2024 ; Xu et al. 2024 ; Park et al. 2023 ) , particularly in the development of non-player characters (NPCs) and player assistance. For NPCs, LLMs are leveraged to enhance character realism and interactivity. For instance, Shao et al. ( Shao et al. 2023 ) propose the Character-LLM, a framework that trains LLMs to simulate specific characters, thereby enriching role-playing experiences. Similarly, the integration of LLMs in 3D games enables players to collaborate with LLM-powered NPCs to complete tasks ( Rao et al. 2024 ) . Another promising direction is the use of LLMs to generate dynamic, interactive game content. A notable application is the text-to-game engine ( Zhang et al. 2024b ) , which transforms text inputs into dynamic RPGs, generating content such as storylines and game mechanics in real-time. A recent commercial game, Naraka ( NetEast and Entertainment 2020 ) , adopts a complex AI companion system, which is close to our work. However, it focuses primarily on language communication and behavioral interactions with players, while neglecting the ability to effectively understand real-time game states and identify relevant objects within these environments. This limitation restricts its ability to assist players in task completion and immersion. In contrast, our work addresses this gap by integrating instruction reasoning, scene recognition,