Leveraging LLM Agents for Automated Video Game Testing

Chengjia Wang, Lanling Tang, Ming Yuan, Jiongchi Yu, Xiaofei Xie, Jiajun Bu · 2025

Read Leveraging LLM Agents for Automated Video Game Testing by Chengjia Wang, Lanling Tang, Ming Yuan, Jiongchi Yu, Xiaofei Xie, Jiajun Bu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-22170.txt.

Introduction

Testing a massive online game by hand is slow, and older automated testers cover little of a game's vast world. Chengjia Wang and five co-authors present TITAN in this 2025 paper, an agent framework that uses large language models to test such games. TITAN is built in four parts to perceive, prioritize, remember, and judge.

One part abstracts the game's high-dimensional state into a usable form. Another optimizes and ranks the actions worth trying. A third gives long-horizon reasoning, with a memory of past actions and self-correction when plans fail. The fourth uses model-based oracles to flag functional and logic bugs and write diagnostic reports. On two large commercial games, across computer and mobile versions, TITAN reached far higher task completion and bug detection than existing testers. It found four previously unknown bugs, and it has been deployed in eight real quality-assurance pipelines.

Catalog record

CREATORChengjia Wang, Lanling Tang, Ming Yuan, Jiongchi Yu, Xiaofei Xie, Jiajun Bu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-22170

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Leveraging LLM Agents for Automated Video Game Testing Chengjia Wang, Lanling Tang, Ming Yuan, Jiongchi Yu, Xiaofei Xie, Jiajun Bu Abstract Testing MMORPGs (Massively Multiplayer Online Role-Playing Games) is a critical yet labor-intensive task in game development due to their complexity and frequent updating nature. Traditional automated game testing approaches struggle to achieve high state coverage and efficiency in these rich, open-ended environments, while existing LLM-based game-playing approaches are limited to shallow reasoning ability in understanding complex game state-action spaces and long-complex tasks. To address these challenges, we propose TITAN , an effective LLM-driven agent framework for intelligent MMORPG testing. TITAN incorporates four key components to: (1) perceive and abstract high-dimensional game states, (2) proactively optimize and prioritize available actions, (3) enable long-horizon reasoning with action trace memory and reflective self-correction, and (4) employ LLM-based oracles to detect potential functional and logic bugs with diagnostic reports. We implement TITAN and evaluate it on two large-scale commercial MMORPGs spanning both PC and mobile platforms. In our experiments, TITAN achieves significantly higher task completion rates (95%) and bug detection performance compared to existing automated game testing approaches. An ablation study further demonstrates that each core component of TITAN contributes substantially to its overall performance. Notably, TITAN detects four previously unknown bugs that prior testing approaches fail to identify. We provide an in-depth discussion of these results, which offer guidance for new avenues of advancing intelligent, general-purpose testing systems. Moreover, TITAN has been deployed in eight real-world game QA pipelines, underscoring its practical impact as an LLM-driven game testing framework. Keywords Large Language Model, Game Testing, Testing Agents 1 Introduction Ensuring the quality of online games is a monumental challenge in software engineering. Modern MMORPGs feature vast open worlds, nonlinear task actions, myriad interactive NPCs (non-player characters), and continuously evolving gameplay mechanics Claypool and Claypool 2006 . Testing such games thoroughly for bugs is costly, time-consuming, and often incomplete Albaghajati and Ahmed 2020 ; Politowski et al. 2021 . Existing studies Insights 2025 estimate that quality assurance (QA) can consume most of a game’s development budget, with manual testers painstakingly playing through content to find glitches and verify game task logic. Moreover, popular games are updated frequently, often involving hundreds of code changes per day and, in some cases, more than three internal releases in a single day Yu et al. 2023 . Manual playtesting under such conditions is not only expensive and slow, but also inherently subjective and non-scalable. Human testers tend to follow common "happy paths" Anderson et al. 2008 ; Albaghajati and Ahmed 2020 while missing edge-case behaviors, and their effectiveness further declines when they are tasked with repetitive or long-duration scenarios. Scripted test automation Alégroth et al. 2016 simulates player actions through predefined rules but is brittle and costly to maintain, as scripts must be manually crafted for each task and frequently break under game updates. More adaptive and automated DRL-based agents (e.g., Wuji Zheng et al. 2019 ) can learn strategies via exploration, yet they require extensive training, carefully designed rewards, and substantial computational resources, making them impractical for dynamic MMORPGs. Furthermore, DRL agents often overfit to short-term rewards and lack semantic understanding of the game tasks or items, limiting their ability to handle long-horizon objectives or detect subtle logic bugs. The rise of large language models (LLMs) has enabled agents that combine reasoning with actions in interactive environments, as demonstrated by frameworks like LangChain Chase 2022 and ReAct Yao et al. 2023 . While LLM agents show promising zero-shot generalization in text-based games and simple tasks Jeurissen et al. 2024 ; Li et al. 2025 , directly applying them to MMORPGs remains challenging Xu et al. 2024 . ❶ MMORPGs present vast, partially observable states—covering world dynamics, NPCs, and player statistics, which is easy to exceed LLMs’ perception limits due to token and memory constraints. ❷ Their enormous action space (movement, skills, items, dialogues) makes unguided decisions prone to invalid or irrelevant actions. Existing LLM agents Jin et al. 2024 are either domain-specific (e.g., Voyager Wang et al. 2023 for Minecraft) or overly generic, struggling with task logic, long-term progress, and subtle bug detection Kwa et al. 2025 . Consequently, they cannot yet support scalable, efficient testing of industrial-scale MMORPGs. Motivation. In real-world practice, MMORPGs are becoming increasingly larger and more complex, which makes thorough testing of sophisticated tasks even more critical. At the same time, automation is urgently needed to compensate for limited human resources. Current approaches often fail to achieve even correct task execution, let alone sufficient bug coverage and performance evaluation, thereby posing significant challenges for reliable game testing. To bridge this gap, we aim to develop an intelligent game-testing agent that addresses the above limitations. Our intuition is designing an agent that requires no task-specific training, generalizes across different games, and can complete long task sequences within complex game tasks. At the same time, it should balance task completion with maximizing game testing coverage and convenience, while effectively discovering various types of bugs, particularly functionality bugs and logic bugs. To this end, we propose TITAN , an LLM-driven agent framework specifically tailored for automated MMORPG testing. TITAN leverages the powerful reasoning capabilities of foundation LLMs while augmenting them with domain-specific expert knowledge to optimize the testing workflow. It provides enhanced perception, document-based extraction of game information, action recommendation, memory-based reflective reasoning, and human-in-the-loop, LLM-assisted oracles for functionality-targeted bug detection. We implement a prototype of TITAN and evaluate its performance on two large-scale commercial MMORPGs from our cooperating company. These games span both PC and mobile platforms, allowing us to assess the generalization ability and effectiveness of TITAN . We construct a benchmark of 20 tasks of different difficulty levels from the two games, ranging from simple tasks with only a few steps to highly complex tasks (i.e., with over 20 steps), to thoroughly evaluate the performance of existing game-testing methods. We choose baseline approaches that include the DRL-based approach Wuji Zheng et al. 2019 , and the vanilla LLM-agent-based ReAct Yao et al. 2023 for evaluating their performance in task completion and bug detection. In addition, we conduct the same set of testing tasks with three professional internal game testers for cross-comparison. We design a comprehensive experimental study to answer the following three key research questions: • RQ1: Can TITAN complete typical MMORPG testing tasks more effectively and efficiently than existing methods? • RQ2: How does TITAN perform in detecting MMORPG bugs compared to existing approaches? • RQ3: What is the contribution of each core component of TITAN to its overall performance? Our comprehensive evaluation results demonstrate that TITAN achieves significantly higher task completion rates (95%) and bug detection performance (15) compared to automated game testing approaches (best at 82% and 9, respectively). Among all the components, we find that the Reflective Reasoning Module contributes most to the performance of TITAN , while all components are indispensable to the overall framework. Notably, TITAN detects bugs that are not identifiable by previous testing approaches and successfully uncovered four new bugs during our study, including new bug types such as Model Logic Bug and Hang Interaction Bug. TITAN has been deployed in 8 real-world game QA pipelines, demonstrating the practical impact of LLM-driven game testing.

Read the complete text