Cultivating Game Sense for Yourself: Making VLMs Gaming Experts

Wenxuan Lu, Jiangyang He, Zhanqiu Zhang †, Yiwen Guo †, Tianning Zang †, Institute of Information Engineering, Chinese Academy of Sciences, School of Cyber Security, University of Chinese Academy of Sciences, School of Computer Science, Artificial Intelligence, Wuhan University of Technology, LIGHTS · 2025

Read Cultivating Game Sense for Yourself: Making VLMs Gaming Experts by Wenxuan Lu, Jiangyang He, Zhanqiu Zhang †, Yiwen Guo †, Tianning Zang †, Institute of Information Engineering, Chinese Academy of Sciences, School of Cyber Security, University of Chinese Academy of Sciences, School of Computer Science, Artificial Intelligence, Wuhan University of Technology, LIGHTS (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-21263.txt.

Introduction

Wenxuan Lu and co-authors stop asking language models to play games directly and instead let them write the players. Their 2025 paper observes that vision-language models pausing to reason over every frame cannot shoot, dodge, or react at game speed. Fluent play needs reflexes, and deliberation is the opposite of a reflex.

In their GameSense framework, the model acts as developer: it watches task attempts, then builds specialized execution modules, from direct action rules to small trained networks, that handle real-time play. The overseeing model refines those modules from feedback. The system achieved fluent play across action games, first-person shooters, and even Flappy Bird, a first for this agent style. The architecture echoes how humans actually acquire game sense: conscious analysis distilled into automatic skill. For agent design generally, the paper argues for a division of labor between slow teachers and fast students inside one system.

Catalog record

CREATORWenxuan Lu, Jiangyang He, Zhanqiu Zhang †, Yiwen Guo †, Tianning Zang †, Institute of Information Engineering, Chinese Academy of Sciences, School of Cyber Security, University of Chinese Academy of Sciences, School of Computer Science, Artificial Intelligence, Wuhan University of Technology, LIGHTS
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-21263

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Cultivating Game Sense for Yourself: Making VLMs Gaming Experts Wenxuan Lu, Jiangyang He, Zhanqiu Zhang †, Yiwen Guo †, Tianning Zang †, Institute of Information Engineering, Chinese Academy of Sciences, School of Cyber Security, University of Chinese Academy of Sciences, School of Computer Science, Artificial Intelligence, Wuhan University of Technology, LIGHTSPEED, Independent Researcher Abstract Developing agents capable of fluid gameplay in first/third-person games without API access remains a critical challenge in Artificial General Intelligence (AGI). Recent efforts leverage Vision Language Models (VLMs) as direct controllers, frequently pausing the game to analyze screens and plan action through language reasoning. However, this inefficient paradigm fundamentally restricts agents to basic and non-fluent interactions: relying on isolated VLM reasoning for each action makes it impossible to handle tasks requiring high reactivity (e.g., FPS shooting) or dynamic adaptability (e.g., ACT combat). To handle this, we propose a paradigm shift in gameplay agent design: instead of directly controlling gameplay, VLM develops specialized execution modules tailored for tasks like shooting and combat. These modules handle real-time game interactions, elevating VLM to a high-level developer. Building upon this paradigm, we introduce GameSense, a gameplay agent framework where VLM develops task-specific game sense modules by observing task execution and leveraging vision tools and neural network training pipelines. These modules encapsulate action-feedback logic, ranging from direct action rules to neural network-based decisions. Experiments demonstrate that our framework is the first to achieve fluent gameplay in diverse genres, including ACT, FPS, and Flappy Bird, setting a new benchmark for game-playing agents. 1 Introduction 1 1 1 {\dagger} Corresponding authors Developing agents that fluidly play first/third-person games without API access remains a critical challenge in AGI, where complexity mirrors real-world embodied tasks Lu et al. (2024) ; Wang et al. (2024) . Agents must navigate diverse tasks, ranging from combat encounters to environmental navigation, while executing precise real-time actions Hu et al. (2024) . Traditional reinforcement learning (RL) approaches struggle to handle such a broad spectrum of demands due to their limited task generalization de Woillemont et al. (2022) ; Justesen et al. (2019) . Recently, the emergence of Vision Language Models (VLMs) has opened new possibilities in this domain. With their strengths in visual understanding and decision-making, VLMs interact with games purely through visual understanding of game screens. This ability offers a promising direction for developing non-API-dependent gameplay agents Tan et al. (2024) ; Liu et al. (2024a) ; Wang et al. (2023b) . Figure 1: The ‘thinking time’ of direct VLM control becomes a critical vulnerability in real-time games, highlighting the need for a paradigm shift on VLM use: from direct controller to execution module developer Recent VLM-based approaches leverage VLMs as direct game controllers through a pause-and-plan paradigm Tan et al. (2024) ; Chen et al. (2024) : the agent periodically pauses gameplay, using VLM and vision tools (e.g., OCR, segmentation) together to analyze game screens, plan actions and then directly output key-mouse command to control game. However, this paradigm suffers from fundamental limitations: (1) it heavily depends on the game’s support to pause at any moment, which disrupts the gameplay flow and limits its applicability to a narrow range of games that support such interruptions; (2) Requiring VLM reasoning for every action makes it unsuitable for tasks demanding high reactivity (e.g., FPS shooting); (3) VLM outputs simple key-mouse control commands without real-time interactive logic for game environments, making it hard solve tasks demanding dynamic adaptation (e.g., action game combat). These limitations reflect a fundamental mismatch : VLMs excel at time-consuming deliberate reasoning (scene understanding and planning) but struggle with rapid, continuous game interactions requiring millisecond-level responses (shown in Figure 1 ). We observe that most human game actions rarely rely on deliberate reasoning, but rather flows from quick-fire game sense - a set of trained reflexes and patterns developed through practice. This observation suggests a fundamental paradigm shift: Unlike using VLMs to directly control every game actions, we should elevate them to develop task-specific execution modules that can handle real-time interactions autonomously. These specialized modules, developed by VLM, solve specific tasks requiring rapid reactions or frequent environmental interactions. This paradigm shift bridges the VLM’s reasoning with real-time gameplay demands, enabling more versatile game agents. Based on this new paradigm, we present GameSense, a framework that empowers VLMs to develop and optimize task-specific execution modules, termed Game Sense Modules (GSMs). GameSense equips VLMs with essential tools, including vision tools and neural network training pipelines, to create GSMs tailored for diverse gameplay tasks. These modules can range from simple action-feedback loops (e.g., combat patterns based on HP bar monitoring) to complex, learned behaviors (e.g., boss fight strategies optimized through RL). These modules are seamlessly integrated into the gameplay loop: when the agent identifies a specific task, it activates the corresponding module and refines it based on execution feedback. By shifting VLMs’ role from direct controller to the developer of GSMs, GameSense achieves efficient execution and promotes continuous improvement in gameplay performance. Experiments demonstrate that GameSense is the first agent to achieve fluent gameplay in diverse game genres. In ACT/FPS games, our framework achieves the highest success rates in combat tasks, while achieving the highest exploration scores without gameplay pausing. In contrast, existing VLM-based methods either fail to complete such tasks or rely heavily on frequent gameplay pausing, disrupting the flow of real-time interactions. In the reflex-intensive game Flappy Bird where pausing is not supported, existing VLM-based methods fail at basic control, and GameSense develops precise control modules through iterative refinement. GameSense exhibits significantly improved real-time performance and adaptation capabilities, setting a new benchmark for game-playing agents. The contributions of this paper are as follows: • We identify limitations of existing VLM-based game-playing approaches, particularly their inability to handle real-time, high-reactivity tasks. • We propose a novel paradigm that uses VLMs to develop task-specific execution modules for autonomous real-time interactions. • We introduce GameSense , a framework that enables VLMs to create and refine Game Sense Modules (GSMs) . • Our experiments demonstrate that GameSense outperforms existing methods and is the first to master reflex-intensive games. 2 Related Work 2.1 Environment for Video Gameplay and RL-based Agents Researchers have made significant strides in various video game environments, including classic games like Atari games Bellemare et al. (2013) , Minecraft Fan et al. (2022) ; Guss et al. (2019) , StarCraft II Ellis et al. (2023) . However, these environments rely heavily on open-source code or official APIs, requiring substantial human effort for implementation. This dependency restricts AI accessibility to general games. Recent RL-based approaches have attempted to overcome API dependencies by directly processing game visuals and simulating keyboard-mouse inputs, including DQN-play-sekiro analoganddigital. (2021) . However, these RL methods typically work for specific tasks and exhibit poor generalization, requiring retraining for new scenarios. The challenge of developing agents capable of generalizing across diverse gaming environments without API access remains largely unsolved. This limitation motivates our research toward a more adaptable solution using only visual inputs and key-mouse controls. 2.2 LLM/VLM-Driven Gameplay Agent Current LLM/VLM-driven gameplay agents follow two main approaches. The first relies on game APIs for state observation and control, as seen in Minecraft Wang et al. (2023a) ; Liu et al. (2024a) and Starcraft II agents Ma et al. (2023)

Read the complete text