Are Large Vision Language Models Good Game Players?
Xinyu Wang, Bohan Zhuang, Qi Wu · 2025
Read Are Large Vision Language Models Good Game Players? by Xinyu Wang, Bohan Zhuang, Qi Wu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-02358.txt.
Introduction
Xinyu Wang, Bohan Zhuang, and Qi Wu test vision-language models as game players in a paper from two thousand twenty-five. Standard benchmarks, like image question answering, miss many skills these models need. Games instead demand perception, rule following, and multi-step reasoning in a structured setting.
Their framework sorts play into four tasks: perceiving the scene, answering questions, following rules, and playing end to end. Extensive experiments expose current limits. Models stumble on long, structured outputs and on dense visual detail. The authors release code and data in public. They argue game play gives a fuller picture of model cognition than static benchmarks.
Catalog record
| CREATOR | Xinyu Wang, Bohan Zhuang, Qi Wu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 95 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-02358
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Are Large Vision Language Models Good Game Players? Xinyu Wang, Bohan Zhuang, Qi Wu Abstract Large Vision Language Models (LVLMs) have demonstrated remarkable abilities in understanding and reasoning about both visual and textual information. However, existing evaluation methods for LVLMs, primarily based on benchmarks like Visual Question Answering and image captioning, often fail to capture the full scope of LVLMs’ capabilities. These benchmarks are limited by issues such as inadequate assessment of detailed visual perception, data contamination, and a lack of focus on multi-turn reasoning. To address these challenges, we propose LVLM-Playground, a game-based evaluation framework designed to provide a comprehensive assessment of LVLMs’ cognitive and reasoning skills in structured environments. LVLM-Playground uses a set of games to evaluate LVLMs on four core tasks: Perceiving, Question Answering, Rule Following, and End-to-End Playing, with each target task designed to assess specific abilities, including visual perception, reasoning, decision-making, etc. Based on this framework, we conduct extensive experiments that explore the limitations of current LVLMs, such as handling long structured outputs and perceiving detailed and dense elements. Code and data are publicly available at https://github.com/xinke-wang/LVLM-Playground . 1 Introduction Large Vision Language Models (LVLMs) ( Liu et al., 2024a ; Bai et al., 2023 ; Zhu et al., 2024 ) have recently demonstrated remarkable capabilities in processing and generating visual and linguistic information. These models extend the strengths of Large Language Models (LLMs) by incorporating the ability to understand and reason about visual data alongside textual inputs, exhibiting human-like reasoning capabilities across a variety of complex tasks that demand strong reasoning skills. However, the current evaluation methods for LVLMs vary significantly across different studies, lacking a unified criterion. Most existing LVLM works are assessed using multimodal data established before the LVLM era, such as Visual Question Answering (VQA) ( Antol et al., 2015 ) , image captioning ( Chen et al., 2015 ) , Optical Character Recognition (OCR) ( Fu et al., 2024 ) , etc. While these tasks have been instrumental in assessing model performance, they present several limitations: • Inadequate Assessment of Detail Perception. Existing evaluation tasks, such as generic VQA and image captioning, often fail to effectively assess models’ ability to perceive and understand fine-grained visual details. For example, in some VQA datasets, questions can be correctly answered without referencing the accompanying images, relying solely on text-based world knowledge ( Chen et al., 2024a ) . This is because some of these questions are not even genuine visual dependent, and thus they might not fully evaluate the models’ visual perception abilities. • Risk of Data Contamination. The extensive training of LVLMs on vast amounts of data from diverse sources increases the risk of data contamination, where training data includes some of the question-answer pairs used for testing ( Dong et al., 2024 ; Zhang et al., 2024 ; Wei et al., 2023 ) . Such overlap can lead to overestimated performance on tasks like VQA, as models may recall answers from their training data rather than genuinely understanding and reasoning about inputs. • Limitations of Metrics. Existing metrics for evaluating LVLMs on generative tasks such as CIDEr and CLIP Score, heavily rely on short, predefined captions. These metrics are not designed to evaluate the rich and varied outputs that LVLMs can produce, especially for open-ended questions without unique correct answers. This limitation forces many LVLM methods to prompt the models to generate short answers to fit these metrics ( Li et al., 2023 ) , which can restrict the models’ ability to showcase their full capabilities. • Inconsistent Prompts. LVLMs often employ specific prompts during training and inference, including zero-shot, few-shot, and Chain-of-Thought (CoT) prompting. These variations can significantly impact performance ( Wei et al., 2022 ; Li et al., 2024b ) , making it difficult to attribute differences to the models themselves rather than the prompts used. • Neglect of Multi-turn/Long-context Reasoning. Current evaluation benchmarks focus predominantly on single-turn interactions, overlooking the importance of multi-turn reasoning and long-context understanding in LVLMs ( Liu et al., 2024b ) . While real-world applications often require engaging in extended dialogues and maintaining context over multiple exchanges, these capabilities are not adequately assessed by existing benchmarks. Building upon the limitations of current benchmarks, there is a need for more comprehensive evaluation frameworks that assess the full scope of LVLMs’ capabilities. Games, with their structured environments, offer a promising solution. They inherently require not only a detailed perception of dynamic game states but also the ability to formulate strategies, anticipate an opponent’s moves, and adapt to new scenarios over multiple turns. This combination of visual understanding, long-term planning, and decision-making under constraints makes games particularly well-suited for evaluating LVLMs across a wide range of cognitive and reasoning tasks. Given that even simple AI algorithms, such as Minimax and Monte Carlo Tree Search, can master games through optimization ( Campbell et al., 2002 ) , it raises the question: Can LVLMs generalize their advanced reasoning and perception skills to perform competitively in these structured, logic-based environments? To address this question, we propose a game-based evaluation framework LVLM-Playground to assess LVLMs’ abilities in structured environments thoroughly. Unlike traditional benchmarks, games provide a controlled yet dynamic setting that naturally tests perception, reasoning, decision-making and competing abilities. This framework directly addresses the key limitations of existing evaluations, as outlined below: • Detailed Perception. Board games like Chess and Go require a precise perception of game states. Models must accurately interpret the positions and identities of pieces, processing fine-grained visual details essential for strategic decision-making. • Unique Data. Game data is largely absent from current LVLM training sets, reducing the risk of contamination. Additionally, diverse environments and varying UI designs in games offer a wide range of novel, procedurally generated scenarios. • Transparent Metrics. Games have clear, rule-based outcomes, such as winning or losing, providing objective metrics for performance evaluation and eliminating the ambiguity often found in current benchmarks that rely on human-annotated answers. • Consistent Prompts. In games, the rules can serve as a uniform prompt, clearly defining what the model should do. This ensures models are evaluated on a level playing field, reducing variability and leading to more consistent, reproducible assessments. • Multi-turn Reasoning. Games naturally require multi-turn interactions and long-term strategic thinking. To perform well, a model must maintain context over several moves, plan ahead, adapt dynamically to changing situations, and anticipate the opponent’s moves, making games an ideal testbed for assessing sustained reasoning over extended periods. In summary, games offer an alternative to existing benchmarks by providing a structured, dynamic, and comprehensive environment for evaluating LVLMs. In this paper, we propose LVLM-Playground , a game-based evaluation framework that includes six unique games: Tic-Tac-Toe, Reversi, Minesweeper, Gomoku, Sudoku, and Chess. The contributions of this work are as follows: • We built LVLM-Playground from scratch, a comprehensive benchmark that integrates game UIs and AI opponents, enabling both online and offline interactions between LLMs and games. The framework supports common interfaces for both commercial models, such as OpenAI API, and open-source models, like those from the HuggingFace Transformers library. LVLM-Playground includes multiple tasks with various settings, as well as automated evaluation mechanisms. • We systematically designed a comprehensive framework to quantify the abilities required for each game and task,