Can LLMs Play Ô Ăn Quan? A Study of Multi-Step Planning and Decision Making

Sang Quang Nguyen 1,2 , Kiet Van Nguyen 1,2 , Vinh-Tiep Nguyen 1,2 , Thanh Duc Ngo 1,2 , Ngan Luu-Thuy Nguyen 1,2 , Duy-Dinh Le 1,2 1 University of Information Technology, Ho Chi Minh City, Vietnam 2 Vietnam National University, Ho Chi Minh City, Vietnam [email protected] {kietnv, tiepnv, th · 2025

Read Can LLMs Play Ô Ăn Quan? A Study of Multi-Step Planning and Decision Making by Sang Quang Nguyen 1,2 , Kiet Van Nguyen 1,2 , Vinh-Tiep Nguyen 1,2 , Thanh Duc Ngo 1,2 , Ngan Luu-Thuy Nguyen 1,2 , Duy-Dinh Le 1,2 1 University of Information Technology, Ho Chi Minh City, Vietnam 2 Vietnam National University, Ho Chi Minh City, Vietnam [email protected] {kietnv, tiepnv, th (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-03711.txt.

Introduction

Sang Quang Nguyen, Kiet Van Nguyen, Vinh-Tiep Nguyen, and a wider team wrote this machine learning paper, published in 2025. It tests whether large language models can play a traditional Vietnamese board game, using the game as a probe of planning and decision making.

The game, Ô Ăn Quan, is a sowing game of token moves and captures played in a chain, where each choice reshapes the board many steps ahead. The authors gave language models different playing personas, from aggressive to defensive, and set them to play. Three models of rising size from one family were tried. The study tracks how the models plan moves, hold the game state in mind, and adapt their strategy. The authors say the results expose both the strengths and the blind spots of current models at step-by-step reasoning.

Catalog record

CREATORSang Quang Nguyen 1,2 , Kiet Van Nguyen 1,2 , Vinh-Tiep Nguyen 1,2 , Thanh Duc Ngo 1,2 , Ngan Luu-Thuy Nguyen 1,2 , Duy-Dinh Le 1,2 1 University of Information Technology, Ho Chi Minh City, Vietnam 2 Vietnam National University, Ho Chi Minh City, Vietnam [email protected] {kietnv, tiepnv, th
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-03711

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Can LLMs Play Ô Ăn Quan? A Study of Multi-Step Planning and Decision Making Sang Quang Nguyen 1,2 , Kiet Van Nguyen 1,2 , Vinh-Tiep Nguyen 1,2 , Thanh Duc Ngo 1,2 , Ngan Luu-Thuy Nguyen 1,2 , Duy-Dinh Le 1,2 1 University of Information Technology, Ho Chi Minh City, Vietnam 2 Vietnam National University, Ho Chi Minh City, Vietnam [email protected] {kietnv, tiepnv, thanhnd, ngannlt, duyld}@uit.edu.vn Abstract In this paper, we explore the ability of large language models (LLMs) to plan and make decisions through the lens of the traditional Vietnamese board game, Ô Ăn Quan. This game, which involves a series of strategic token movements and captures, offers a unique environment for evaluating the decision-making and strategic capabilities of LLMs. Specifically, we develop various agent personas, ranging from aggressive to defensive, and employ the Ô Ăn Quan game as a testbed for assessing LLM performance across different strategies. Through experimentation with models like Llama-3.2-3B-Instruct, Llama-3.1-8B-Instruct, and Llama-3.3-70B-Instruct, we aim to understand how these models execute strategic decision-making, plan moves, and manage dynamic game states. The results will offer insights into the strengths and weaknesses of LLMs in terms of reasoning and strategy, contributing to a deeper understanding of their general capabilities. I Introduction Large Language Models (LLMs) such as GPT [ 1 ] , Llama [ 2 , 3 , 4 ] , and Gemini [ 5 ] have demonstrated remarkable performance across various natural language understanding and generation tasks. While their capabilities in handling tasks like translation [ 6 , 7 ] , summarization [ 8 ] , and question answering [ 9 ] have been extensively studied. Most existing game testbed focus on imperfect-information games (e.g., Werewolf [ 10 , 11 ] , Avalon [ 12 ] , Spyfall [ 13 ] ) or games with heavy reliance on deception and bluffing. In contrast, Ô Ăn Quan offers a unique structure of fully observable, deterministic turn-based gameplay with circular movement and resource looping, which presents a new challenge for LLMs in reasoning under structured progression. Fig. 1 : Overview of the LLM-based agent framework for Ô Ăn Quan . The system takes as input the current game state \mathcal{G} , reasoning history \mathcal{H} , and rule instructions \mathcal{R} , then generates a natural language rationale and action via a LLaMA-based model. The predicted action updates the board, and the process continues turn-by-turn in a closed-loop interaction. Ô Ăn Quan 1 1 1 https://en.wikipedia.org/wiki/Ô_ăn_quan is played between two players who take turns distributing tokens across the board and capturing tokens from their opponent. The game involves strategic planning, including managing resources and controlling key areas (referred to as the ”Quan”). In the context of LLMs, we can model different agent types (personas) to simulate different strategies and evaluate the LLM’s ability to plan, reason, and adapt in a dynamic environment. By leveraging the rules and structure of Ô Ăn Quan, we can analyze how LLMs make decisions based on the game state, evaluate their ability to handle both short-term and long-term planning, and determine their overall strategic thinking. This paper aims to bridge this gap by designing a series of experiments using different LLMs, ranging from smaller models such as Llama-3.2-3B-Instruct to larger counterparts such as Llama-3.1-8B-Instruct and Llama-3.3-70B-Instruct. The objective is to evaluate their performance in strategic multi-agent settings, specifically focusing on their ability to engage in the Ô Ăn Quan game—a traditional Vietnamese board game that combines elements of turn-based strategy, resource management, and foresight. Additionally, the study will examine how the adoption of diverse personas influences model decision-making and assess whether these models can adapt their strategies in response to dynamic game environments. II Related Work Decision-making in interactive environments has been a significant area of research in both artificial intelligence (AI) and game theory. Previous works have explored various methods for modeling and evaluating strategic decision making, from traditional game theory [ 14 ] to modern reinforcement learning (RL) and adversarial AI strategies [ 15 ] . In particular, recent studies have used board games such as Chess [ 16 ] and Go [ 17 ] as benchmarks to evaluate the performance of AI systems in complex and dynamic environments that require planning and long-term strategy. In the realm of natural language processing (NLP), LLMs have been extensively tested for tasks involving decision-making, reasoning, and planning. For example, studies such as those of [ 1 ] and [ 18 ] have demonstrated LLM capabilities in structured planning environments and strategic dialogue systems. These works underscore the importance of evaluating language models not only for their ability to answer factual questions but also for their capability to handle open-ended, multi-step tasks that require strategic reasoning. However, to date, there has been limited work that applies LLMs in interactive games that require both strategic decision-making and dynamic adaptability. Our approach is inspired by the growing interest in using games as testing ground [ 10 , 11 , 12 , 13 ] . Specifically, we focus on Ô Ăn Quan, a traditional Vietnamese game with a rich set of rules that offers a novel domain to test the limits of LLM planning abilities. III Game Environment Implementation Ô Ăn Quan is a traditional Vietnamese board game played between two players on a 2x6 grid as shown in Figure 1 . Each player controls five regular positions and a central Mandarin position ( Quan ) on their side of the board. The objective is to maximize one’s score by capturing as many tokens as possible, including the opponent’s Mandarin. Gameplay consists of two alternating phases as shown in Figure 2 : the Scattering Phase , where a player selects one of their positions and redistributes all peasant tokens it contains clockwise or counterclockwise around the board; and the Capturing Phase , where tokens may be captured based on specific spatial configurations. The game concludes when both Mandarin tokens are captured or no legal moves remain. Beyond its simple rules, Ô Ăn Quan presents a rich set of strategic challenges. Players must engage in strategic planning , carefully selecting where to begin scattering in order to chain moves, set traps, and avoid future vulnerabilities. Effective resource management is also critical, as aggressive captures must be weighed against the need to maintain sufficient tokens to sustain future turns. Finally, the game demands high adaptability , as the board state evolves dynamically with each move, requiring continuous reassessment of threats and opportunities. Fig. 2 : Illustration of token movement in Ô Ăn Quan: from scattering to capturing. III-A Extra Rules: Strategic Challenges for LLMs E1 – Immature Mandarin Rule. A Mandarin with fewer than 5 tokens (also referred to as Quan Non ) cannot be captured. E2 – Forced Redistribution. If a player has no peasants remaining in their positions, they must return 5 captured tokens to the board to continue the game. E3 – Early Game Restriction. No Mandarin captures are allowed during the first two rounds, encouraging early-game setup rather than aggression. E4 – Two-Empty Rule. If the two positions immediately beyond the last scattered token are empty or invalid, the player’s turn ends immediately. E5 – Forced Capture Chain. Whenever a valid capture is available, it must be

Read the complete text