FairGamer : Evaluating Biases in the Application of Large Language Models to Video Games
Bingkang Shi, Jen-tse Huang, Guoyi Li, Xiaodan Zhang, Zhongjiang Yao · 2025
Read FairGamer : Evaluating Biases in the Application of Large Language Models to Video Games by Bingkang Shi, Jen-tse Huang, Guoyi Li, Xiaodan Zhang, Zhongjiang Yao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-17825.txt.
Introduction
Bingkang Shi and co-authors present FairGamer, a benchmark that tests social bias in game characters run by language models, in a 2025 paper. Such models now power many non-player characters, and they can carry biases about race, class, age, or nationality into play. Fair tests for this risk have been lacking.
FairGamer covers three kinds of in-game dealings: trade, cooperation, and competition. It spans twelve tasks and uses a new score to rate biased choices. Seven leading models were tested. All showed biased decisions, and the largest models showed the strongest bias, which suggests that bigger models can amplify unfair patterns. The benchmark is released in public to support work on fairer game characters.
Catalog record
| CREATOR | Bingkang Shi, Jen-tse Huang, Guoyi Li, Xiaodan Zhang, Zhongjiang Yao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-17825
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FairGamer : Evaluating Biases in the Application of Large Language Models to Video Games Bingkang Shi, Jen-tse Huang, Guoyi Li, Xiaodan Zhang, Zhongjiang Yao Abstract Leveraging their advanced capabilities, Large Language Models (LLMs) demonstrate vast application potential in video games—from dynamic scene generation and intelligent NPC interactions to adaptive opponents—replacing or enhancing traditional game mechanics. However, LLMs’ trustworthiness in this application has not been sufficiently explored. In this paper, we reveal that the models’ inherent social biases can directly damage game balance in real-world gaming environments. To this end, we present FairGamer , the first bias evaluation Benchmark for LLMs in video game scenarios, featuring six tasks and a novel metrics D l s t d D_{lstd} . It covers three key scenarios in games where LLMs’ social biases are particularly likely to manifest: Serving as Non-Player Characters , Interacting as Competitive Opponents , and Generating Game Scenes . FairGamer utilizes both reality-grounded and fully fictional game content, covering a variety of video game genres. Experiments reveal: (1) Decision biases directly cause game balance degradation, with Grok-3 (average D l s t d D_{lstd} score=0.431) exhibiting the most severe degradation; (2) LLMs demonstrate isomorphic social/cultural biases toward both real and virtual world content, suggesting their biases nature may stem from inherent model characteristics. These findings expose critical reliability gaps in LLMs’ gaming applications. Our code and data are available at anonymous GitHub https://github.com/Anonymous999-xxx/FairGamer . Introduction Large Language Models (LLMs) have emerged as powerful tools capable of processing and generating human-like text ( Qin et al. 2023 ; Dubey et al. 2024 ; Liu et al. 2024 ) . Beyond natural language processing tasks such as text translation ( Jiao et al. 2023 ) , sentence revision ( Wu et al. 2023a ) , and computer programming ( Lee et al. 2024 ) , their abilities have extended to diverse fields, including education ( Baidoo-Anu and Ansah 2023 ) , legal ( Guha et al. 2023 ) and medical ( Johnson et al. 2023 ) advice. Given their advanced functionalities, LLMs hold the potential to revolutionize the video game industry by replacing or augmenting traditional game mechanics. The rationale for integrating LLMs into video games is grounded in two primary objectives: simplifying implementation and enabling functionalities that are difficult or impossible to achieve with traditional, fixed-program logic. Based on this premise, we identify three key scenarios where LLMs can effectively replace specific functionalities in video games: Figure 1: Heatmap of DeepSeek-V3’s Discount in Task SNPC-Virtual. Values represent the average discount output by the LLM (e.g., 0.75 indicates a 75% discount). (1) Serving as Non-Player Characters (SNPC) : Traditionally, NPC behavior is constrained by predefined scripts and limited response options, often resulting in interactions that feel mechanical and predictable. In contrast, NPCs built on LLMs can comprehend player input and dynamically generate natural responses that align with their character and the storyline. (2) Interacting as Competitive Opponents (ICO) : Nowadays, NPCs in multiplayer online battle arena (MOBA) and strategy games often rely on fixed rules or traditional RL algorithms, lacking flexibility and diversity. 1 1 1 https://en.wikipedia.org/wiki/Multiplayer˙online˙battle˙arena LLMs offer a highly promising alternative—they can adapt to diverse gaming scenarios and make strategic decisions based on real-time data, delivering a more authentic and dynamic competitive experience. (3) Generating Game Scenes (GGS) : In rogue-like games, for example, players encounter procedurally generated scenes designed through fixed algorithms. 2 2 2 https://en.wikipedia.org/wiki/Roguelike LLMs, by contrast, can create varied and contextually rich scenes based on specific parameters, enhancing the novelty and replayability of the game. Figure 2: An illustration of FairGamer evaluation pipeline. By loading character information into the game scenario description, FairGamer evaluates the LLM’s social or cultural biases in video games. SELF-ROLE is the role played by the LLM, defined by the scenario, while OBS-ROLE is the character interacting with the LLM. ACT-SPACE refers to LLM’s action space defined by available choices and the action count. Previous studies have focused on using LLMs to facilitate development process, including software coding ( Chen et al. 2023 ) , plot design ( Alavi et al. 2024 ) , and testing ( Paduraru, Staicu, and Stefanescu 2024 ) . Despite this transformative potential in game design, the trustworthiness of LLMs—particularly concerning their inherent social biases ( Felkner et al. 2023 ; Zheng et al. 2023 ; Naous et al. 2024 ; Ross, Kim, and Lo 2024 ; Taubenfeld et al. 2024 ) —remains underexplored in video game contexts. While several benchmarks assess social bias in LLMs ( May et al. 2019 ; Kumar et al. 2024 ; Luo et al. 2024 ; Wang et al. 2024a ; Felkner et al. 2023 ; Zheng et al. 2023 ; Naous et al. 2024 ; Ross, Kim, and Lo 2024 ; Taubenfeld et al. 2024 ; Borah and Mihalcea 2024 ) , few address their implications in interactive game environments. Such biases may subtly affect game balance: stereotyped NPC dialogue could reinforce harmful norms, and biased training data may lead to unfair advantages or disadvantages in competitive scenarios. To explore the impact of LLMs’ biases on the above gaming scenarios, we introduce FairGamer , a benchmark for evaluating social/cultural bias in game-integrated LLMs and measuring its effect on game balance. For the three scenarios above, FairGamer defines specific evaluation modes and tasks: 1. Role-Playing as Merchants: It evaluates social bias in NPC merchants offering discounts to different customers. It includes two tasks: SNPC-Real and SNPC-Virtual . The former uses in-game customer information derived from the real world (race, career), while the latter uses entirely fictional ones. 2. Competing Against Players: This mode detects cultural biases exposed in the diplomatic decision-making of LLMs when they assume the role of nations in strategy games. It consists of two tasks: ICO-Real and ICO-Virtual . Similarly, the former uses in-game nations derived from real-world countries, while the latter uses entirely fictional ones. 3. Generating Menus for Bistros: This scenario tests cultural bias in LLM-generated bar menus. It includes two tasks: GGS-Real and GGS-Virtual . The former uses alcoholic beverages from the real world, while the latter uses fictional drinks from Steam games. 3 3 3 https://store.steampowered.com/ We collect a dataset comprising 89.98K test cases in English and Chinese from 58 Steam games, with five games contributing the majority of examples. These cases span three output formats: single-choice, multiple-choice, and numerical. FairGamer performs multiple evaluations of each test case with randomized answer options. For all six tasks, we propose a novel evaluation metrics: Decision Log Standard Deviation ( D_{lstd} ) , which quantifies LLMs’ social/cultural bias from its decision distribution. Using FairGamer , we evaluate eight state-of-the-art models, including three closed-sourced models and five open-sourced models. The results in Table 3 reveal that Grok-3 exhibits the highest average bias across six tasks, with the averaged D_{lstd} score reaching 0.431. While LLaMA-3.1 8B shows the lowest average bias, with the averaged D_{lstd} score of 0.226. And based on bias thresholds derived from real-player sampling surveys, we determined that only DeepSeek-V3 (on GGS-Virtual) and Qwen2.5-72B (on SNPC-Virtual) showed no significant bias. The contributions of this paper include: • We identify three scenarios where LLMs can efficiently be used to replace functionalities