ChronoPlay: A Framework for Modeling Dual Dynamics and Authenticity in Game RAG Benchmarks

Liyang He, Yuren Zhang, Ziwei Zhu, Zhenghui Li, Shiwei Tong · 2025

Read ChronoPlay: A Framework for Modeling Dual Dynamics and Authenticity in Game RAG Benchmarks by Liyang He, Yuren Zhang, Ziwei Zhu, Zhenghui Li, Shiwei Tong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-18455.txt.

Introduction

Liyang He and his four co-authors build a framework for testing retrieval-augmented generation systems inside online games. Such systems answer questions by first looking up source material, but games change constantly: content updates arrive, and player interests shift with them. No standard benchmark captured both kinds of change, and any automatic benchmark still needs questions that feel like real player questions.

Their framework, ChronoPlay, tracks the two moving parts with a dual update mechanism. Its question engine draws on official sources for factual accuracy and on player community material for authentic wording. Applying the framework to three different games produced the first changing benchmark of its kind for gaming. The results show how retrieval systems cope when both the game world and its audience keep moving under them.

Catalog record

CREATORLiyang He, Yuren Zhang, Ziwei Zhu, Zhenghui Li, Shiwei Tong
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE83 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-18455

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ChronoPlay: A Framework for Modeling Dual Dynamics and Authenticity in Game RAG Benchmarks Liyang He, Yuren Zhang, Ziwei Zhu, Zhenghui Li, Shiwei Tong Abstract Abstract: Retrieval Augmented Generation (RAG) systems are increasingly vital in dynamic domains like online gaming, yet the lack of a dedicated benchmark has impeded standardized evaluation in this area. The core difficulty lies in Dual Dynamics : the constant interplay between game content updates and the shifting focus of the player community. Furthermore, the necessity of automating such a benchmark introduces a critical requirement for player-centric authenticity to ensure generated questions are realistic. To address this integrated challenge, we introduce ChronoPlay, a novel framework for the automated and continuous generation of game RAG benchmarks. ChronoPlay utilizes a dual-dynamic update mechanism to track both forms of change, and a dual-source synthesis engine that draws from official sources and player community to ensure both factual correctness and authentic query patterns. We instantiate our framework on three distinct games to create the first dynamic RAG benchmark for the gaming domain, offering new insights into model performance under these complex and realistic conditions. Code is avaliable at: https://github.com/hly1998/ChronoPlay. 1 Introduction The advancement of Retrieval Augmented Generation (RAG) has been largely driven by its benchmarks [ 39 ] . From foundational benchmarks [ 36 , 18 ] to specialized evaluations in domains [ 19 , 32 , 42 , 13 ] , these platforms provide the essential, standardized means to measure progress. Recent efforts have further pushed the frontier from static snapshots to dynamic benchmarks that evolve over time [ 26 , 16 ] , reflecting the currency of information in the real world. The global gaming industry, a vast and highly dynamic digital frontier, represents a critical domain for such advancements [ 11 ] . Within this ecosystem, RAG is emerging as a key technology to enhance player experiences, from intelligent assistants to automated support bots [ 23 , 8 ] . However, a significant gap emerges: there are currently no RAG benchmarks for the gaming domain, leaving the application of RAG systems in this area without standardized evaluation. Figure 1 : An illustration of Dual Dynamics in Game, including Knowledge Evolution , which traces the game’s knowledge updates, and User Interest Drift , which maps the changing interests of players. This absence stems from the unique nature of the gaming ecosystem, which is composed of two constantly evolving entities: the game itself and its player community. This structure gives rise to a core challenge we term Dual Dynamics , as illustrated in Figure 1 . On one hand, Knowledge Evolution occurs as the game’s content and rules are in a constant state of flux due to frequent patches and version updates [ 33 ] . This is different from some stable domains, where a comprehensive static benchmark is enough to evaluate a general-purpose model. In the world of live-service games, such a static evaluation would become obsolete. On the other hand, a benchmark must also track User Interest Drift , which represents the systematic evolution of the player community’s focus, from initial onboarding questions to late game content [ 17 , 37 ] . Capturing this shifting focus is crucial, as a benchmark that fails to do so would lead to models being optimized on a distribution of problems that no longer reflects what the community cares about, resulting in a biased and irrelevant evaluation. Given the sheer velocity of these dual dynamics, manually curating a benchmark that remains consistently up-to-date is practically impossible. This challenge is compounded by the vast diversity across different games. Consequently, automated synthesis has emerged as the only viable path forward for creating dynamic benchmarks in this domain. However, existing synthesis approaches [ 14 , 16 , 15 , 26 , 28 ] have focused primarily on a single dimension of change: the evolution of knowledge. By concentrating only on tracking knowledge updates, they have overlooked the critical need for authenticity in their generation process. In a user-centered domain, a benchmark is fundamentally invalid if it is filled with unrealistic questions that no real player would ask, regardless of their grammatical correctness. To tackle the challenges mentioned above, we introduce ChronoPlay , a novel framework for modeling both dual dynamics and authenticity in the generation of game RAG benchmarks. Its core features include: (1) a dual-source synthesis engine that ensures factual correctness by drawing from official sources, while capturing authentic player question patterns and interest preferences by mining player community information; and (2) a dual-dynamic update mechanism that precisely refreshes question-answer pairs by identifying entities in game updates and dynamically adjusts the distribution of evaluation questions in response to shifts in community interest. Using this framework, we instantiate three distinct games. Our subsequent analysis provides an initial baseline for classical RAG systems, and the results demonstrate that our benchmark effectively captures the unique challenges of these dually dynamic conditions, highlighting aspects of model performance that existing evaluation methods cannot measure. Notably, while we focus on the gaming domain, our methodology is applicable to other domains characterized by an evolving knowledge base and an active user community, such as e-commerce and social media platforms. Our contributions are as follows: 1. We identify dual dynamics as a fundamental challenge for RAG systems operating in dynamic domains such as gaming, and argue that addressing it requires an automated synthesis process with player-centric authenticity as a critical requirement. 2. We propose ChronoPlay, the first framework designed to automatically generate a dynamic benchmark that integrally addresses the challenges of temporal relevance and player-centric authenticity. 3. We instantiate our framework on three distinct games to create the first dynamic RAG benchmarks for the gaming domain. Our analysis offers new insights into the performance of existing RAG systems under these complex, realistic conditions. 2 Related Work The development of RAG systems benefited significantly from question-answering benchmarks like Natural Questions (NQ) [ 18 ] , HotpotQA [ 36 ] , PopQA [ 24 ] , and CRAG [ 35 ] . They provided a standardized environment for evaluating a model’s retrieval and reasoning capabilities within a closed, static knowledge world. However, their static nature makes them incapable of assessing a system’s adaptability to the continuously evolving knowledge of the real world. To address this limitation, some works have focused on creating dynamic benchmarks [ 29 ] . These efforts can be broadly categorized by their update trigger: being either period-driven or fact-driven. The period-driven approach updates at fixed intervals, often leveraging retrospective snapshots of sources like Wikipedia. For instance, HOH [ 26 ] , GrowOVER [ 16 ] , EvolvingQA [ 15 ] , and DynaQuest [ 20 ] use monthly snapshots to generate question-answer pairs that test a model’s ability to handle outdated information. Besides, DRAGON [ 4 ] is built upon a periodically updated news corpus. The fact-driven approach aims for lower latency by triggering updates upon the emergence of new information. While not all are designed strictly for RAG, methods like REALTIMEQA [ 14 ] , which uses live news feeds, and LIVEXIV [ 28 ] , which scrapes new preprints, exemplify this paradigm. These dynamic benchmarks share a common trait: their evolution is driven entirely by factual knowledge

Read the complete text