BugCraft: End-to-End Crash Bug Reproduction Using LLM Agents in Minecraft DOI: XXXXXXX.XXXXXXX ISBN: 978-1-4503-XXXX-X/24/06 CCS: Software and its engineering Software testing and debugging CCS: Software and its engineering Maintaining software CCS: Software and its engineering Interactive games
Eray Yapağcı, Yavuz Alp Sencer Öztürk, Eray Tüzün · 2025
Read BugCraft: End-to-End Crash Bug Reproduction Using LLM Agents in Minecraft DOI: XXXXXXX.XXXXXXX ISBN: 978-1-4503-XXXX-X/24/06 CCS: Software and its engineering Software testing and debugging CCS: Software and its engineering Maintaining software CCS: Software and its engineering Interactive games by Eray Yapağcı, Yavuz Alp Sencer Öztürk, Eray Tüzün (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-20036.txt.
Introduction
Eray Yapağcı and two co-authors built BugCraft in 2025, a system that uses AI agents to reproduce crash bugs in the game Minecraft from player reports. The hosted text titles the work as agents in a sandbox, while the catalog title foregrounds the system name. Reproducing game crashes by hand is slow, and an interview with a Minecraft decision maker confirmed most reports need manual rebuilding. Their framework works in two stages.
First, a step writer turns a report into ordered reproduction steps, aided by the game's wiki. Then a vision-driven agent plays those steps inside the game through a custom action library. On a new benchmark of crash reports, the system reproduced about a third of crashes end to end and beat baseline agents clearly. Code is public.
Catalog record
| CREATOR | Eray Yapağcı, Yavuz Alp Sencer Öztürk, Eray Tüzün |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-20036
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BugCraft: End-to-End Crash Bug Reproduction Using LLM Agents in Minecraft DOI: XXXXXXX.XXXXXXX ISBN: 978-1-4503-XXXX-X/24/06 CCS: Software and its engineering Software testing and debugging CCS: Software and its engineering Maintaining software CCS: Software and its engineering Interactive games Eray Yapağcı, Yavuz Alp Sencer Öztürk, Eray Tüzün Abstract . Context: Reproducing game bugs, in our case crash bugs in continuously evolving games like Minecraft, is a notoriously manual, time-consuming, and challenging process to automate. Despite the success of LLM-driven bug reproduction in other software domains, games, with their complex interactive environments, remain largely unaddressed. Objective: This paper introduces BugCraft, a novel end-to-end framework designed to automate the reproduction of crash bugs in Minecraft directly from user-submitted bug reports, addressing the critical gap in automated game bug reproduction. Method: BugCraft employs a two-stage approach: first, a Step Synthesizer leverages LLMs and Minecraft Wiki knowledge to transform bug reports into high-quality, structured steps to reproduce (S2R). Second, an Action Model, powered by a vision-based LLM agent (GPT-4o) and a custom macro API, executes these S2R steps within Minecraft to trigger the reported crash. To facilitate evaluation, we introduce BugCraft-Bench, a curated dataset of Minecraft crash bug reports. Results: Evaluated on BugCraft-Bench, our framework successfully reproduced 30.23% of crash bugs end-to-end. The Step Synthesizer demonstrated a 66.28% accuracy in generating correct bug reproduction plans, highlighting its effectiveness in interpreting and structuring bug report information. Conclusion: BugCraft demonstrates the feasibility of automated reproduction of crash bugs in complex game environments using LLMs, opening promising avenues for game testing and development. The framework and the BugCraft-Bench dataset pave the way for future research in automated game bug analysis and hold potential for generalization to other interactive game platforms. Finally, we make our code open at https://bugcraft2025.github.io/ † † cc-license: by-sa 1. Introduction In the maintenance phase of the game development cycle, it is crucial to provide updates that improve the game to maintain player retention and enhance engagement ( Zhong_Xu_2021 ) . These game updates should also fix bugs for improved user experience. As a type of bug, crash bugs are especially severe because they prevent the user from experiencing the game. They are also difficult to fix, being the most recurring bug type even after updates addressing the bug ( Truelove_Santana_De_Almeida_Ahmed_2021 ) . A primary source of information for these updates is the user-submitted bug reports. A prime example is Mojira 1 1 1 https://bugs.mojang.com/issues , the public bug tracker for Minecraft developer Mojang Studios. Mojira hosts more than 450,000 bug reports, with over half related to Minecraft: Java Edition 2 2 2 https://www.minecraft.net/en-us/about-minecraft . However, many of these reports are of low quality, often missing essential components like steps to reproduce (S2R), expected outcomes, and can have mistakes in them ( Zimmermann_Premraj_Bettenburg_Just_Schroter_Weiss_2010 ; Soltani_Hermans_Bäck_2020 ) . The studies showing low-quality bug reports across numerous projects suggest that Mojira could potentially face similar challenges. Due to the excessive amount of labor involved, game testing is an entire field in itself, and it is a notoriously manual process that takes a significant amount of time ( Politowski_Petrillo_Gueheneuc_2021 ) . Most games lack automated test frameworks because implementing and generalizing such systems is challenging. For example, Minecraft’s internal unit test framework GameTest 3 3 3 https://www.minecraft.net/en-us/creator/article/get-started-gametest-framework lacks the tools to reproduce UI movement and precise player interactions, only covering a part of what is a huge sandbox game. Looking at a more general usage, automated bug reproduction systems have been developed ( Kang_Yoon_Yoo_2023 ) . With the advent of large language models (LLMs) that are capable of tool calling (i.e., interacting with external tools/APIs) ( qin2024toollearningfoundationmodels ) and code generation ( jiang2024surveylargelanguagemodels ) , the implementations of these systems have become more general. Similar works ( Wang_Zhao_Feng_Zhang_Halfond ) exist for reproducing bugs in Android apps through interacting with the available GUI. These works are limited in interacting with the game, which needs action taking in a complex environment. To our knowledge, there are no works for automating bug reproduction in an open-ended game environment like Minecraft. In this paper, we present BugCraft, an agentic bug reproduction framework for Minecraft. BugCraft can take in the initial bug report, synthesize high-quality steps to reproduce bugs alongside relevant game data from external sources and reproduce them step by step through scripting. In the end, our framework generates a log that can reproduce the bug. In addition, due to the lack of a dataset of crash bugs in a game, we present BugCraft-Bench, a crash bug dataset collected through Mojira. Our dataset contains 86 manually confirmed, reproducible crash bug reports. Finally, our reproduction package, alongside with our results can be found at: https://figshare.com/s/70b5d141e69ff822cd40 . Figure 1. The BugCraft framework, illustrating the two-stage process of S2R synthesis and action model execution. Our paper begins by discussing related work in Section 2 , followed by an overview of the framework in Section 3 . We define our research questions in Section 3.3 , evaluate the framework in Section 4 , and discuss our findings in Section 5 . The paper mentions the threats to validity in Section 6 , and concludes with a summary of contributions in Section 7 . 2. Related Work This section provides an overview of related topics, focusing on bug reproduction in games and the development of agents powered by LLMs. We also discuss similar research efforts relevant to our framework. 2.1. Bug Reproduction in Games Game testing is traditionally a manual and time-consuming process, and the primary approach involves playtesting, in which testers manually play the game to identify issues ( Politowski_Petrillo_Gueheneuc_2021 ) . Automated unit testing is often challenging for game developers due to the dynamic nature of game code and the complexity involved in capturing game states programmatically ( Pascarella_Palomba_Di_Penta_Bacchelli_2018 ) . In continuous development cycles, it becomes essential for developers to address existing bugs through regular updates to improve the gameplay experience. However, certain types of bugs, particularly crash bugs, are prone to reoccurrence even after an update is released to fix them ( Truelove_Santana_De_Almeida_Ahmed_2021 ) . One developer in a recent survey attributed this issue to limited or non-available testing time, highlighting the constraints of manual testing in the fast-paced, deadline-driven environment typical of game development. Wuji ( Zheng_Xie_Su_Ma_Hao_Meng_Liu_Shen_Chen_Fan_2019 ) represents the closest related work in automated game testing, using deep reinforcement learning to explore the game model to discover new bugs. However, Wuji’s capabilities are limited to predefined game areas and narrow action spaces. It requires manual environment setup without the ability to follow specific test prompts. Furthermore, Wuji operates in the domain of bug finding, while our framework focuses on reproducing known bugs. Our approach provides end-to-end bug reproduction capabilities, starting from game initialization and covering specific scenarios, enabling a more comprehensive approach to addressing reported issues. 2.2. Automated Bug Reproduction Using LLMs Automated bug reproduction entails converting unstructured bug reports into executable scripts reliably replicating the reported issues. LLMs have shown considerable success in generating code, particularly in creating unit tests ( Kang_Yoon_Yoo_2023 ) and enhancing bug report details ( Bo_Ji_Sun_Zhang_Wu_Wei_2024 ) . Recent frameworks leveraging LLMs have been developed to automate bug reproduction in mobile applications, particularly within Android environments, by interacting with the GUI ( Wang_Zhao_Feng_Zhang_Halfond ) . These frameworks