AutoUE: Automated Generation of 3D Games in Unreal Engine via Multi-Agent Systems

Lei Yin, Wentao Cheng, Zhida Qin, Tianyu Huang, Yidong Li, Gangyi Ding · 2026

Read AutoUE: Automated Generation of 3D Games in Unreal Engine via Multi-Agent Systems by Lei Yin, Wentao Cheng, Zhida Qin, Tianyu Huang, Yidong Li, Gangyi Ding (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-07106.txt.

Introduction

Yin and colleagues build AutoUE, a multi-agent system that generates 3D games end to end in Unreal Engine. Separate agents handle model retrieval, scene building, gameplay and interaction code, and automated play-testing. To curb tool-use hallucinations, a retrieval step grounds agents in relevant engine tool docs, and game design patterns plus engine constraints guide code generation. A play-testing pipeline runs runtime test commands, giving systematic checks of dynamic behavior.

Catalog record

CREATORLei Yin, Wentao Cheng, Zhida Qin, Tianyu Huang, Yidong Li, Gangyi Ding
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35110
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2603.07106

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AutoUE: Automated Generation of 3D Games in Unreal Engine via Multi-Agent Systems Lei Yin, Wentao Cheng, Zhida Qin, Tianyu Huang, Yidong Li, Gangyi Ding Abstract Automatically generating 3D games in commercial game engines remains a non-trivial challenge, as it involves complex engine-related workflows for generating assets such as scenes, blueprints, and code. To address this challenge, we propose a novel multi-agent system, AutoUE , which coordinates multiple agents to end-to-end generate 3D games, covering model retrieval, scene generation, gameplay and interaction code synthesis, and automated game testing for evaluation. In order to mitigate tool-use hallucinations in LLMs, we introduce a retrieval-augmented generation mechanism that grounds agents with relevant UE tool documentation. Additionally, we incorporate game design patterns and engine constraints into the code generation process to ensure the generation of correct and robust code. Furthermore, we design an automated play-testing pipeline that generates and executes runtime test commands, enabling systematic evaluation of dynamic behaviors. Finally, we construct a game generation dataset and conduct a series of experiments that demonstrate AutoUE’s ability to generate 3D games end-to-end, and validate the effectiveness of these designs. 1 1 1 The code is released at https://github.com/Pluto156/AutoUE . 1 Introduction Digital games, as one of the most influential forms of interactive media, play an essential role in entertainment and simulation 16 ; 17 . Due to the need to design complex visual environments and system logic, the game creation remains a highly creative yet labor-intensive process. Recent advances in Large Language Models (LLMs) 8 and generative models 12 have motivated studies on automated game generation from natural language descriptions 3 ; 26 ; 6 , aiming to reduce development barriers. A promising direction is to integrate LLMs with game engines to generate in-engine assets 15 ; 2 , which offers better controllability, editability, and practical value for game development. Figure 1: An example of game scene and gameplay. Despite these advances, automatically generating 3D games in game engines remains a non-trivial challenge 7 . Compared with 2D or simple game settings 25 ; 13 ; 21 , 3D games require more complex spatial layouts, gameplay behaviors, and coordination across multiple subsystems. Recent works have integrated LLMs with commercial engines such as Unreal Engine (UE) and Unity to generate 3D scenes 20 , gameplay logic 30 , and engine code 32 . However, most of these efforts focus on isolated individual components, making it difficult to generate a complete, playable 3D game end-to-end. To address these challenges, we propose AutoUE , a novel multi-agent system that coordinates agents for model retrieval, scene generation, gameplay code, interactive object, and automated play-testing to produce complete 3D games in UE. Specifically, the model retrieval agent builds an embedding database over 858K 3D models, enabling efficient semantic retrieval of relevant model assets. Based on the retrieved models, the scene generation agent uses UE’s built-in tool Procedural Content Generation (PCG) to generate a visually editable layout graph that produces a coherent 3D scene. Then, the gameplay code agent follows common game-development design patterns and respects engine-specific constraints to generate modular gameplay C++ code. Building on these modules, the interactive object agent derives object interaction flows and implements the corresponding C++ code, enabling in-scene entities to provide player interactions consistent with the description. Finally, the automated play-testing agent automatically generates and executes runtime test commands to evaluate the scene and gameplay behaviors. These agents communicate via structured specifications to convey contextual information and ensure consistency across the system. Our system has the following innovations: (1) Compared to other engines, UE provides comprehensive built-in tools for efficient game development. To mitigate tool-use hallucinations by LLMs, we propose a retrieval-augmented generation (RAG) based mechanism that retrieves relevant information from tool documents and provides the agent with accurate usage instructions. (2) Existing methods often overlook design patterns in game development, resulting in unconstrained code that is difficult to maintain and extend. To address this limitation, we generate functional gameplay module code and design a module management framework based on common practices. This enables the system to produce reusable and composable gameplay code, which can be invoked by interactive objects in a controlled manner. (3) Prior work evaluates generated assets via human inspection or static checks. However, a complete game is defined by dynamic gameplay and interaction behaviors, making objective evaluation without human effort challenging. We introduce an automated play-testing pipeline that generates test commands and implements a Model Context Protocol (MCP) to execute them at runtime, enabling automated and systematic evaluation. Finally, we construct a dataset of 20 game-generation tasks and demonstrate that our method produces higher-quality 3D scenes than existing approaches. Meanwhile, we design multiple metrics that evaluate dynamic gameplay behaviors based on the generated gameplay and interaction code as well as runtime logs, providing a benchmark for end-to-end 3D game generation in game engines. Overall, our contributions are summarized as: • We present a practical end-to-end multi-agent system for generating 3D games in UE. The system collaboratively constructs a coherent 3D scene, synthesizes robust gameplay and interaction code, and performs automated play-testing. • We design a RAG mechanism that provides agents with tool-usage guidance, thereby reducing hallucinations for LLMs. During code generation, we incorporate game design patterns and engine constraints to improve the maintainability and extensibility of the generated code. • We construct a game-generation dataset and establish an automated play-testing pipeline, providing a benchmark for 3D game generation. It offers a solid foundation for multi-agent game generation, and the overall system provides an extensible infrastructure for future work. 2 Related Work Existing research on applying agents to games can be divided into two types: 2.1 LLMs as Game Playing Agents One line of work treats LLMs as game playing agents and uses games as testbeds to improve the agents’s ability to process complex observations. Early studies primarily focus on text-based games: 10 proposes an evaluation framework for interactive fiction and adopts a template-based action space to reduce complexity. Recent studies have further applied LLMs to open-world 3D games. DEPS 29 conditions planning on environment feedback and sub-goal selection to robustly complete diverse Minecraft tasks. Voyager 27 combines an expanding skill library with a feedback-driven iterative loop to achieve open-ended exploration in Minecraft. In addition, recent work has also explored agents in a wide range of game genres, including competitive 28 and action 4 games. The latest advances 24 train foundation models on large-scale gameplay data, enabling generalization to unseen games. Figure 2: The overall framework of our proposed multi-agent system AutoUE. Blue arrows indicate the workflow. 2.2 LLMs as Game Generation Agents Another line of work uses LLMs to generate game content, including text/video-based and engine-integrated game generation. The former generates game experiences in the form of text narratives 5 , scripted interactions 23 , or video-like content 1 . Such games are easier to synthesize end-to-end, but they lack free-form spatial interactions and developer-editable content. Therefore, recent studies couple LLMs with game engines to generate in-engine assets for modern game development. Sketch2Scene 31 uses diffusion models to turn user sketches into isometric 2D scene guides, and feeds guides into procedural generation to build 3D game scenes. 3Dify 11 uses RAG and MCP to invoke tools like Blender, Unity, and UE to

Read the complete text