: Creating New Games with Generative Interactive Videos

Jiwen Yu, Yiran Qin, Xintao Wang, Pengfei Wan, Di Zhang, Xihui Liu · 2025

Read : Creating New Games with Generative Interactive Videos by Jiwen Yu, Yiran Qin, Xintao Wang, Pengfei Wan, Di Zhang, Xihui Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-08325.txt.

Introduction

Six authors, led by Jiwen Yu, present GameFactory in 2025, a system that builds new playable game videos with generative models. Video models can now invent scenes, but steering them with keys and a mouse is hard. Past tools also stuck to one fixed game style or scene.

The team built a data set of Minecraft play marked with each action, free of human bias. A control module learned to follow key and mouse input, and the system can roll out scenes of any length. A training plan splits style learning from action learning. That split lets the same controls drive fresh scenes in new styles.

Catalog record

CREATORJiwen Yu, Yiran Qin, Xintao Wang, Pengfei Wan, Di Zhang, Xihui Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-08325

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

: Creating New Games with Generative Interactive Videos Jiwen Yu, Yiran Qin, Xintao Wang, Pengfei Wan, Di Zhang, Xihui Liu Abstract Generative videos have the potential to revolutionize game development by autonomously creating new content. In this paper, we present GameFactory, a framework for action-controlled scene-generalizable game video generation. We first address the fundamental challenge of action controllability by introducing GF-Minecraft, an action-annotated game video dataset without human bias, and developing an action control module that enables precise control over both keyboard and mouse inputs. We further extend to support autoregressive generation for unlimited-length interactive videos. More importantly, GameFactory tackles the critical challenge of scene-generalizable action control, which most existing methods fail to address. To enable the creation of entirely new and diverse games beyond fixed styles and scenes, we leverage the open-domain generative priors from pre-trained video diffusion models. To bridge the domain gap between open-domain priors and small-scale game datasets, we propose a multi-phase training strategy with a domain adapter that decouples game style learning from action control. This decoupling ensures that action control learning is no longer bound to specific game styles, thereby achieving scene-generalizable action control. Experimental results demonstrate that GameFactory effectively generates open-domain action-controllable game videos, representing a significant step forward in AI-driven game generation. Figure 1 : We propose GameFactory, a framework that leverages the powerful generative capabilities of pre-trained video models for the creation of new games. By learning action control from a small-scale first-person Minecraft dataset , this framework can transfer these control abilities to open-domain videos, ultimately allowing the creation of new games within open-domain scenes. As illustrated in (1)-(4), GameFactory supports action control across diverse, newly generated scenes in open domains, paving the way for the development of entirely new game experiences. The yellow buttons indicate pressed keys , and the arrows represent the direction of mouse movements . 1 1 footnotetext: Corresponding authors. ∗ Equal contribution. † Part of the work done during internship at KwaiVGI, Kuaishou Technology. 1 Introduction Video diffusion models have demonstrated impressive capabilities in video generation [ 31 , 26 , 52 , 50 ] . These models have the potential to become promising candidates for generative game engines [ 42 , 2 , 11 , 6 , 12 ] , which could transform game development by not only significantly reducing manual work in the traditional game industry through automatic content creation, but also enabling the generation of unlimited game content for players to explore. Given this promising direction, it is important to explore how to build a qualified generative game engine. Generative game engines are typically implemented as video generation models with action controllability, enabling responses to user inputs like keyboard and mouse interactions [ 42 , 11 , 12 ] . Current research works [ 42 , 2 , 11 , 7 , 47 ] mainly focus on specific games such as DOOM [ 42 , 47 ] , Atari [ 2 ] , CS:GO [ 2 ] , Super Mario Bros [ 47 ] , and Minecraft [ 11 ] , or use limited game-specific datasets [ 7 ] . This game-specific approach lacks scene generalization capability, preventing models from creating content beyond existing games and limiting their potential for developing new games. Thus, scene generalization remains a key challenge in advancing generative game engines. Figure 2 : A schematic of our GameFactory creating new games based on pre-trained large video generation models. The upper blue section shows the generative capabilities of the pre-trained model in an open-domain, while the lower green section demonstrates how the action control module, learned from a small amount of game data, can be plugged in to create new games. To achieve scene generalization in game videos, collecting large-scale action-annotated video datasets would be the most direct approach. With sufficient data covering all possible scenarios, it could enable arbitrary game scene generation. However, such action annotations are prohibitively expensive and impractical for open-domain videos. In contrast, open-domain videos are abundant on the Internet, and video generation models trained on these videos contain rich generative priors. Leveraging these scene-generation priors presents a more feasible path to scene generalization ( upper blue section in Fig. 2 ). This observation motivates us to ask: With the video generation models pre-trained on large-scale open-domain videos, along with small-scale action-annotated game videos, can we empower scene-generalizable video generation with action control? This is a nontrivial problem, because directly fine-tuning the pre-trained video generation models with action-annotated game videos will lead to degraded scene generalizability ( i.e . , the model will collapse to the specific domain of the game videos). In this work, we present GameFactory , a framework for action-controlled and scene-generalizable game video generation. To achieve action-controlled video generation, we firstly collect an action-annotated game video dataset GF-Minecraft. This dataset, sourced from the Minecraft platform [ 13 ] , features an unbiased action distribution, diverse scenes, and text annotations (Sec. 4.1 ). Secondly, we design action control mechanisms for continuous mouse movements and discrete keyboard inputs, respectively. The action control modules can be seamlessly injected into the pre-trained video diffusion models without affecting the video generation abilities of the pre-trained models (Sec. 4.2 ). Lastly, we extend our model to autoregressive long video generation by allowing varying noise levels across different frames, continuously conditioning on previously generated frames to generate new video frames (Sec. 4.3 ). However, directly fine-tuning the pretrained model on Minecraft data not only imparts action control capabilities, but also risks embedding a distinct Minecraft style into the generated games, thereby compromising the model’s open-domain generalization. To address this challenge, our key idea is to disentangle the learning of game style and action control with different modules and parameters, so that we can remove the game style without affecting the action controllability for generating open-domain videos with action control (Sec. 5.1 ). Specifically, we propose a domain adapter to fit Minecraft style, combined with a multiphase decoupled training strategy (Sec. 5.2 ). In summary, our key contributions include: • We propose GameFactory for action-controlled and scene-generalizable video generation, enabling diverse interactive game generation beyond existing games. • We introduce GF-Minecraft, an action-annotated game video dataset featuring unbiased action distributions, diverse scenes, and text descriptions. We also design dedicated mechanisms for action control and autoregressive long video generation. • We propose a domain adapter with multiphase decoupled training strategy that disentangles game style from action control, enabling open-domain video generation while preserving action controllability. 2 Related Work 2.1 Video Diffusion Models With the rise of diffusion models [ 19 , 41 , 40 ] , significant progress has been made in visual generation, spanning both image [ 38 , 10 , 34 ] and video generation [ 9 , 5 , 52 , 26 , 50 ] . In particular, recent advancements in video generation have seen a shift from the U-Net architecture to Transformer-based architectures, enabling video diffusion models to generate highly realistic, longer-duration videos [ 33 , 31 ] . The quality of these generated videos has led to the belief that video diffusion models are capable of understanding and simulating real-world physical rules [ 45 , 31 , 49 ] , suggesting their potential use

Read the complete text