A Survey of Interactive Generative Video

Jiwen Yu *, Yiran Qin *, Haoxuan Che * · 2025

Read A Survey of Interactive Generative Video by Jiwen Yu *, Yiran Qin *, Haoxuan Che * (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-21853.txt.

Introduction

Jiwen Yu and a team of co-authors survey interactive generative video, in two thousand twenty-five. Such systems make video that users can steer as it plays, answering control signals with feedback. Demand for high-quality interactive video is growing across games, training, and driving tests.

The survey maps uses in three fields: games with endless worlds, embodied agents trained in synthetic scenes, and self-driving tests in closed loops. It splits an ideal system into five parts: generation, control, memory, dynamics, and intelligence. For each part, the authors review hard problems such as real-time speed, long-term coherence, and true physics. They close with directions for future work.

Catalog record

CREATORJiwen Yu *, Yiran Qin *, Haoxuan Che *
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE76 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-21853

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Survey of Interactive Generative Video Thanks: * indicates equal contributions. † indicates corresponding authors. Jiwen Yu, Yiran Qin, and Xihui Liu are with the The University of Hong Kong, Pok Fu Lam, Hong Kong. Quande Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Kun Gai are with Kuaishou Technology, Shenzhen, China. Haoxuan Che and Hao Chen are with The Hong Kong University of Science and Technology (HKUST), Hong Kong. Jiwen Yu *, Yiran Qin *, Haoxuan Che * Abstract Interactive Generative Video (IGV) has emerged as a crucial technology in response to the growing demand for high-quality, interactive video content across various domains. In this paper, we define IGV as a technology that combines generative capabilities to produce diverse high-quality video content with interactive features that enable user engagement through control signals and responsive feedback. We survey the current landscape of IGV applications, focusing on three major domains: 1) gaming, where IGV enables infinite exploration in virtual worlds; 2) embodied AI, where IGV serves as a physics-aware environment synthesizer for training agents in multimodal interaction with dynamically evolving scenes; and 3) autonomous driving, where IGV provides closed-loop simulation capabilities for safety-critical testing and validation. To guide future development, we propose a comprehensive framework that decomposes an ideal IGV system into five essential modules: Generation, Control, Memory, Dynamics, and Intelligence. Furthermore, we systematically analyze the technical challenges and future directions in realizing each component for an ideal IGV system, such as achieving real-time generation, enabling open-domain control, maintaining long-term coherence, simulating accurate physics, and integrating causal reasoning. We believe that this systematic analysis will facilitate future research and development in the field of IGV, ultimately advancing the technology toward more sophisticated and practical applications. I Introduction The demand for high-quality interactive video has grown significantly in recent years across various domains, from digital entertainment to industrial applications. This trend reflects the expanding role of video technology in applications such as simulation, decision-making, and content creation. Meanwhile, video generation technology has witnessed remarkable progress [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 ] , driven by recent advances in generative modeling paradigms, particularly diffusion models [ 12 , 13 , 14 , 15 ] and next-token prediction approaches [ 16 , 17 , 18 ] . Modern video generation systems can now produce remarkably realistic outputs while offering precise control over the generated content, enabling unprecedented opportunities across different fields. Motivated by these emerging needs and rapid technological developments, we present a comprehensive survey of interactive generative video technology in this paper. To establish a foundation for our discussion, we first introduce the concept of Interactive Generative Video (IGV), which is characterized by two key aspects. First, it is fundamentally generative, using generative models trained on vast video datasets to produce diverse, high-quality, open-domain video content. Second, it is inherently interactive, enabling online user engagement through control signals and responsive feedback, allowing users to achieve specific tasks or experiences through their interactions. Following our definition of IGV, we present in Fig. 1 the evolution and development trajectory of three major IGV applications: Gaming, Embodied AI, and Autonomous Driving. In gaming applications [ 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 ] , video games inherently combine visual output with player interaction, aligning perfectly with the core characteristics of IGV. IGV enables the creation of infinitely explorable interactive virtual worlds, where game content can be dynamically generated and personalized based on individual player preferences and skill levels. Moreover, IGV’s generative capabilities significantly streamline game development by eliminating manual asset creation processes, substantially reducing development costs and improving efficiency. Notable examples include Oasis [ 22 ] and WHAM [ 26 ] , both of which have released publicly playable versions that, although early-stage, demonstrate the preliminary exploration of IGV’s potential in gaming. In the field of embodied AI [ 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 ] , IGV plays a crucial role in creating realistic and interactive simulations for robotic systems. It enables high-fidelity video sequence generation for task planning and visualization, allowing robots to better understand and interact with their environments. IGV also addresses the challenge of insufficient training data by providing diverse synthetic scenarios, thus improving policy learning and helping robots generalize across different tasks and environments. For autonomous driving [ 49 , 50 , 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 , 62 , 63 , 64 ] , IGV offers advanced simulation capabilities that surpass traditional physics-based simulators in capturing real-world complexity. It generates high-fidelity video simulations conditioned on various control inputs, enabling comprehensive training across diverse driving scenarios. In addition, IGV enhances real-time decision making by predicting environmental changes and potential hazards, while also providing a safe platform to test autonomous systems in rare or dangerous situations. Fig. 1: Evolutionary tree of Interactive Generative Video (IGV) models from 2020 to 2025. This diagram categorizes the development of IGV research from three major application domains: Game Simulation, Embodied AI, and Autonomous Driving, each represented by a colored trunk. While IGV has demonstrated promising applications across gaming, embodied AI, and autonomous driving, to better guide its future development in these domains, we propose a comprehensive framework that outlines the essential components of an ideal IGV system, as illustrated in Fig. 3 . This framework helps identify key challenges and development directions through five fundamental modules: The Generation module focuses on fundamental video generation capabilities; despite significant advances in generation quality, it still faces challenges in achieving real-time performance and frame-by-frame autoregressive generation. The Control module manages user interactions with the virtual world, with the primary challenge of enabling precise control while generalizing to open-domain scenarios. The Memory module ensures consistency in generated content across both static and dynamic aspects, struggling with maintaining long-term coherence. The Dynamics module simulates physical laws within the virtual world, with challenges in accurately simulating various physical phenomena and enabling precise tuning over specific physical parameters. The Intelligence module integrates causal reasoning capabilities, representing a higher level of intelligence that could potentially lead IGV to evolve into a self-evolving metaverse. In this paper, we make three main contributions. First, we provide a comprehensive survey of Interactive Generative Video applications and their current development status across various domains, including Gaming, Embodied AI, and Autonomous Driving. Second, we propose a systematic framework that decomposes an ideal IGV system into five essential components, offering a structured approach to understand and develop IGV systems. Finally, based on our proposed framework, we systematically analyze the technical challenges in realizing each component of an ideal IGV system, thereby providing clear directions for future research in this emerging field. The structure of this survey is as follows. Sec. II presents the fundamentals technology of video generation, covering models like VAE, GAN, diffusion models, autoregressive models, and their hybrid forms. Sec. III

Read the complete text