Yan : Foundational Interactive Video Generation
Yan Team · 2025
Read Yan : Foundational Interactive Video Generation by Yan Team (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-08601.txt.
Introduction
Deheng Ye and co-authors present a framework for playable generated video in 2025. Interactive video that answers a player's actions in real time needs speed, control, and editing in one system. Their framework splits that job into three modules.
A compressed autoencoder plus a windowed denoising scheme runs game-like simulation at full high definition and sixty frames per second. A caption method feeds game knowledge into a video diffusion model, making it frame-wise controllable without end. A hybrid editor separates game mechanics from rendering, so text can restyle content mid-play. The team pitches the stack as a base for new creative tools.
Catalog record
| CREATOR | Yan Team |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-08601
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Yan : Foundational Interactive Video Generation Yan Team Abstract We present Yan , a foundational framework for interactive video generation, covering the entire pipeline from simulation and generation to editing. Specifically, Yan comprises three core modules. AAA-level Simulation : We design a highly-compressed, low-latency 3D-VAE coupled with a KV-cache-based shift-window denoising inference process, achieving real-time 1080P/60FPS interactive simulation. Multi-Modal Generation : We introduce a hierarchical autoregressive caption method that injects game-specific knowledge into open-domain multi-modal video diffusion models (VDMs), then transforming the VDM into a frame-wise, action-controllable, real-time infinite interactive video generator. Notably, when the textual and visual prompts are sourced from different domains, the model demonstrates strong generalization, allowing it to blend and compose the style and mechanics across domains flexibly according to user prompts. Multi-Granularity Editing : We propose a hybrid model that explicitly disentangles interactive mechanics simulation from visual rendering, enabling multi-granularity video content editing during interaction through text. Collectively, Yan offers an integration of these modules, pushing interactive video generation beyond isolated capabilities toward a comprehensive AI-driven interactive creation paradigm, paving the way for the next generation of creative tools, media, and entertainment. The project is shown at: here . Figure 1: Comprehensive capabilities of Yan . Yan supports real-time interactive video generation, with all interactions driven by user input. It offers a wide range of capabilities, including AAA-level simulations, multi-modal generation, and multi-granularity editing. Notably, during the editing process, users can dynamically modify prompts to edit subsequent generated content interactively. 1 Introduction Recent breakthroughs in Artificial Intelligence Generated Content (AIGC) have evolved from text ( Yu et al. 2024 ; Li et al. 2024 ; Zhao et al. 2023 ) and images ( Rombach et al. 2022b ) to video synthesis ( Brooks et al. 2024 ) , now reaching a transformative frontier: interactive generative video (IGV) ( Yu et al. 2025a ) . Unlike conventional video generation, IGV dynamically responds to user inputs, enabling real-time, personalized visual experiences. This paradigm holds the potential to revolutionize fields that demand ongoing human-AI interaction, ranging from immersive media and virtual simulation to embodied intelligence. However, it confronts significant challenges: existing methods ( Alonso et al. 2024 ; Bruce et al. 2024 ; Parker-Holder et al. 2024 ) struggle to simultaneously attain high visual fidelity, sustained temporal coherence, and rich interactivity. Moreover, generated content typically remains static post-creation, lacking real-time adaptability or user customization. The challenges manifest acutely in complex interactive scenarios. Consider modern video games: they demand not only 1080P/60FPS rendering and intricate physics simulation, but also granular response to diverse inputs and dynamic content adaptation. These requirements make games an ideal testbed for IGV capabilities. However, existing methods remain with critical gaps. For instance, GameNGen ( Valevski et al. 2024 ) , PlayGen ( Yang et al. 2024 ) , and MineWorld ( Guo et al. 2025 ) suffer from limited visual quality and generalizability. Meanwhile, The-Matrix ( Feng et al. 2024 ) , GameFactory ( Yu et al. 2025b ) , and Matrix-Game ( Zhang et al. 2025 ) lack intricate physics simulation and fall short of real-time performance. Besides, these methods treat interactive video as fixed content generation, unable to support dynamic modification during interaction, severely restricting practical creativity and flexibility. Thus, three core challenges remain unresolved: (1) achieving high-fidelity, real-time visual experience; (2) enabling generalizable and prompt-controllable generation; and (3) empowering dynamic, interactive editing and on-the-fly content customization during interaction. To tackle these challenges, we propose Yan , a foundational framework for interactive video generation, integrating three targeted modules trained on a shared dataset from modern 3D game environments 1 1 1 https://en.wikipedia.org/wiki/Yuan_Meng_Star . Specifically, for the first challenge of real-time, high-fidelity visual experience, our AAA-level Simulation module leverages a high-compression, low-latency 3D-VAE together with KV-cache-based shift-window denoising inference process, achieving unprecedented performance at 1080P/60FPS while preserving complex mechanics. To address the second challenge of prompt-controllable and generalizable content generation, our multi-modal generation module introduces a hierarchical captioning method that injects game-specific knowledge into open-domain video diffusion models, and further transforms them for frame-wise, action-controllable generation. This supports real-time interactive content creation driven by both text and image prompts across diverse domains. Finally, to tackle interactive editing, our multi-granularity editing module adopts a hybrid architecture that explicitly disentangles interactive mechanics simulation from visual rendering, supporting coarse-to-fine control and real-time editing of video content at any moment during interaction. The comprehensive capabilities of Yan are illustrated in Fig. 1 . We believe Yan marks a milestone—moving interactive video generation from fragmented prototypes toward a coherent, and truly generative paradigm, and charting a path for the next generation of open, creative, AI-powered digital worlds. 2 Related Work Interactive Generative Video Prior work falls into two lines. (i) Game-centric interactive video generation transfers action control and structural priors learned from game datasets to open-ended settings ( Yu et al. 2025b ; Guo et al. 2025 ; Zhang et al. 2025 ; Feng et al. 2024 ; Bruce et al. 2024 ; Parker-Holder et al. 2024 ) . These methods typically rely on action annotations and demonstrate some generalization beyond the source game, but they are often not truly frame-wise interactive (chunked control or high latency), emphasize navigation over accurate physics, and run at modest resolutions and real-time capability. (ii) Real-world world models aim to predict future frames or trajectories under control ( Bar et al. 2025 ; Yang et al. 2023 ; Agarwal et al. 2025 ) , but they generally do not provide real-time, frame-wise action responsiveness with promptable content control. In contrast, Yan targets real-time, causal, frame-wise control with accurate physics, high visual fidelity, and prompt-controllable generation in open domains. Game Simulation with Neural Networks Works in this category aim to simulate a game using neural networks. MarioVGG ( Protocol 2024 ) generates game video segments of Super Mario Bros ( Pinto 2021 ) from text-based actions and the first frame of the game, but it lacks real-time capabilities. GameNGen ( Valevski et al. 2024 ) uses a diffusion model to simulate games at 20 FPS on Doom ( Kempka et al. 2016 ) . PlayGen ( Yang et al. 2024 ) achieves 20 FPS on both Doom and Super Mario Bros ( Amidos 2019 ) . Oasis ( Decart et al. 2024 ) is able to simulate Minecraft ( Studios 2011 ) at 20 FPS. However, all the above approaches are constrained to relatively low-resolution results and limited frame rates, and are mainly tested in simplified or classic 2D/3D game environments. In contrast, Yan targets a modern 3D game Yuan Meng Star ( Tencent 2023 ) , achieving 1080P rendering at 60 FPS while preserving intricate game mechanics, expanding the capabilities and application range of game simulation. Video Editing. Diffusion-based video editing methods transform a given source video while preserving spatiotemporal coherence ( Brooks et al. 2024 ; Wan et al. 2025a ; Kong et al. 2024 ; Ceylan et al. 2023 ; Liu et al. 2024 ; Jiang et al. 2025 ) . However, these approaches assume non-interactive inputs and do not ensure that edited content remains responsive to user actions in real time. Within interactive video generation, most efforts prioritize high-fidelity simulation ( Valevski et al. 2024 ; Decart et al.