VideoMemory: Toward Consistent Video Generation via Memory Integration

Jinsong Zhou 1 , 3 ∗ 1,3* Yihua Du 1 ∗ 1* Xinli Xu 1 ∗ † 1*\dagger Luozhou Wang 1 1 Zijie Zhuang 1 1 Yehang Zhang 1 1 Shuaibo Li 1 1 Xiaojun Hu 3 3 Bolan Su 3 3 Ying-cong Chen 1 , 2 ‡ 1,2\ddagger 1 HKUST(GZ) 2 HKUST 3 ByteDance Project Page: VideoMemory · 2026

Read VideoMemory: Toward Consistent Video Generation via Memory Integration by Jinsong Zhou 1 , 3 ∗ 1,3* Yihua Du 1 ∗ 1* Xinli Xu 1 ∗ † 1*\dagger Luozhou Wang 1 1 Zijie Zhuang 1 1 Yehang Zhang 1 1 Shuaibo Li 1 1 Xiaojun Hu 3 3 Bolan Su 3 3 Ying-cong Chen 1 , 2 ‡ 1,2\ddagger 1 HKUST(GZ) 2 HKUST 3 ByteDance Project Page: VideoMemory (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-03655.txt.

Introduction

Jinsong Zhou and nine co-authors present VideoMemory, a framework for keeping characters and settings consistent across a long generated video. Today's models make fine short clips but often change a character's face or lose props when scenes cut or time passes. VideoMemory plans the story first, then generates each shot with help from a Dynamic Memory Bank.

That bank stores visual and written descriptors for characters, props, and backgrounds, updating after every shot as the story changes them. A multi-agent system breaks the script into shots, retrieves the right memories, and synthesizes frames and video from them. The team built a fifty-four item test set, described in the work, for this long-form setting. Consistent long video is a key step toward AI-made narrative film.

Catalog record

CREATORJinsong Zhou 1 , 3 ∗ 1,3* Yihua Du 1 ∗ 1* Xinli Xu 1 ∗ † 1*\dagger Luozhou Wang 1 1 Zijie Zhuang 1 1 Yehang Zhang 1 1 Shuaibo Li 1 1 Xiaojun Hu 3 3 Bolan Su 3 3 Ying-cong Chen 1 , 2 ‡ 1,2\ddagger 1 HKUST(GZ) 2 HKUST 3 ByteDance Project Page: VideoMemory
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-03655

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VideoMemory: Toward Consistent Video Generation via Memory Integration Jinsong Zhou 1 , 3 ∗ 1,3* Yihua Du 1 ∗ 1* Xinli Xu 1 ∗ † 1*\dagger Luozhou Wang 1 1 Zijie Zhuang 1 1 Yehang Zhang 1 1 Shuaibo Li 1 1 Xiaojun Hu 3 3 Bolan Su 3 3 Ying-cong Chen 1 , 2 ‡ 1,2\ddagger 1 HKUST(GZ) 2 HKUST 3 ByteDance Project Page: VideoMemory Abstract Maintaining consistent characters, props, and environments across multiple shots is a central challenge in narrative video generation. Existing models can produce high-quality short clips but often fail to preserve entity identity and appearance when scenes change or when entities reappear after long temporal gaps. We present VideoMemory, an entity-centric framework that integrates narrative planning with visual generation through a Dynamic Memory Bank. Given a structured script, a multi-agent system decomposes the narrative into shots, retrieves entity representations from memory, and synthesizes keyframes and videos conditioned on these retrieved states. The Dynamic Memory Bank stores explicit visual and semantic descriptors for characters, props, and backgrounds, and is updated after each shot to reflect story-driven changes while preserving identity. This retrieval–update mechanism enables consistent portrayal of entities across distant shots and supports coherent long-form generation. To evaluate this setting, we construct a 54-case multi-shot consistency benchmark covering character-, prop-, and background-persistent scenarios. Extensive experiments show that VideoMemory achieves strong entity-level coherence and high perceptual quality across diverse narrative sequences. Figure 1 : From a single story prompt, VideoMemory generates coherent multi-shot videos using dynamic Character, Prop, and Background Memory Banks. This ensures exceptional entity consistency, as seen with the feather prop remaining perfectly stable across distant shots (e.g., 2, 10, and 12) despite significant scene and viewpoint variations. † † footnotetext: ∗ Equal contribution. † Project leader. ‡ Corresponding author. 1 Introduction Modern text-to-video generation has made rapid progress in recent years, with large-scale diffusion transformers 10 ; 30 ; 29 and video generative models 17 ; 33 ; 31 ; 25 ; 7 capable of producing short clips with high visual fidelity, smooth motion, and diverse appearance styles from open-domain prompts. Beyond standalone clips, this progress has sparked growing interest in story-driven video generation, where a concise script is translated into a coherent sequence of scenes and shots that follow a narrative structure and can be directly used for pre-visualization, storyboarding, and content creation. However, realizing such story-level generation imposes far stricter requirements than conventional short video synthesis: main and supporting characters, signature props, and key environments must remain consistent across distant shots under varying viewpoints, lighting conditions, and narrative contexts, while only exhibiting intentional, story-driven changes. Recent works have explored several directions toward more coherent story-level video generation. Plan-then-generate pipelines 19 ; 12 ; 44 use large language models to expand a brief prompt into structured scene–shot scripts, providing explicit narrative planning and improving control over the presence and arrangement of entities. One-pass multi-shot generation methods 9 ; 35 ; 16 ; 24 model multiple shots jointly or sequentially, leveraging extended temporal context to improve cross-shot coherence. In parallel, memory-inspired approaches 42 ; 41 ; 37 manage long-range dependencies by compressing historical frames, aligning viewpoints across time, or maintaining persistent scene-level representations for retrieval. Despite this progress, long-range entity consistency remains a core challenge in narrative video generation. Plan-then-generate pipelines provide narrative structure but lack persistent visual representations, causing each shot to be generated without reference to past appearances. One-pass multi-shot generation methods capture broader temporal context, yet their histories are feature-centric and do not preserve explicit entity states, leading to drift when characters or props reappear after long gaps. Memory-inspired approaches store information at the frame or feature level and offer no dedicated slots for named entities, making identity recovery unreliable once an entity leaves the local context window. Figure 2 : Comparison between existing methods and our method. (a) Existing methods lack entity-level memory, resulting in inconsistent appearances across shots. (b) VideoMemory retrieves and updates an entity-aware Dynamic Memory Bank during generation, enabling consistent multi-shot videos from text alone. We draw inspiration from human production workflows. In real film and television projects 2 ; 11 , production crews first define a set of story assets, including cast appearances, costumes, makeup, key props, and sets, and then update this inventory as the narrative unfolds so that recurring entities remain recognizable while reflecting plot-driven changes. This motivates an explicit, dynamically updated memory for key entities in story-centric video generation. This motivates an explicit, dynamically updated memory for key entities in story-centric video generation. In this work, we introduce VideoMemory , an entity-centric framework for script-to-multi-shot video generation built around a Dynamic Memory Bank. The Dynamic Memory Bank is an explicit, entity-aware external memory that stores and updates the evolving visual and semantic states of characters, props, and backgrounds, enabling long-range consistency and controlled narrative evolution. Given a short script, a Storyboard Agent expands it into structured scene and shot descriptions. A Memory Agent extracts the entities required for each shot, forms attribute-based keys, and interacts with the Dynamic Memory Bank, which stores explicit visual and semantic representations of characters, props, and backgrounds. Before generating a shot, the agent retrieves the corresponding entity states from the memory. A Visualization Agent then synthesizes keyframes and shot videos conditioned on these retrieved representations to ensure consistent appearance of recurring entities. After each shot is produced, the Memory Agent updates the Dynamic Memory Bank to incorporate narrative changes such as aging, clothing variations, or prop condition while preserving identity. Through this coordinated retrieve-and-update mechanism over a shared external memory, VideoMemory maintains coherent entity evolution across distant shots without relying on full-frame history caching. A qualitative illustration is shown in Figure 2 . To systematically assess entity-level consistency under diverse narrative settings, we further construct a dedicated multi-shot consistency benchmark comprising 54 structured cases that span character-, prop-, and background-persistent scenarios with varying shot lengths. The benchmark provides story-driven test situations that require models to preserve identity while accommodating intentional narrative changes. Extensive experiments on this benchmark demonstrate that VideoMemory achieves strong cross-shot coherence, stable long-range entity continuity, and high perceptual realism across all categories. Our contributions are three-fold: • We introduce VideoMemory , an entity-centric framework for script-to-multi-shot video generation built around a Dynamic Memory Bank that explicitly tracks and updates the visual and semantic states of narrative entities. • We develop a retrieval–and-update memory mechanism with dedicated slots for characters, props, and environments, enabling long-range entity consistency while enabling long-range entity consistency while allowing story-guided changes in their visual state. • We construct a multi-shot consistency benchmark with 54 story-driven cases covering character-, prop-, and background-persistent scenarios, providing the first structured evaluation protocol for long-range entity coherence in narrative videos. 2 Related Work 2.1 Story-Driven Video Generation Storyboard visualization aims to automatically generate visual imagery or videos from textual or conceptual inputs, serving as a crucial component in story planning and cinematic pre-visualization. With the rapid progress of generative models, this area has seen significant improvements in both visual quality and narrative coherence. A common line of work adopts a keyframe-first strategy, where representative frames are first synthesized and then expanded into complete videos using image-to-video (I2V) generation models. These approaches 45 ; 18 ; 23

Read the complete text