SmartDirector: Keyframe-Conditioned Cinematic Video Generation with Narrative Pacing Control

Zhida Zhang, Jie Ma, Zhan Peng, Haoxue Wu, Yang Han, Jun Liang, Jie Cao, Jing Li · 2026

Read SmartDirector: Keyframe-Conditioned Cinematic Video Generation with Narrative Pacing Control by Zhida Zhang, Jie Ma, Zhan Peng, Haoxue Wu, Yang Han, Jun Liang, Jie Cao, Jing Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27891.txt.

Introduction

Zhang and colleagues give video-generation models firmer control over story and pacing through multiple keyframes. Their SmartDirector first drafts a low-resolution video from the keyframes, then sharpens it using high-resolution keyframes as anchors for fine detail. Trained on sequences cut from movies, it beat existing methods by a wide margin in tests.

Catalog record

CREATORZhida Zhang, Jie Ma, Zhan Peng, Haoxue Wu, Yang Han, Jun Liang, Jie Cao, Jing Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49761
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.27891

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SmartDirector: Keyframe-Conditioned Cinematic Video Generation with Narrative Pacing Control Zhida Zhang, Jie Ma, Zhan Peng, Haoxue Wu, Yang Han, Jun Liang, Jie Cao, Jing Li Abstract The narrative quality of a video fundamentally determines its perceptual value. Although existing video generation methods can produce visually appealing content, they predominantly rely on sparse conditioning signals such as text prompts or first/last frames, which limits precise control over narrative structure and temporal pacing. In this paper, we propose SmartDirector, a framework that enhances the narrative capacity of video generation models through multiple keyframes. SmartDirector supports flexible generation scenarios including single-shot generation, multi-shot narrative synthesis, and video extension. The framework operates in two stages: Director-Gen generates a low-resolution video conditioned on the provided keyframes, and Director-SR refines the output by exploiting high-resolution keyframes as semantic anchors to recover fine-grained details. To enable robust multi-keyframe training, we construct a data pipeline that curates single-shot and multi-shot sequences from movies. Extensive experiments demonstrate that SmartDirector substantially outperforms existing state-of-the-art approaches. We will release the code to facilitate further research. 1]NLPR, CISIA 2]Youku Moku-Lab 3]HUST \project https://orange-3dv-team.github.io/SmartDirector/ 1 Introduction Figure 1 : Examples generated by SmartDirector. SmartDirector enables high-fidelity video generation guided by arbitrary keyframes. Recent advancements in video generation have propelled a paradigm shift from synthesizing short, single-shot clips [ 22 , 11 , 10 ] to creating long, multi-shot narratives [ 25 , 21 , 17 , 2 , 8 , 27 ] . Although existing methods have demonstrated remarkable capabilities in generating visually stunning and high-fidelity videos, they predominantly rely on sparse conditioning signals, such as text prompts or first/last frames. Consequently, these approaches struggle to achieve precise control over fine-grained spatial-temporal content and narrative structure, significantly restricting their practical utility in real-world applications. In professional filmmaking, directors use storyboards [ 26 ] to guide the production process and exercise fine-grained control over visual content. Storyboards serve as visual anchors that maintain coherence across multiple shots and regulate the temporal pacing (i.e., the rhythm and timing of visual content) within each individual shot. In this work, we identify keyframes as the direct counterpart of storyboards in video generation. Building on this perspective, we focus on the task of multi-keyframe-conditioned video generation. A naive approach is to treat each pair of adjacent keyframes as the start and end frames of a short clip, generate the clips autoregressively, and concatenate the results. However, this strategy neglects global context during synthesis, resulting in abrupt temporal discontinuities at keyframe boundaries and a loss of narrative consistency across the entire video. Recent work [ 16 , 14 ] proposes an alternative that inserts keyframes directly into noisy latents at their corresponding temporal positions before denoising with a video diffusion model. Yet this method is fundamentally limited by the causal structure of the temporal VAE [ 22 , 11 , 29 ] . In a standard 3D VAE, the first frame is encoded independently, while subsequent frames are encoded in groups (e.g., every four frames) with causal dependence on preceding frames. Direct latent replacement at arbitrary positions violates this causal dependency, producing temporal discontinuities and visual artifacts near the keyframes. In this paper, we introduce SmartDirector, a flexible framework for video generation guided by arbitrary keyframes that seamlessly supports both single-shot and multi-shot synthesis. Beyond keyframe-conditioned generation, SmartDirector also supports video-conditioned generation for video extension, as illustrated in Fig. 2 . To fully exploit the conditioning provided by multiple keyframes, the framework consists of two stages: a keyframe-conditioned generation stage and a keyframe-conditioned super-resolution stage, referred to as Director-Gen and Director-SR. In the Director-Gen stage, we propose a Multi-Chunk VAE strategy to address the causal limitation of the temporal VAE. During training, the video is partitioned into multiple chunks at the keyframe positions, with each keyframe serving as the first frame of its respective chunk and encoded independently by the VAE. The resulting multi-chunk latents are then processed by a Diffusion Transformer (DiT) [ 18 ] . To maintain global consistency, we apply full spatio-temporal attention within the DiT, enabling each chunk to attend to the global context across all chunks. Figure 2 : SmartDirector is a flexible framework that accommodates diverse input conditions and supports a wide range of generation tasks, including first-frame-to-video, last-frame-to-video, multi-shot synthesis, and video extension. Videos produced by the Director-Gen stage are typically low-resolution (e.g., 480p), which is lower than the resolution of the provided keyframes. To leverage the fine-grained details in the high-resolution keyframes, we design a keyframe-conditioned super-resolution module in the Director-SR stage that upsamples the generated video to high definition (e.g., 1080p), explicitly conditioned on the high-resolution keyframes. Training our framework requires carefully curated data. We construct a data processing pipeline for curating long video sequences, as illustrated in Fig. 4 . For the Director-Gen stage, we collect copyright-free movies and segment them into single shots. We then compute visual similarities between these shots to aggregate them into coherent multi-shot video sequences, which are further annotated with structured descriptions using Vision-Language Models (VLMs) [ 1 , 20 ] . The resulting dataset contains both single-shot and multi-shot sequences, enabling robust training for both generation settings. For the super-resolution task, we use the open-source UltraVideo dataset [ 28 ] . Our main contributions are summarized as follows: • We propose SmartDirector, a unified framework that enables flexible keyframe-conditioned video generation, covering single-shot, multi-shot, and video extension. • We identify the fundamental limitation imposed by the causal structure of the temporal VAE on keyframe insertion and propose a Multi-Chunk VAE strategy. This design circumvents the causal constraints, allowing keyframes to be placed at arbitrary temporal positions while ensuring smooth and continuous generation. • We design a keyframe-conditioned super-resolution module that exploits high-resolution keyframes as semantic anchors to recover fine-grained details. 2 Related Work 2.1 Video Generation Video generation has evolved rapidly from synthesizing short single-shot clips [ 22 , 11 , 10 ] to producing long multi-shot narratives [ 25 , 21 , 17 , 2 , 8 , 27 ] . However, these methods rely on sparse conditioning signals such as text prompts or the first/last frame, which limits their ability to control fine-grained spatial-temporal content and narrative structure. Recently, several approaches have attempted to incorporate multiple keyframes into the generation process to enable more precise control. For single-shot video generation, Pusa [ 16 ] injects noise of different timesteps into distinct frames, while DreaMontage [ 14 ] directly inserts keyframes into noisy latents at corresponding positions. For multi-shot video generation, CaptainCinema [ 27 ] generates video by conditioning on the first frames of each shot. However, these methods are limited to specific scenarios and lack the flexibility to support arbitrary keyframe placement for precise temporal and spatial control. In this work, we propose a unified framework that enables flexible keyframe control for both single-shot and multi-shot video generation. Additionally, our method supports video extension by using video frames as input to extend the content temporally. 2.2 Video Super Resolution Video super-resolution has been studied extensively over the past decades. Early approaches were predominantly based on GANs [ 7 , 4 ] , while recent methods have shifted toward diffusion models [ 23 , 6 , 30 ] . SeedVR [ 23

Read the complete text