CameraCtrl II: Dynamic Scene Exploration via Camera-controlled Video Diffusion Models
Hao He Ceyuan Yang Shanchuan Lin Yinghao Xu Meng Wei Liangke Gui, Qi Zhao Gordon Wetzstein Lu Jiang Hongsheng Li · 2025
Read CameraCtrl II: Dynamic Scene Exploration via Camera-controlled Video Diffusion Models by Hao He Ceyuan Yang Shanchuan Lin Yinghao Xu Meng Wei Liangke Gui, Qi Zhao Gordon Wetzstein Lu Jiang Hongsheng Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-10592.txt.
Introduction
Hao He and co-authors present a 2025 paper on video made by computer models that follow a moving camera. Older tools lost motion and could show only a narrow range of views when the camera moved a long way. Their work aims to let a user roam through a lively scene in a video.
The team first built training clips with strong motion and full camera notes. A small added part feeds camera data into the model without killing the motion of the base model. Users can then set one camera path after another to extend the scene. Tests across many scenes showed wide, smooth roaming that stayed coherent.
Catalog record
| CREATOR | Hao He Ceyuan Yang Shanchuan Lin Yinghao Xu Meng Wei Liangke Gui, Qi Zhao Gordon Wetzstein Lu Jiang Hongsheng Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-10592
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CameraCtrl II: Dynamic Scene Exploration via Camera-controlled Video Diffusion Models Hao He Ceyuan Yang Shanchuan Lin Yinghao Xu Meng Wei Liangke Gui, Qi Zhao Gordon Wetzstein Lu Jiang Hongsheng Li Abstract This paper introduces CameraCtrl II , a framework that enables large-scale dynamic scene exploration through a camera-controlled video diffusion model. Previous camera-conditioned video generative models suffer from diminished video dynamics and limited range of viewpoints when generating videos with large camera movement. We take an approach that progressively expands the generation of dynamic scenes—first enhancing dynamic content within individual video clip, then extending this capability to create seamless explorations across broad viewpoint ranges. Specifically, we construct a dataset featuring a large degree of dynamics with camera parameter annotations for training while designing a lightweight camera injection module and training scheme to preserve dynamics of the pretrained models. Building on these improved single-clip techniques, we enable extended scene exploration by allowing users to iteratively specify camera trajectories for generating coherent video sequences. Experiments across diverse scenarios demonstrate that CameraCtrl II enables camera-controlled dynamic scene synthesis with substantially wider spatial exploration than previous approaches. Figure 1 : Illustration of CameraCtrl II . Our camera-controlled video diffusion model generates consistent video sequences for dynamic scenes based on user-defined camera trajectories. The first row represents a generated video clip conditioned on the starting image and a user input camera trajectory. After watching the generated video clip, user can decide next step and specify the corresponding camera trajectories. Subsequent rows show clips conditioned on previous generated videos and these newly provided camera trajectories. The model strictly follows these user camera trajectory inputs while maintaining scene consistency across multiple video clips, enabling seamless navigation around pedestrians and exploration of the environment from various perspectives. 1 Introduction Recent years have witnessed remarkable advances in video diffusion models [ 6 , 9 , 29 ] , which can generate high-fidelity and temporally coherent videos from text descriptions. These models accept user-defined control [ 18 , 45 , 21 , 16 ] and are also scalable w.r.t. dataset size and computational resources, producing long and physically plausible videos. For example, Sora [ 9 ] can generate minute-long videos with realistic physics and complex motions. Therefore, these video diffusion models have become a promising tool for modeling and simulating dynamic real-world scenes. Beyond generating individual dynamic scenes, enabling users to actively explore these digital worlds has become increasingly important. Recent works have made progress in learning to explore generated spaces. In the domain of game generation, methods like [ 59 , 14 , 41 , 52 , 52 ] learn to simulate state transitions and predict future observations from action sequences such as keyboard inputs. For general video generation, camera control has emerged as a natural interface for scene exploration. Recent works [ 21 , 56 , 2 , 54 , 3 , 30 , 55 , 2 ] inject camera parameters into pretrained video diffusion models to enable precise camera viewpoint manipulation. By controlling virtual camera movements within these generated environments—analogous to camera navigation in the real world—users can explore these generated digital scenes from various perspectives. Despite their effectiveness in camera control and achieving the exploration in certain spatial range, existing methods face two limitations that hinder their practical applications. First, after incorporating camera control, these models often suffer from significant degradation in generating dynamic content. Second, they are restricted to generating short video clips (e.g., 25 frames for CameraCtrl [ 6 ] , 49 frames for AC3D [ 2 ] ) and cannot generate new clips in the same scene based on previously generated content and new camera trajectories given by users. These limitations fundamentally limit both the types of scenes that can be generated (constrained to largely static content) and the spatial range that can be explored, thus significantly diminishing the user experience. We introduce CameraCtrl II to address these two limitations. We address the challenge of generating highly dynamic videos using two key techniques. First, existing approaches primarily rely on static video datasets with camera parameter annotations, such as RealEstate10K [ 62 ] and DL3DV10K [ 35 ] . Training on these datasets inevitably compromises the dynamic capabilities of camera-controlled video diffusion models. Therefore, we construct a new dataset by extracting camera trajectory annotations from real dynamic videos using Structure-from-Motion (SfM), specifically VGGSfM [ 53 ] . Then, we propose methods to address challenges of arbitrary scale and long-tailed camera trajectory distributions in the constructed dataset. Second, for the model architecture, we inject camera parameters only at the initial layer of the diffusion model, avoiding over-constraining pixel generation to preserve dynamic content. Besides, we jointly train all model parameters on both labeled and unlabeled videos to preserve the pretrained model’s capability for generating dynamic and diverse scenes while maintaining its ability to perform general video generation tasks, such as text-to-video generation without camera input. This strategy also enables camera classifier-free guidance [ 22 ] during inference to enhance camera control accuracy. To enable exploration of broader scene ranges, we carefully design a video extension scheme and corresponding training strategy that allows our model to generate multiple coherent video clips sequentially. Specifically, we extend our single-clip camera-controlled video diffusion model to support clip-wise autoregressive video generation with a novel technique. During training, the model learns to generate new clips by conditioning on clean frames from previous clips and new camera trajectories, while optimizing only on the newly generated frames. At inference time, the model can generate new video segments by conditioning on both the previous clip frames and new camera trajectories, maintaining visual consistency while following user desired camera path. To preserve the high-quality generation capability of single clips, we jointly train this video extension task with the original single-clip camera-controlled video generation. Fig. 1 shows an example of our CameraCtrl II model generating multiple video sequences in a dynamic urban scene. It maintains consistent scene content while strictly adhering to camera controls, allowing for diverse exploration patterns such as navigating around pedestrians or changing direction while preserving object motion. In summary, our key contributions include: 1) A systematic data curation pipeline for constructing a dynamic video dataset with camera trajectory annotations; 2) A lightweight camera control injection module and corresponding training strategy that preserves dynamic video generation capabilities while adding camera control effect; 3) A clip-wise autoregressive generation recipe that enables extended range exploration of generated scenes. 2 Related Work Video Diffusion Models. Driven by the advances in model architectures [ 23 ] , large-scale datasets [ 5 , 10 , 57 ] , comprehensive benchmarks [ 26 , 27 ] , and improved training techniques [ 28 , 37 ] , the field of video diffusion models has seen remarkable progress in recent years. A major focus of this field has been text-to-video (T2V) generation [ 7 , 24 , 46 , 6 , 33 , 39 , 9 , 58 , 49 ] , where models create videos from text descriptions. Early works [ 7 , 6 , 17 , 19 , 24 ] efficiently transform UNet-based text-to-image (T2I) models into video generators by incorporating additional temporal modeling layers. Recent models [ 9 , 49 , 29 , 58 , 1 ,