MVTrack4Gen: Multi-View Point Tracking as Geometric Supervision for 4D Video Generation

JoungBin Lee, Jaewoo Jung, Jongmin Lee, Tongmin Kim, Hyunsung Kim, Takuya Narihira, Kazumi Fukuda, Jahyeok Koo, Jisang Han, Yuki Mitsufuji, Seungryong Kim · 2026

Read MVTrack4Gen: Multi-View Point Tracking as Geometric Supervision for 4D Video Generation by JoungBin Lee, Jaewoo Jung, Jongmin Lee, Tongmin Kim, Hyunsung Kim, Takuya Narihira, Kazumi Fukuda, Jahyeok Koo, Jisang Han, Yuki Mitsufuji, Seungryong Kim (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-26087.txt.

Introduction

JoungBin Lee and colleagues improve novel-view video generation with a tracking-based training signal. The task turns one reference video into a new video along a chosen camera path. Geometry and motion must both survive the move. Explicit 3D pipelines inherit errors from reconstruction tools. Camera-only conditioning looks great but drifts in geometry and motion.

Attention layers hold the clue. Certain layers already attend to geometrically matching points across views and time. Misaligned attention tracks with motion inconsistency in the output. MVTrack4Gen routes those features into an auxiliary multi-view point-tracking head. The diffusion model trains jointly on video generation and point tracking. Strengthened correspondences make outputs follow the reference motion better. Cross-view geometry holds up more reliably too. Across varied benchmarks, the method reaches state-of-the-art geometric consistency. Camera accuracy stays competitive with the best alternatives.

Catalog record

CREATORJoungBin Lee, Jaewoo Jung, Jongmin Lee, Tongmin Kim, Hyunsung Kim, Takuya Narihira, Kazumi Fukuda, Jahyeok Koo, Jisang Han, Yuki Mitsufuji, Seungryong Kim
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.26087

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MVTrack4Gen: Multi-View Point Tracking as Geometric Supervision for 4D Video Generation JoungBin Lee, Jaewoo Jung, Jongmin Lee, Tongmin Kim, Hyunsung Kim, Takuya Narihira, Kazumi Fukuda, Jahyeok Koo, Jisang Han, Yuki Mitsufuji, Seungryong Kim Abstract Synthesizing a novel-view video from a monocular reference video along a target camera trajectory requires both geometric consistency and motion fidelity with respect to the reference video. Existing methods based on explicit 3D representations are limited by the accuracy of off-the-shelf reconstruction modules, which often produce inaccurate geometry for dynamic objects in monocular videos. In contrast, camera-conditioning-only methods can achieve high visual quality but often struggle to preserve geometric and motion consistency. In this work, we introduce MVTrack4Gen ( M ulti- V iew point Track ing for Novel-View Gen eration), a motion-aware training framework that leverages multi-view point tracking as an additional geometric and motion supervision signal for camera-conditioning-only novel-view video diffusion models. Our key finding is that specific attention layers encode strong correspondence cues, where query features attend to key features at geometrically corresponding locations across views and over time, and the misalignment of these correspondences causes motion inconsistency. Based on this observation, we route these features into an auxiliary multi-view tracking head and jointly train the diffusion model with a point-tracking objective. By explicitly strengthening these motion-aware correspondences, MVTrack4Gen improves existing models to better follow the motion in the reference view and maintain cross-view geometric consistency. Across diverse benchmarks, our method achieves state-of-the-art geometric consistency and competitive camera accuracy. 2 2 footnotetext: Corresponding authors. Figure 1: MVTrack4Gen jointly generates a novel-view video and multi-view point tracks, given a monocular reference video with query points and a user-specified camera trajectory. Lifting both the reference and generated frames into 3D space using Depth Anything 3 30 shows that the target views and multi-view point tracks faithfully preserve the dynamic motion of the reference video while remaining geometrically consistent. 1 Introduction Novel-view video generation from a monocular video aims to synthesize a target-view video along a user-specified camera trajectory, with applications in virtual cinematography, robotics, and immersive AR/VR. To be useful in these real-world scenarios, the generated video should satisfy four requirements: (1) accurate camera control that faithfully follows the user-specified trajectory, (2) geometric consistency in which the scene structure of the reference video is preserved across the synthesized view, (3) motion consistency that maintains the dynamics of the reference video, and (4) a photorealistic visual appearance. The central challenge is achieving all four at once, since the model must respect the underlying 3D geometry and motion of the scene while still producing high-quality videos. Recent works leverage generative priors from pretrained video diffusion models 1 ; 5 ; 27 ; 40 ; 44 to address this task, achieving notable progress in dynamic scenes 36 ; 29 ; 45 ; 10 ; 43 ; 3 ; 33 ; 19 . These approaches largely fall into two paradigms. The first paradigm follows a reconstruct-then-generate pipeline 36 ; 29 ; 45 ; 10 ; 43 , which first reconstructs an explicit 3D representation from the reference video, projects this geometry along the target camera trajectory, and feeds the projected appearance as a spatial condition. By supplying explicit geometric guidance, this design relieves the model from having to jointly infer geometry and synthesize photorealistic frames. However, its quality hinges on the accuracy of the reconstructed geometry, which becomes challenging for dynamic scenes, where inaccurate dynamic-object geometry introduces distortions and flying-pixel artifacts near object boundaries. These artifacts corrupt the spatial condition and prevent the diffusion model from faithfully capturing the geometry and motion of dynamic objects. To avoid this dependence on error-prone reconstruction, a second and more recent line of work performs camera-conditioning-only generation 3 ; 33 ; 19 , without any explicit 3D representation. These methods feed the reference video to the diffusion model as additional input tokens and inject the target trajectory through camera embeddings such as Plücker coordinates 37 , and can be trained from pairs of multi-view videos. Because the reference and target views are processed jointly within the same 3D attention modules, cross-view and intra-view information is exchanged implicitly at every layer. This implicit design yields markedly more photorealistic results, but the lack of explicit geometric grounding leaves the model with a weak understanding of scene structure. As a result, dynamic objects are often placed at incorrect locations or assigned inconsistent motions, producing cross-view geometric and motion inconsistencies. In this work, we start from the observation that camera-conditioning-only methods already excel at photorealistic synthesis, and ask whether their geometric understanding can be improved without reintroducing explicit 3D conditioning. To this end, we analyze the 3D attention maps of these diffusion models 3 ; 33 across layers and denoising timesteps, where information is implicitly exchanged across views and frames. Inspired by 32 , which reveals emergent correspondence in the attention maps of video diffusion models, our analysis yields three key observations. First, query-key matching within the 3D attention blocks provides clear correspondence cues, capturing both intra-video temporal correspondences within each view and inter-video cross-view correspondences between the generated and reference views. Second, these temporal and cross-view correspondences become simultaneously prominent at specific intermediate layers, revealing which regions the model attends to when synthesizing each part of the generated frame. Third, in regions where dynamic objects exhibit geometric or motion inconsistencies, the attention maps at these layers exhibit incorrect cross-view correspondences—indicating that the quality of these correspondences directly governs the geometric consistency of the output. Building on this insight, we explore whether geometric and motion consistency can be improved by directly supervising the correspondences in these dominant attention layers. Since the motion of dynamic objects can be described by the trajectories of physical points over time 13 , we introduce MVTrack4Gen ( M ulti- V iew point Track ing for Novel-View Gen eration), a framework that leverages ground-truth multi-view point tracks as auxiliary supervision, where each track follows the same physical point within and across views. Specifically, we build a multi-view tracking head on top of local 4D correlation volumes computed from the query and key features of the selected attention layer, and jointly train it with the diffusion model. This encourages the model to encode motion-aware correspondences in its attention features, so that the target view more faithfully reflects the motion of the reference video. In addition, we introduce a multi-view correspondence loss that applies a cross-entropy objective directly to the attention map, encouraging each query token in the target view to attend to its corresponding ground-truth location. Together, we observe that these objectives improve both cross-view geometric consistency and intra-view temporal consistency of the generated novel-view videos. To validate its generality, we apply our method to two camera-conditioning-only backbones, ReCamMaster 3 and Redirector 33 , and evaluate on the DAVIS 34 and iPhone 15 benchmarks. Our method consistently improves both backbones, achieving the best scores on most VBench 20 visual-quality metrics while reaching state-of-the-art geometric consistency and camera accuracy comparable to both reconstruction-based and camera-conditioning-only baselines. Notably, it markedly improves geometric consistency and visual quality for dynamic objects, without requiring any explicit 3D reconstruction at inference time. 2 Related Work Novel-View Generation Using Explicit 3D Representation. Recent video diffusion models 7 ; 44 ; 40 have

Read the complete text