Real2SAM2Real: Generative 3D Caches as Complementary Context for Video Diffusion
Jiayi Wu, Haoming Cai, Cornelia Fermuller, Christopher Metzler, Yiannis Aloimonos · 2026
Read Real2SAM2Real: Generative 3D Caches as Complementary Context for Video Diffusion by Jiayi Wu, Haoming Cai, Cornelia Fermuller, Christopher Metzler, Yiannis Aloimonos (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-00299.txt.
Introduction
Wu and co-authors feed video makers a coarse 3D cache to keep form true across views. Video spread tools make fine clips but lose grip on moves. Wide view shifts, blocks, and fast acts bare spots the first frame hid. Pure priors can warp shapes, while bent guides can mislead.
Their route, Real2SAM2Real, builds full mesh caches from one base frame. Dream-based lift fills in back sides no frame shows. Path maps from set cam and move paths then guide the clip build. The base frame still sets the look of each thing. Soft place-matched feeds blend guide and prior with light touch. Training uses some 2,000 lone-eye clips with masked path maps. Tests show firm form and top marks for control.
Catalog record
| CREATOR | Jiayi Wu, Haoming Cai, Cornelia Fermuller, Christopher Metzler, Yiannis Aloimonos |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.00299
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Real2SAM2Real: Generative 3D Caches as Complementary Context for Video Diffusion Jiayi Wu, Haoming Cai, Cornelia Fermuller, Christopher Metzler, Yiannis Aloimonos Abstract While Video Diffusion Models (VDMs) excel at synthesizing high-fidelity videos, enabling precise camera and scene control remains challenging. Existing methods predominantly rely on implicit diffusion priors to "hallucinate" unobserved regions, inevitably leading to structural collapse during high-dynamic movements or complex occlusions where a large portion of the scene is unobserved. To address this challenge, we propose Real2SAM2Real, a framework that leverages 3D lifting models (e.g., SAM3D) to extract an explicitly editable 3D cache, serving as a robust geometric scaffold/conditioning for the video diffusion model. By capturing the entire 3D volume of foreground entities rather than just their unclosed visible shells, this cache injects holistic spatial priors into the VDM, providing dependable 3D-aware guidance for complex scene dynamics. To enable the model to effectively leverage this 3D guidance while maximally preserving its powerful pre-trained priors, we design a Soft Spatial-Aligned Injection mechanism alongside a minimally invasive fine-tuning strategy tailored for VDMs. Furthermore, we employ masked normal maps as a cross-modal bridge to construct a 3D-free data curation and perturbation pipeline. Extensive experiments demonstrate that Real2SAM2Real enables precise and decoupled control over both camera trajectories and multi-entity motions. Effectively utilizing the complementary context from generative 3D caches, our framework overcomes the typical breakdowns caused by an over-reliance on diffusion priors, maintaining exceptional spatiotemporal consistency even under large camera shifts and severe occlusions. Crucially, by decoupling geometry from appearance, our VDM-tailored 3D cache eradicates the perspective ambiguities caused by structural holes and erroneous facades, as well as the misleading cues arising from reflections and refractions—artifacts that commonly plague traditional "warp-and-inpaint" pipelines. Project website is available at https://jiayi-wu-leo.github.io/real2sam2real Figure 1: Versatile 3D-Aware Video Generation via Real2SAM2Real. (Left) Our pipeline overview: starting from a single reference image, we extract an editable, instance-complete 3D geometric cache. This interactive proxy allows for intuitive spatial modifications before being injected into a Video Diffusion Model. (Right) Real2SAM2Real unlocks a wide array of downstream applications, including precise camera control, independent 3D entity manipulation, seamless object removal, and motion transfer. As demonstrated in the results, our framework exhibits remarkable robustness against severe occlusions, perspective ambiguities, and complex non-Lambertian dynamics (e.g., reflections, refractions, flames), ensuring exceptional spatiotemporal coherence. 1 Introduction Recent advancements in Video Diffusion Models (VDMs) [ 34 , 41 , 14 , 21 , 4 ] have achieved unprecedented success in synthesizing high-fidelity videos. Trained on massive datasets, these models inherently encode rich spatiotemporal priors to simulate real-world dynamics. Leveraging these priors, recent research seeks to ’tame’ VDMs via post-training [ 47 , 16 , 26 ] to achieve production-level controllability, particularly for precise camera trajectories and entity motion manipulation. However, their fundamental reliance on 2D video data introduces a critical limitation. While an ideal world model should simulate complete 4D spacetime, videos intrinsically capture dynamics solely on a projected 2D plane. This inherent dimensional loss severely restricts geometric fidelity and spatiotemporal consistency, frequently causing these models to struggle in complex dynamic scenarios and profoundly undermining their capacity for explicit 3D-aware control. To bridge this gap, recent research explores two paradigms. The first fine-tunes VDMs with parameterized camera embeddings [ 15 , 37 , 1 , 33 , 24 , 2 ] . However, introducing these unseen signals entails substantial overhead, risks degrading pre-trained priors, and suffers from scale ambiguity. The second paradigm utilizes explicit pixel-wise 3D reconstructions (e.g., point clouds) via a "warp-and-inpaint" strategy [ 45 , 44 , 31 , 32 , 43 , 23 ] . While this reduces the modality gap, relying on incomplete geometry introduces cascading artifacts driven by perspective ambiguities (e.g., geometric voids, distorted facades) and misleading geometry in non-lambertian regions like reflections or refractions. Crucially, both paradigms fail to provide reliable 3D context for unobserved regions, forcing VDMs to "hallucinate" missing content and inevitably causing structural breakdowns during complex dynamics. Motivated by these limitations, we distill the intrinsic capabilities of VDMs into three core conclusions. First, while VDMs encode robust 2D temporal and semantic priors, navigating complex spatial dynamics strictly necessitates supplementary 3D-aware in-context guidance. Second, foreground entities with large parallax are highly susceptible to generative hallucinations and demand explicit geometric constraints, whereas backgrounds can rely entirely on inherent model priors. Third, compared to parameterized camera embeddings, scene-anchored visual-domain guidance aligns more naturally with human intention and enables rapid adaptation with minimal training data, thereby completely preserving the VDM’s zero-shot generalization capabilities without degradation. Building upon these insights, we introduce Real2SAM2Real , a highly 3D-controllable video generation pipeline that leverages instance-complete 3D geometry caches as complementary context (See Fig. 1 ). Starting from a single reference image, we utilize generative 3D lifting models (e.g., [ 8 , 39 , 20 , 38 , 42 ] ) to construct an editable coarse 3D proxy cache. This generative cache supplies complementary 3D geometric priors for viewpoint-sensitive foreground entities, serving as robust spatial anchors for the Video Diffusion Model (VDM) and facilitating intuitive manipulation of camera trajectories and scene layouts. Coupled with instance-masked normal maps as an intermediate representation and a minimally invasive condition injection mechanism, our framework can be efficiently adapted with minimal training samples while strictly preserving pre-trained priors. Extensive experiments reveal that Real2SAM2Real synthesizes geometrically coherent and high-quality videos with precise controllability, demonstrating remarkable robustness against conditions that natively break existing (See Fig. 2 )—such as large-scale dynamics, heavy occlusions, and complex non-Lambertian optics (e.g., reflections and refractions). The core contributions of this work can be summarized as follows: 1. We introduce an explicitly editable, instance-complete 3D geometry cache to serve as complementary in-context guidance for VDMs. This formulation effectively unlocks the high-dynamic generation capabilities inherent to VDMs while enabling intuitive, precise 3D instance manipulation and camera trajectory control. Serving as robust spatial anchors and generative geometric priors, our instance-complete 3D cache elegantly provides complementary 3D context while fundamentally circumventing the inherent flaws of pixel-level reconstructions—namely, the perspective ambiguities caused by structural holes and erroneous facades, and the cascading artifacts stemming from misleading geometric cues on non-Lambertian surfaces (e.g., reflections and refractions). 2. We employ sequences of instance-masked normal maps as a cross-modal bridge to inject the explicit 3D cache into the generative VDM. By offering high geometric information density and scale invariance while inherently excluding appearance details, this representation provides orthogonal guidance that fundamentally avoids feature competition. Benefiting from their ease of acquisition, we adopt a 3D-free data curation strategy that drives the automated construction of training datasets by relying exclusively on off-the-shelf 2D dense prediction and instance segmentation models, supplemented by an instance-level data perturbation strategy to effectively bridge the potential training-inference gap. 3. We design a Soft Spatial-Aligned Injection mechanism and a minimally invasive fine-tuning strategy tailored for decoupled appearance and coarse geometry. This paradigm maximizes the preservation of pre-trained priors and zero-shot generalization, provides inherent fault tolerance against the imperfections of current 3D generation models, and achieves rapid adaptation on few-shot data, ultimately rendering our framework highly democratized. 2 Background & Related Works Figure 2: Overcoming the limitations of warp-and-inpaint pipelines. Conventional point cloud warping struggles with geometry-appearance entanglement and unclosed representations. (a) Reflections & (b) Refractions: In non-Lambertian regions (e.g., mirrors and transparent boxes), traditional methods bake