Towards Realistic and Consistent Orbital Video Generation via 3D Foundation Priors
Rong Wang, Ruyi Zha, Ziang Cheng, Jiayu Yang, Pulak Purkait, Hongdong Li, Australian National University Amazon · 2026
Read Towards Realistic and Consistent Orbital Video Generation via 3D Foundation Priors by Rong Wang, Ruyi Zha, Ziang Cheng, Jiayu Yang, Pulak Purkait, Hongdong Li, Australian National University Amazon (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-12309.txt.
Introduction
Wang and five colleagues generate orbital videos of an object from a single image, using shape priors from a 3D foundation model to keep views geometrically consistent. Pixel attention alone fails at long-range extrapolation such as rear views, where input correspondences run thin. Their method prompts generation with a global latent vector plus projected latent images for view-dependent detail, injected through a multi-scale 3D adapter. Tests show superior quality, shape realism, and multi-view consistency.
Catalog record
| CREATOR | Rong Wang, Ruyi Zha, Ziang Cheng, Jiayu Yang, Pulak Purkait, Hongdong Li, Australian National University Amazon |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32466 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.12309
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Realistic and Consistent Orbital Video Generation via 3D Foundation Priors Rong Wang, Ruyi Zha, Ziang Cheng, Jiayu Yang, Pulak Purkait, Hongdong Li, Australian National University Amazon Abstract We present a novel method for generating geometrically realistic and consistent orbital videos from a single image of an object. Existing video generation works mostly rely on pixel-wise attention to enforce view consistency across frames. However, such mechanism does not impose sufficient constraints for long-range extrapolation, e.g. rear-view synthesis, in which pixel correspondences to the input image are limited. Consequently, these works often fail to produce results with a plausible and coherent structure. To tackle this issue, we propose to leverage rich shape priors from a 3D foundational generative model as an auxiliary constraint, motivated by its capability of modeling realistic object shape distributions learned from large 3D asset corpora. Specifically, we prompt the video generation with two scales of latent features encoded by the 3D foundation model: (i) a denoised global latent vector as an overall structural guidance, and (ii) a set of latent images projected from volumetric features to provide view-dependent and fine-grained geometry details. In contrast to commonly used 2.5D representations such as depth or normal maps, these compact features can model complete object shapes, and help to improve inference efficiency by avoiding explicit mesh extraction. To achieve effective shape conditioning, we introduce a multi-scale 3D adapter to inject feature tokens to the base video model via cross-attention, which retains its capabilities from general video pretraining and enables a simple and model-agonistic fine-tuning process. Extensive experiments on multiple benchmarks show that our method achieves superior visual quality, shape realism and multi-view consistency compared to state-of-the-art methods, and robustly generalizes to complex camera trajectories and in-the-wild images. Figure 1 : In this work, we generate realistic and consistent orbital videos from a single object image. By leverage shape priors from 3D foundation model, our method can robustly produce consistent frames with more plausible structures compared to the baseline 42 . * * footnotetext: Work done while Rong Wang was a research intern at Amazon. HL is funded in part by ARC Grant DP220100800. 1 Introduction Generating realistic videos from an object image and camera trajectory enables a wide range of applications, including e-commerce, extended reality, and game design. To this end, it has received increasing attention ( 19 ; 8 ; 42 ; 47 ) . A key challenge in this task is to ensure all generated frames agree on a coherent and plausible object shape, especially for unseen objects and challenging viewing angles. To this end, we aim to generate consistent video frames with realistic object shapes in this work, thereby improving the model robustness desired by downstream applications. Existing video generation works 3 ; 4 ; 27 ; 50 have demonstrated impressive quality and fidelity by scaling large diffusion transformers 31 . While self-attention layers 41 in these transformer blocks can model view consistency between actively matching patches, they are ineffective for learning dependencies between frames with large viewpoint changes, e.g. input front view and target rear view, where pixel-wise correspondences are limited and unreliable. Without further 3D world knowledge, these methods often produce unrealistic results, containing artifacts of distorted and unnatural shapes as illustrated in Figure 1 . Meanwhile, recent works 34 ; 29 ; 5 attempt to incorporate geometry conditions from 2D foundation models 2 ; 49 , e.g. one-view depth maps, to align the video generation. However, such 2.5D priors can not model complete object shapes, hence generation for unobserved or occluded object parts remains under-constrained. In this work, we present a novel method to effectively improve shape realism and view consistency for orbital video generation via 3D foundation priors , which is achieved by leveraging the power of recently developed 3D generative models 22 ; 53 ; 55 . These 3D foundation models learn to directly reconstruct complete object shapes in a native 3D latent space, and thus are capable of encoding rich object structural information and modeling realistic object shape distributions after being trained with large-scaled 3D asset corpora and explicit geometry supervision. A key observation is that their latent features can be used as an effective 3D shape prior, which not only provides auxiliary constraints for synthesizing challenging target views, but also enhances view consistency thanks to a unified 3D reference. To this end, we propose to prompt the video generation with an additional shape condition, i.e. two scales of latent features encoded by the 3D foundation model as: ( i ) a global latent vector acts as an overall structural guidance, and ( ii ) a set of latent images projected from volumetric features queried on the global vector, which provides fine-grained and view-dependent geometry details. By using compact latent features as condition signals, we avoid time-consuming mesh extraction process and ensure an efficient inference pipeline. Furthermore, we introduce a multi-scale 3D adapter with alternating cross attention layers to effectively inject both features to the base video generation model. In this way, the adapter can be efficiently and flexibly plugged into any base video transformers, while retaining its capability inherited from general video pretraining. We train our model on a large-scale dataset with synthetic videos rendered from high-quality assets, which ensures model generalizability across diverse object types. Our contributions can be summarized as follows. ( i ) We propose a novel method that incorporates the video generation pipeline with 3D foundation priors, which noticeably improves shape realism and view consistency in generated results. ( ii ) We introduce a multi-scale 3D adapter, which effectively and efficiently injects latent shape features into the base video models to prompt video generation with 3D shape conditions. Extensive experiments on multiple benchmarks show that our method generates results with more realistic shapes and consistent frames compared to various baseline methods, and consistently applies to complex camera trajectories and in-the-wild examples. 2 Related Works Novel View Synthesis. Recently developed diffusion models 32 ; 39 ; 35 have shown promising success in generating high-quality images under various text and image conditioning. In view of this, several works 23 ; 37 ; 18 ; 56 ; 24 ; 25 ; 21 have extended them for novel view synthesis (NVS), which is formulated as camera-conditioned image generation. Early works 23 ; 9 mostly condition on coarse camera prompts such as relative pose differences, and independently generate individual frames, thus often suffering inaccurate camera control and inconsistency across views. In contrast, 56 ; 18 then propose to refine the camera condition with more precise ray 38 or position embeddings for more accurate camera control. Moreover, most works 48 ; 25 ; 21 ; 37 ; 24 observe that pixel-wise attention is critical for enforcing view consistency between concurrently generated frames, which can be specialized as epipolar 14 ; 48 or scanline 21 attention for memory efficiency. However, all above works only make use of image diffusion models and generate sparse frames. In this work, we extend to dense -frame video diffusion for improved temporal smoothness and visual fidelity. Orbital Video Generation. In contrast to single-image generation models, video diffusion models 4 ; 31 ; 50 are pretrained with large-scale video data, therefore preserve rich temporal priors which are beneficial for