Omni-View: Unlocking How Generation Facilitates Understanding in Unified 3D Model based on Multiview images
JiaKui Hu Shanshan Zhao Qing-Guo Chen Xuerui Qiu Jialun Liu, Zhao Xu Weihua Luo Kaifu Zhang Yanye Lu · 2025
Read Omni-View: Unlocking How Generation Facilitates Understanding in Unified 3D Model based on Multiview images by JiaKui Hu Shanshan Zhao Qing-Guo Chen Xuerui Qiu Jialun Liu, Zhao Xu Weihua Luo Kaifu Zhang Yanye Lu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07222.txt.
Introduction
JiaKui Hu and eight co-authors build a single model that understands and generates three-dimensional scenes from multiple photographs. Omni-View tests the principle that creation facilitates understanding: a model that can synthesize new views and estimate geometry should comprehend scenes more deeply. It combines an understanding model with a texture module for appearance synthesis and a geometry module supplying explicit geometric constraints, trained in two stages.
The parts interact synergistically, so scene question answering benefits from the same representations that render novel views. On the VSI-Bench benchmark of spatial understanding, Omni-View scored 55.4, a state-of-the-art result ahead of specialized 3D understanding models, while also performing strongly at novel view synthesis and scene creation. Code and pre-trained models are open-sourced, making the unified design available for further research.
Catalog record
| CREATOR | JiaKui Hu Shanshan Zhao Qing-Guo Chen Xuerui Qiu Jialun Liu, Zhao Xu Weihua Luo Kaifu Zhang Yanye Lu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-07222
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Omni-View: Unlocking How Generation Facilitates Understanding in Unified 3D Model based on Multiview images JiaKui Hu Shanshan Zhao Qing-Guo Chen Xuerui Qiu Jialun Liu, Zhao Xu Weihua Luo Kaifu Zhang Yanye Lu Abstract This paper presents Omni-View, which extends the unified multimodal understanding and generation to 3D scenes based on multiview images, exploring the principle that “generation facilitates understanding”. Consisting of understanding model, texture module, and geometry module, Omni-View jointly models scene understanding, novel view synthesis, and geometry estimation, enabling synergistic interaction between 3D scene understanding and generation tasks. By design, it leverages the spatiotemporal modeling capabilities of its texture module responsible for appearance synthesis, alongside the explicit geometric constraints provided by its dedicated geometry module, thereby enriching the model’s holistic understanding of 3D scenes. Trained with a two-stage strategy, Omni-View achieves a state-of-the-art score of 55.4 on the VSI-Bench benchmark, outperforming existing specialized 3D understanding models, while simultaneously delivering strong performance in both novel view synthesis and 3D scene generation. https://jkhu29.github.io/omni_view 1 Introduction Recently, unified multimodal models (UMMs) ( 26 ; 41 ; 9 ) have emerged as a pivotal area of research. The primary goal is to empower multimodal large language models (MLLMs) to both understand and generate visual signals present in our world, laying the foundation for artificial general intelligence. Numerous methods ( 26 ; 36 ; 40 ; 41 ; 5 ; 9 ) have achieved coexistence between 2D image understanding and generation. Among them, some studies ( 38 ; 30 ; 44 ) aim to improve model performance by exploring the synergies arising from the interaction between understanding and generation. For example, Metamorph ( 30 ) has extensively validated the role of understanding in improving generation performance. However, the potential and effectiveness of generation to improve understanding capabilities within unified models remain underexplored. Due to the intrinsic geometric and spatiotemporal nature of 3D scenes and their multiview images, generative tasks such as geometry estimation and novel view synthesis are particularly well-suited for facilitating understanding in the 3D domain. This is attributable to the fact that 3D understanding tasks ( 45 ; 51 ) inherently necessitate robust geometric and spatiotemporal modeling capabilities, which can be effectively acquired through these generative tasks. Moreover, biomedical evidence ( 20 ; 21 ) suggests that human understanding of 3D environments is governed by the capacity to generate and imagine future sensory and geometric data ( 14 ; 16 ) in the observed scene. These findings provide us with a guidance: the “generation facilitates understanding” paradigm presents a promising approach to building a unified model for 3D scene understanding and generation. Inspired by the above analysis, this paper aims to fully unlock and maximize the benefits of generation to understanding, thereby constructing a unified model for 3D scene understanding and generation, called Omni-View . We emphasize that geometry estimation and novel view synthesis can leverage their inherent geometric measurement and spatiotemporal modeling capabilities to improve 3D scene understanding, localization, and spatial reasoning. Specifically, we achieve this through two key aspects, architectural design and training strategy . The architectural design aims at unifying 3D scene understanding and generation, leveraging geometry estimation and novel view synthesis to advance 3D scene understanding. Our Omni-View is built on Bagel ( 9 ) , a strong unified framework in which the interaction between understanding and generation is facilitated by its shared multimodal self-attention. However, Bagel’s generative capacity is limited to RGB images, thus only capturing texture information, whereas 3D scene generation necessitates inclusion of both texture and geometric structure. In accordance with the dual generative objectives, the generation model in Omni-View is split into two distinct modules: a texture module and a geometry module. The texture module receives reference images, a list of targeted camera poses, and prompt tokens encoded by the understanding model to generate novel views of the scene. Meanwhile, the geometry module employs the hidden features from the understanding model and the latent output of the texture module to infer geometric information of novel views, such as depth maps and camera poses. This dual-pathway architecture empowers the model to develop both geometric and spatiotemporal modeling capabilities, which are essential for 3D scene understanding tasks. The principal goal of training strategy is to comprehensively improve the performance of the model. A two-stage training strategy is employed. The first stage aims to augment the benefits of generation for understanding 3D scenes, as introduced by the proposed architecture. The subsequent stage is intended to refine the generation performance. In stage 1, the understanding model, geometry module, and texture module are trained simultaneously. Geometry estimation assists the model in comprehending the relative positional relationships among objects, thus directly enhancing the model’s capability to evaluate the relative distances and directions of objects in 3D scenes. The autoregressive generation forces the understanding model to discern the spatiotemporal relations between the generated novel views, thus improving its understanding capabilities. As iterations progress in stage 1, the number of reference images gradually decreases. This progressive shift from dense to sparse views supports a curriculum-like, easy-to-difficult training approach, ultimately enhancing the performance of the understanding model. In stage 2, the understanding model is forzen. The generation model is finetuned via RGB-Depth-Pose joint generation, thereby enhancing its capabilities in 3D scene generation. We evaluate Omni-View on scene understanding, spatial reasoning, and novel view synthesis tasks. The model achieves an impressive score of 55.4 on the VSI-Bench, exceeding current MLLMs designed for visual reasoning. It manifests particularly notable improvements in subtasks such as Relative Distance and Appearance Order, which require spatiotemporal modeling and the estimation of geometry acquired through generation tasks. It also exhibits superior performance compared to existing 3D understanding MLLMs in the area of 3D question answering ( 19 ; 1 ) . Furthermore, it effectively narrows the performance gap between unified models and specialized models focused on 3D visual localization ( 4 ) . Furthermore, we attain robust results in the domain of novel view synthesis and scene generation, with particular emphasis on enhanced perceptual quality. 2 Related work Scene understanding. Recent advances in understanding 3D scenes have been greatly provided by incorporating 3D or video input and 3D reconstruction prior. LLaVA-3D ( 60 ) and GPT4Scene ( 23 ) function within the voxel space and BEV. Video3DLLM ( 57 ) improves localization ability by encoding 3D coordinates as position embeddings, while Ross3D ( 32 ) improves 3D understanding through visual-centric reconstruction. However, the dependency on 3D input of these methods poses practical application challenges. To alleviate this issue, VG-LLM ( 56 ) and SpatialMLLM ( 37 ) use the features of VGGT ( 33 ) as input, embedding the 3D piror in the model. Scene generation. A vital element of scene generation methods is the presence of an efficient proxy to represent the 3D scene, with panoramic images ( 28 ) , point clouds ( 48 ) , and Gaussian Splatting ( 46 ) among the viable options. In consideration of the advances in image and video diffusion models ( 25 ; 31 ) , contemporary 3D scene generation strategies reconstruct the scene’s geometry from a single view, employing conditioned video diffusion models to render the scene’s texture. ViewCrafter ( 48 ) brings explicit 3D information (point cloud) to generation