RnG: A Unified Transformer for Complete 3D Modeling from Partial Observations

Mochu Xiang, Zhelun Shen, Xuesong Li, Jiahui Ren, Jing Zhang, Chen Zhao, Shanshan Liu, Haocheng Feng, Jingdong Wang, Yuchao Dai · 2026

Read RnG: A Unified Transformer for Complete 3D Modeling from Partial Observations by Mochu Xiang, Zhelun Shen, Xuesong Li, Jiahui Ren, Jing Zhang, Chen Zhao, Shanshan Liu, Haocheng Feng, Jingdong Wang, Yuchao Dai (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-01194.txt.

Introduction

Xiang and nine colleagues infer complete 3D shapes from partial 2D views with RnG, a unified net for rebuilding and making. A rebuilding-guided causal focus trick splits the two tasks at the focus level. It treats the key-value cache as a hidden 3D model that any pose can query for novel-view RGBD output. The model rebuilds visible shapes well while making likely unseen shapes and looks, leading the field on both rebuilding and novel-view making at live speed.

Catalog record

CREATORMochu Xiang, Zhelun Shen, Xuesong Li, Jiahui Ren, Jing Zhang, Chen Zhao, Shanshan Liu, Haocheng Feng, Jingdong Wang, Yuchao Dai
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44163
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.01194

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RnG: A Unified Transformer for Complete 3D Modeling from Partial Observations Mochu Xiang, Zhelun Shen, Xuesong Li, Jiahui Ren, Jing Zhang, Chen Zhao, Shanshan Liu, Haocheng Feng, Jingdong Wang, Yuchao Dai Abstract Human perceive the 3D world through 2D observations from limited viewpoints. While recent feed-forward generalizable 3D reconstruction models excel at recovering 3D structures from sparse images, their representations are often confined to observed regions, leaving unseen geometry un-modeled. This raises a key, fundamental challenge: Can we infer a complete 3D structure from partial 2D observations? We present RnG (Reconstruction and Generation), a novel feed-forward Transformer that unifies these two tasks by predicting an implicit, complete 3D representation. At the core of RnG, we propose a reconstruction-guided causal attention mechanism that separates reconstruction and generation at the attention level, and treats the KV-cache as an implicit 3D representation. Then, arbitrary poses can efficiently query this cache to render high-fidelity, novel-view RGBD outputs. As a result, RnG not only accurately reconstructs visible geometry but also generates plausible, coherent unseen geometry and appearance. Our method achieves state-of-the-art performance in both generalizable 3D reconstruction and novel view generation, while operating efficiently enough for real-time interactive applications. Project page: https://npucvr.github.io/RnG Figure 1 : What can RnG do? Given a few unposed images of an object, 3D reconstruction foundation models like VGGT can recover the structure of observed regions , but leaves the unseen part un-modeled. RnG can estimate its complete 3D geometry within a second on an A800 GPU, using a single feed-forward transformer. RnG implicitly reconstructs 3D and render onto new viewpoints with appearance and geometry. By accumulating these rendered point maps , RnG can generate a complete 3D object, working like a virtual 3D scanner. 1 1 footnotetext: Project leader. 2 2 footnotetext: Corresponding authors. 3 3 footnotetext: Work done during an internship at Baidu. 1 Introduction 3D perception from sparse visual data is a cornerstone of modern computer vision, supporting advancements in robotics 12 , augmented reality 31 , and large-scale digital content creation 54 . Recent progress in generalizable 3D reconstruction 52 ; 50 has demonstrated the feasibility of recovering scene geometry and appearance from just a few unposed input images. However, a fundamental limitation persists across most reconstruction pipelines: these models are explicitly trained to recover geometry only for regions visible in the input camera views. Consequently, their geometric outputs are incomplete, hindering downstream applications that demand a complete object representation, such as physics simulation 27 or content generation 10 . Novel view synthesis (NVS) extends these capabilities by generating images from unseen viewpoints based on observed data. While certain NVS pipelines exhibit generalization to unobserved regions, they typically lack consistent 3D structure. As Tab. 1 summarizes, these generalized NVS methods can render photorealistic novel views but are often limited by their dependence on posed inputs, lack of geometric reconstruction, or restricted camera control. Recently, Matrix3D 30 explores a unified reconstruction and novel view generation approach. However, its diffusion-based design incurs high computational costs, making it unsuitable for real-time applications. Despite these challenges, we hypothesize that feed-forward 3D reconstruction foundation models 52 ; 50 , while primarily trained to recover visible geometry, may possess the potential to infer a more complete 3D representation within their latent space. If this potential can be effectively harnessed and then rendered through viewpoint-conditioned neural decoding, it could reveal the model’s internal, comprehensive 3D understanding. We introduce RnG , a feed-forward transformer designed to activate and make this latent perception explicit. By treating viewpoint-conditioned neural rendering as a query to the model’s latent features, RnG bridges reconstruction and generation within a single, unified framework. Specifically, to handle unposed input while maintaining explicit camera control, we build on 3D reconstruction foundation models 50 and inherit their meaningful latent 3D representations. Our single model jointly reconstructs latent 3D and generates novel view appearance and geometry. This is achieved via a reconstruction-guided causal attention mechanism, which decouples these two processes at the attention level while using the same set of parameters for both tasks. It represents a new direction that transfers knowledge from 3D reconstruction to image generation. Although prior work has focused on adapting generative priors for reconstruction 21 ; 30 , we demonstrate that the reverse direction of transferring reconstruction priors to generation is viable and effective. Table 1: Comparison between representative 3D reconstruction and novel view synthesis methods. Task 3D reconstruction Novel view synthesis Unified model Method VGGT DUSt3R LVSM LGM RayZer Matrix3D RnG 50 14 18 44 17 30 Ours unposed infer ✓ ✓ ✗ ✗ ✓ ✓ ✓ camera control N/A N/A ✓ ✓ ✗ ✓ ✓ generate unseen ✗ ✗ ✓ ✗ ✓ ✓ ✓ explicit 3D ✓ ✓ ✗ ✓ ✗ ✓ ✓ real-time infer ✓ ✓ ✓ ✓ ✓ ✗ ✓ To generalize to unobserved viewpoints, we adopt an implicit formulation: novel views are used as queries to the latent 3D space, and the network’s generation stage renders the implicit 3D onto those viewpoints. Additionally, we render a point map at each queried viewpoint, converting the implicit latent 3D to explicit geometric representations. Since RnG is a deterministic feed-forward transformer, it provides real-time inference capability. The causal attention design enables a KV-Cache mechanism, allowing the model to R econstruct from input images once (in \sim 0.2s) and subsequently G enerate novel-view appearances and geometry with high efficiency (in < 0.1s). RnG runs over 100\times faster than its diffusion-based counterparts 30 . Experimentally, RnG achieves state-of-the-art performance across multiple 3D reconstruction and novel view generation benchmarks, including camera pose estimation, input and novel view depth prediction and novel view synthesis. Remarkably, it surpasses specialized models designed for individual tasks by a significant margin. We summarize the key highlights of RnG as follows: • Unified architecture via Causal Attention. We introduce a reconstruction-guided causal attention mechanism. This decouples the two tasks at the attention level, allowing a single feed-forward transformer to coherently perform implicit 3D reconstruction, novel view synthesis, and explicit geometry generation. • KV-Cache as an Implicit 3D Representation. Enabled by our causal attention design , the network’s KV-Cache serves as a flexible implicit 3D representation. This facilitates highly efficient novel view generation (rendering) and provides a transparent link between internal memory and reconstructed geometry. • Reconstruction-driven generation. RnG demonstrates that leveraging reconstruction priors, rather than generative ones, enables superior novel view synthesis performance while significantly reducing computational cost. 2 Related Work 2.1 Neural Rendering Neural rendering represents a scene by mimicking how cameras capture the real world 47 . Some differentiable rasterization 25 or ray-tracing 15 methods directly formulate the 3D-to-2D imaging process from 3D meshes. To recover the scene from image collections, radiance fields can provide photorealistic images. These representations are encoded in neural network weights 32 , voxel grids 11 or hash-grids 34 . 3D Gaussian Splatting 23 explicitly model the scene using 3D Gaussian primitives that are similar to point clouds. To get a novel view image, all these methods require a physically explainable rendering process. More recently, there are attempts that directly use a neural network to generate novel view images, whether from multi-view images 41 ; 18 or from 3D meshes 63 . 2.2 Novel View Synthesis (a) The model architecture of RnG

Read the complete text