VolFill: Single-View Amodal 3D Scene Reconstruction with Volumetric Flow Matching

Tuan Duc Ngo, Chuang Gan, Evangelos Kalogerakis, [2pt] UMass Amherst · 2026

Read VolFill: Single-View Amodal 3D Scene Reconstruction with Volumetric Flow Matching by Tuan Duc Ngo, Chuang Gan, Evangelos Kalogerakis, [2pt] UMass Amherst (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-31466.txt.

Introduction

Ngo, Gan, and Kalogerakis present VolFill, a generative tool that rebuilds a full 3D scene from a single photo. It covers parts hidden from view, not just what the camera sees. A hybrid 3D autoencoder compresses sparse distance grids into a compact latent code. Then a diffusion transformer denoises that code into the whole scene, guided by geometry models. Unlike methods tied to single rays or loose point clouds, it allows direct surface extraction and occupancy queries, and it beats current baselines on two test sets.

Catalog record

CREATORTuan Duc Ngo, Chuang Gan, Evangelos Kalogerakis, [2pt] UMass Amherst
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42027
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.31466

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VolFill: Single-View Amodal 3D Scene Reconstruction with Volumetric Flow Matching Tuan Duc Ngo, Chuang Gan, Evangelos Kalogerakis, [2pt] UMass Amherst Abstract Reconstructing the complete geometry of a scene from a single RGB image remains challenging—especially when inferring hidden structures where visual evidence is incomplete. We introduce VolFill, a generative framework that predicts the 3D structure of the complete scene rather than relying on traditional pixel-aligned regression. Our method utilizes a hybrid 3D VAE to compress sparse truncated unsigned distance function grids into a compact latent space, paired with a latent Diffusion Transformer that denoises this representation to recover the complete scene. We condition the generation on geometry foundation models, leveraging rich spatial priors for robust reasoning. Unlike existing methods limited by per-ray constraints or unstructured point-cloud queries, VolFill provides a structured representation that supports direct surface extraction and occupancy queries at scale. Extensive experiments on the SCRREAM and NRGB-D datasets demonstrate that our approach significantly outperforms current baselines, providing a robust foundation for holistic spatial understanding. Figure 1: VolFill synthesizes structured amodal 3D geometry from (a) a single-view image, recovering holistic scene layouts from partial visibility. (b) Pixel-aligned methods are restricted to visible surfaces. (c) Amodal baselines produce sparse, noisy or artifact-heavy geometry, yielding fragmented meshes. Our approach delivers clean, sharp point clouds and smooth, consistent meshes. 1 Introduction This paper addresses amodal 3D scene reconstruction, seeking to recover the complete geometry, including observed and occluded structures, from a single RGB image. Beyond the inherent ill-posed problem, amodal reconstruction must infer hidden structures where geometric evidence is partially absent. This requires synthesizing under-determined regions that remain physically plausible and consistent with the observed scene. Such spatial awareness is critical for practical applications, as navigating or interacting with an environment requires a comprehensive understanding of the scene that extends well beyond the immediate line of sight. Recent feed-forward methods [ 78 , 79 , 5 , 87 , 80 , 77 , 89 , 37 , 81 ] can recover accurate 3D structure from images in seconds, but all share a hard constraint: pixel alignment . Every predicted point lies on a source camera ray, bounding reconstruction strictly to visible surfaces and producing duplicated geometry in overlapping views. Optimization-based scene representations [ 49 , 56 , 69 , 52 , 38 ] reconstruct complete geometry but require dense captures and per-scene optimization which can take hours to compute. Large reconstruction models [ 30 , 97 ] and generative 3D models [ 85 , 84 , 42 , 73 ] demonstrate impressive 3D reconstruction quality from sparse inputs, but are predominantly optimized for object-centric scenarios, limiting their applicability to complex scenes with unbounded extents and challenging amodal structures. At the scene level, LaRI [ 41 ] proposes a layered ray intersection representation, regressing multiple ordered surface intersections per camera ray. While this enables amodal recovery within a pixel-aligned framework, it remains a partial solution; the fixed layer count under-represents densely occluded structures while wasting capacity in open regions. Furthermore, it frequently yields layered artifacts (the “multiple wall” effect) rather than accurately recovering the actual unobserved surfaces. Alternatively, NOVA3R [ 11 ] employs flow matching to directly generate amodal 3D point clouds , but often yields noisy and disintegrated outputs (Fig. 1 ). To address these limitations, we propose VolFill, a latent diffusion framework that predicts a complete Truncated Unsigned Distance Function (TUDF) voxel grid from a single image. A TUDF encodes the distance to the nearest scene surface, including occluded regions, as a continuous scalar field, enabling direct surface extraction via isosurfacing [ 23 ] and without requiring post-processing reconstruction of point clouds. Unlike point clouds, it scales well with scene complexity; unlike layered ray representations, it places no constraint on the number or layers of recoverable occluded surfaces. VolFill consists of a hybrid 3D VAE that encodes the sparse TUDF grid into a compact dense latent space, and a Diffusion Transformer (DiT) trained with flow matching. To ensure robust in-the-wild generalization, we introduce a dual conditioning strategy combining high-level image tokens with explicit visible geometry from a frozen MoGe2 [ 79 ] model, grounding amodal reasoning in the observed scene while leveraging strong geometric priors to compensate for limited 3D training data. Trained on 3D-FRONT [ 17 ] and ScanNet++ [ 91 ] , VolFill achieves state-of-the-art performance on the SCRREAM [ 33 ] and NRGB-D [ 2 ] benchmarks, synthesizing high-fidelity amodal geometry with significantly greater sharpness and structural accuracy than existing methods (Fig. 1 ). In summary, our main contributions are: • VolFill , a generative framework that utilizes volumetric flow matching to recover complete scene-level geometry from single-view images. • A hybrid 3D VAE enabling efficient spatial compression of high-resolution TUDF grids to a compact latent space, facilitating high-fidelity reconstruction of complex amodal structures. • A dual-conditioning strategy leveraging geometry foundation models to integrate high-level image tokens with explicit visible geometry. 2 Related Work Volumetric 3D Representations provide a flexible foundation for 3D vision, spanning applications from scene understanding to generative modeling. To mitigate the cubic complexity of dense grids, sparse convolutional engines [ 22 , 13 , 71 ] have become the standard for efficiently processing geometry along active surfaces. In scene-level reconstruction, “lifting” paradigms map 2D features into these volumes through depth-guided projection [ 70 ] , ray-sampling [ 6 ] , or transformer-based voxel queries [ 43 , 82 ] . Within the generative domain, early models applied diffusion directly to raw voxels [ 53 ] , whereas contemporary frameworks leverage hierarchical latent diffusion [ 12 , 62 ] or rectified flow on sparse latents [ 85 ] to achieve high-fidelity structural synthesis. Building on these advances, our approach adopts a TUDF representation within a structured 3D grid, using sparse convolutions to capture complex scene geometry while maintaining high computational efficiency. Pixel-Aligned Single-View 3D Reconstruction recovers visible surface geometry, evolving from early handcrafted features [ 29 , 66 , 34 , 67 ] to deep architectures [ 75 , 18 , 3 , 39 , 15 ] . Progress in scaling has yielded models with remarkable zero-shot generalization [ 61 , 4 , 92 , 31 , 59 , 24 , 89 , 90 , 5 ] . Recently, methods have further refined geometric accuracy by jointly predicting intrinsics [ 59 , 5 ] , regressing dense 3D pointmaps [ 78 , 79 ] , or utilizing diffusion priors [ 65 , 60 ] for high-fidelity, arbitrary-resolution depth synthesis [ 36 , 26 , 19 , 21 , 58 , 87 , 93 ] . Nonetheless, these approaches remain fundamentally confined to 2.5D visible surfaces. Their inability to infer occluded regions or amodal structure directly motivates our shift from surface-level estimation to full-scene reconstruction. Geometry Foundation Models as Priors. Recent geometry foundation models (GFM) [ 80 , 16 , 78 , 79 , 77 , 44 ] have demonstrated strong performance in 3D reconstruction and serve as robust feature extractors for downstream tasks requiring explicit spatial priors [ 87 , 54 , 83 , 8 ] . Current approaches typically adopt one of two strategies: (1) using explicit geometric outputs (e.g., depth or point clouds) as structural anchors to guide synthesis [ 95 , 94

Read the complete text