PanoImager: Geometry-Guided Novel View Synthesis and Reconstruction from Sparse Panoramic Views

Zhisong Xu, Takeshi Oishi · 2026

Read PanoImager: Geometry-Guided Novel View Synthesis and Reconstruction from Sparse Panoramic Views by Zhisong Xu, Takeshi Oishi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-27071.txt.

Introduction

Xu and Oishi build PanoImager, a system that rebuilds three-dimensional scenes from just a few panoramic photos. Standard mapping tools then fail to even start, because they need varied viewpoints to triangulate, meaning to locate points by crossing sight lines. PanoImager skips that fragile start entirely. It cuts each panorama into ordinary perspective views and leans on learned geometric hints from the start in 2026 work.

Feed-forward models first guess camera poses and depths for the cut views. A diffusion model, guided by that geometry, then invents extra views to fill sparse evidence. Depth guidance steadies the final step, where a cloud of soft three-dimensional blobs is tuned for rendering and reconstruction. Across several benchmarks, the pipeline stays stable even under extreme sparsity. Xu and Oishi suggest it as a background map-refining tool for robots when standard initialisation fails.

Catalog record

CREATORZhisong Xu, Takeshi Oishi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.27071

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PanoImager: Geometry-Guided Novel View Synthesis and Reconstruction from Sparse Panoramic Views Zhisong Xu, Takeshi Oishi Abstract Panoramic sensing offers wide field-of-view coverage, yet 3D reconstruction from sparse panoramas remains challenging under rotation-dominant, weak-parallax motion. In such regimes, SfM/SLAM initialization is often ill-conditioned and unreliable. We present PanoImager , an SfM-free framework that combines feed-forward pose/depth priors, geometry-conditioned diffusion view completion, and depth-guided 3DGS optimization. Given only a few panoramic images, PanoImager decomposes them into local perspective views, synthesizes auxiliary observations to enrich sparse evidence, and stabilizes Gaussian optimization for improved cross-view consistency. Experiments on multiple benchmarks show improved stability under extreme sparsity, suggesting PanoImager as an offline/background component for map refinement when SfM/SLAM fails to initialize. I Introduction Fig. 1: Overview of our panoramic reconstruction pipeline. Starting from sparse panoramic images in \Omega_{1} , we first sample them into a perspective observation set in \Omega_{2} . A visual foundation model then predicts camera poses and depth on \Omega_{2} , providing geometric priors. Conditioned on these priors, we extend the observations from \Omega_{2} to \Omega_{3} via novel view synthesis, and map the generated views back to \Omega_{2} to enrich the observation set. Finally, the augmented observations are lifted to \Omega_{4} , where a 3D Gaussian representation is optimized for reconstruction and rendering. Three-dimensional (3D) reconstruction plays a central role in robotic perception by supporting spatial reasoning, mapping, and interaction in complex environments [ 1 ] . In practical settings such as indoor inspection, teleoperation, and narrow-space exploration, robots often observe the scene under constrained motion and limited sensing budgets. In particular, rapid in-place scanning often leads to rotation-dominant capture with only weak translational parallax, making reliable reconstruction from sparse observations fundamentally difficult. This limitation is not merely an engineering issue, but a geometric one. Conventional SfM and SLAM pipelines rely on sufficient viewpoint diversity and parallax to triangulate stable 3D structure. Under pure rotation or extremely small baselines, triangulation becomes unstable, geometric constraints become poorly conditioned, and initialization may become effectively rank-deficient. As a result, feature-correspondence-based methods often fail to provide a reliable starting point for downstream mapping and reconstruction. Panoramic cameras are attractive for robotic mapping and localization because they capture a full 360^{\circ} field of view in a single image [ 4 , 5 , 27 , 17 ] . While this wide coverage improves scene observability under sparse capture, panoramic imagery also introduces severe spherical distortion and large appearance variation, which complicate both geometric estimation and learning-based reconstruction. Thus, panoramic sensing alleviates field-of-view limitations, but does not by itself resolve the geometric degeneracy caused by weak-baseline motion. Recent neural scene representations, such as NeRF [ 18 ] and 3D Gaussian Splatting (3DGS) [ 12 ] , have significantly improved novel-view synthesis and 3D reconstruction. However, in panoramic settings, they remain sensitive to projection distortion [ 13 , 7 , 15 ] and still depend on geometry initialization or multi-view optimization under sufficiently well-conditioned motion. However, with extremely sparse panoramic inputs, SfM/SLAM initialization becomes unreliable and frequently fails. Feed-forward geometric models can directly predict depth and camera pose from sparse inputs [ 21 , 20 , 23 ] , and recent panoramic variants further reduce dependence on iterative optimization [ 14 , 25 , 6 ] . Yet under extreme sparsity, they often produce incomplete geometry and view-dependent artifacts. Diffusion models offer a complementary prior for synthesizing plausible observations from limited inputs [ 2 , 24 , 16 , 3 ] , but diffusion alone cannot ensure cross-view geometric consistency. To address these challenges, we propose PanoImager , a prior-guided SfM-free framework for 3D reconstruction from sparse panoramic observations in degenerate robotic exploration regimes. Instead of treating generative completion as an independent view synthesis module, PanoImager uses it as a geometry-aware scaffold for reconstructing a consistent 3D Gaussian scene. Specifically, feed-forward pose and depth priors provide an initial geometric anchor, pose- and depth-guided diffusion densifies the sparse observation space, and depth-constrained 3DGS optimization integrates real and synthesized views through reliability-aware geometric supervision. This design removes the dependence on stable feature correspondences across sparse panoramas and improves reconstruction robustness under rotation-dominant, weak-parallax, or feature-poor capture. Our contributions are summarized as follows: • We present PanoImager , an SfM-free reconstruction framework for extremely sparse panoramic observations under rotation-dominant, weak-parallax motion, where conventional SfM/SLAM initialization becomes poorly conditioned or fails altogether. • We propose a training-free, geometry-conditioned view augmentation scheme that synthesizes auxiliary observations to densify the sparse measurement space, while using reliability-aware soft supervision to mitigate hallucinated content. • We introduce a depth-constrained 3D Gaussian optimization strategy with explicit geometric anchors and anti-floater regularization to improve structural integrity and spatial consistency in the reconstructed scene. II Related Work II-A Panoramic Gaussian Splatting 3D Gaussian Splatting (3DGS) enables efficient, high-quality novel view synthesis using explicit geometric primitives [ 12 ] . While sphere-native extensions adapt 3DGS to panoramic imagery via spherical rays or tangent-plane parameterizations [ 13 , 7 , 15 ] , these modifications often make optimization sensitive under sparse observations. When only a few panoramas are available with rotation-dominant motion, limited overlap leaves large unobserved regions, degrading geometric completeness, cross-view consistency, and extrapolation to unseen viewpoints—motivating the incorporation of structured view completion priors while preserving the stable optimization of perspective 3DGS. II-B SfM and Feedforward Geometry Conventional 3D reconstruction pipelines typically rely on SfM or SLAM frameworks to estimate camera poses and recover scene geometry, including large-scale multi-sensor mapping systems and recent dense Gaussian SLAM approaches [ 26 , 8 , 10 , 28 , 9 ] . However, for panoramic imagery with wide fields of view and predominantly rotational motion, pose estimation from sparse observations remains challenging. Nevertheless, in the extremely sparse panoramic setting, predictions can still be uncertain in unobserved regions, and the lack of dense multi-view constraints makes it difficult to recover coherent geometry across the full viewing sphere, motivating geometry-aware view completion to densify observations. II-C Diffusion-Guided View Synthesis Diffusion-based generative models have recently shown strong capability in synthesizing novel views from sparse observations by leveraging learned appearance and structural priors. Recent approaches further incorporate explicit geometric cues (e.g., depth- or pose-conditioned warping priors) to improve multi-view consistency and enable more reliable 3D optimization. Nevertheless, diffusion models are predominantly developed for perspective imagery and are rarely tailored to panoramic sensing, where spherical distortions, structured view parameterization, and rotation-dominant motion impose stricter geometric constraints and exacerbate extrapolation to non-visible viewpoints. III Method III-A Problem Definition and Overview The input consists of only N_{p} panoramic frames, typically N_{p}\in[3,6] , leaving many 3D surfaces weakly constrained due to limited baseline, occlusion, and insufficient viewpoint diversity. We denote this sparse panoramic input domain by \Omega_{1} . Under such limited spatial coverage and viewpoint diversity, reconstruction becomes severely ill-conditioned: insufficient viewpoint overlap and weak parallax lead to unstable pose estimation, fragmented geometry, and unreliable multi-view optimization. These failure modes make purely geometric pipelines fragile and motivate the introduction of stronger geometric and generative priors. To address this setting, we propose PanoImager , illustrated in Fig. 1 . Starting from \Omega_{1} , we first decompose each panorama into a structured set of local perspective views, yielding an intermediate domain \Omega_{2} . On \Omega_{2} , we build a feed-forward geometric scaffold by directly

Read the complete text