PE3R: Perception-Efficient 3D Reconstruction

Jie Hu, Shizun Wang, Xinchao Wang · 2025

Read PE3R: Perception-Efficient 3D Reconstruction by Jie Hu, Shizun Wang, Xinchao Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-07507.txt.

Introduction

Jie Hu, Shizun Wang, and Xinchao Wang rebuild scenes in three dimensions from plain photos in a 2025 paper. Past methods lean on per-scene tuning and drift in meaning across views. Their frame, PE3R, needs no tuning for a new scene.

The frame joins multi-view shape math with two-dimensional meaning cues in one forward pass. It then reads both depth and labels for unseen scenes and object types at once. Tests on open-word sorting and depth gauging set new best scores on both fronts. It also runs up to nine times faster than rivals. The work points toward language-led scene reading at scale.

Catalog record

CREATORJie Hu, Shizun Wang, Xinchao Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-07507

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PE3R: Perception-Efficient 3D Reconstruction Jie Hu, Shizun Wang, Xinchao Wang Abstract Recent advances in 2D-to-3D perception have enabled the recovery of 3D scene semantics from unposed images. However, prevailing methods often suffer from limited generalization, reliance on per-scene optimization, and semantic inconsistencies across viewpoints. To address these limitations, we introduce PE3R, a tuning-free framework for efficient and generalizable 3D semantic reconstruction. By integrating multi-view geometry with 2D semantic priors in a feed-forward pipeline, PE3R achieves zero-shot generalization across diverse scenes and object categories without any scene-specific fine-tuning. Extensive evaluations on open-vocabulary segmentation and multi-view depth estimation show that PE3R not only achieves up to 9 × \times faster inference but also sets new state-of-the-art accuracy in both semantic and geometric metrics. Our approach paves the way for scalable, language-driven 3D scene understanding. Code is available at github.com/hujiecpp/PE3R . 1 Introduction Figure 1 : Examples of Perception-Efficient 3D Reconstruction. Our framework reconstructs 3D semantic scenes from unposed RGB images and supports open-vocabulary interaction, without any training or per-scene optimization. It is efficient in two critical ways: (1) Input efficiency : no depth maps or camera parameters are needed; (2) Inference efficiency : significant speed gains on semantic reconstruction compared to prior pipelines. These properties make PE3R practical for real-world deployment in large-scale and time-sensitive scenarios. While machine vision has achieved remarkable success in 2D perception tasks, human perception is inherently three-dimensional, seamlessly integrating multiple viewpoints to construct a coherent understanding of the world [ 2 , 43 , 54 ] . This fundamental gap motivates a central challenge in computer vision: can we harness powerful 2D perception models to reconstruct and comprehend 3D scenes without relying on 3D supervision? The problem is twofold: it requires not only estimating accurate geometry from unposed images but also lifting 2D semantic information into a consistent 3D representation. Recent methods have made significant strides in 2D-to-3D perception, recovering both geometry and semantics from unposed multi-view images [ 15 , 23 , 46 , 47 , 32 , 45 , 20 , 6 , 27 , 17 , 18 , 56 , 62 , 33 , 5 ] . Despite this progress, these approaches often face a trilemma: they struggle with scene generalization, fail to maintain semantic consistency across views, and suffer from high computational costs. For instance, Neural Radiance Fields (NeRF) [ 29 ] and 3D Gaussian Splatting (3DGS) [ 19 ] can produce high-quality reconstructions but typically require per-scene training and involve heavy semantic lifting, making them impractical for real-time or large-scale applications. To overcome these limitations, we introduce PE3R, a tuning-free framework for efficient, accurate, and generalizable 3D semantic reconstruction. Inspired by recent advances in feed-forward geometry estimation [ 52 , 24 , 49 ] , PE3R employs a feed-forward pipeline that eliminates scene-specific fine-tuning while enabling rapid inference. As illustrated in Fig. 1 , our framework integrates three carefully designed modules: (1) Pixel Embedding Disambiguation ensures semantic consistency across views and object hierarchies; (2) Semantic Point Cloud Reconstruction fuses geometry with dense 2D semantics to refine 3D structure; and (3) Global View Perception enables open-vocabulary object retrieval through language grounding. This cohesive design empowers robust zero-shot generalization to previously unseen environments. We conduct extensive evaluation across diverse benchmarks, including Mip-NeRF360 [ 3 ] , Replica [ 44 ] , ScanNet++ [ 57 ] for open-vocabulary segmentation, and KITTI [ 14 ] , ScanNet [ 9 ] , DTU [ 1 ] , ETH3D [ 39 ] , Tanks and Temples [ 22 ] for multi-view depth estimation. Results demonstrate that PE3R achieves up to 9\times faster 3D semantic reconstruction than prior methods while advancing in both semantic fidelity and geometric precision. Our main contributions are summarized as follows: • We propose PE3R, a fast and scalable framework for 3D semantic reconstruction that operates directly on unposed 2D images, requiring no 3D supervision, per-scene fine-tuning, or camera parameters. • We introduce three novel modules that address key challenges in 2D-to-3D perception: pixel embedding disambiguation for cross-view consistency, semantic point cloud reconstruction for geometry-semantics fusion, and global view perception for open-vocabulary interaction. • Through comprehensive evaluations on multiple benchmarks, we demonstrate state-of-the-art performance, effective zero-shot generalization, and practical scalability. Code is provided to ensure reproducibility. 2 Related Work 2D-to-3D Reconstruction. Significant progress has been made in reconstructing 3D scenes from 2D images, with many methods achieving high surface fidelity and geometric coherence. A number of approaches leverage signed distance functions (SDFs) combined with volume rendering for accurate mesh extraction [ 13 , 16 , 28 , 51 , 53 ] . Neural Radiance Fields (NeRF) [ 29 ] popularized the use of implicit neural representations for view synthesis, while 3D Gaussian Splatting (3DGS) [ 19 ] later introduced an explicit, point-based alternative that enables highly efficient rendering. To overcome the dependency on known camera parameters, several recent works operate directly on unposed images. Among them, DUSt3R [ 52 ] predicts 3D structure from image sets. MASt3R [ 24 ] extends this direction by incorporating keypoint correspondences, and MASt3R-SfM [ 10 ] further unifies structure-from-motion and multi-view stereo within a transformer framework. Hybrid techniques such as Vggsfm [ 50 ] combine traditional geometry with learned features to improve robustness. Scalable systems like Spann3R [ 48 ] , FASt3R [ 55 ] , and VGGT [ 49 ] integrate attention mechanisms or visibility reasoning to enhance multi-view consistency and computational efficiency. While these methods achieve impressive geometric results, they typically lack integrated semantic understanding or require task-specific adaptation. PE3R builds upon these geometric foundations but moves beyond them by incorporating semantic priors, enabling efficient and context-aware 3D scene understanding without per-scene training. Figure 2 : The PE3R Framework. Our pipeline comprises three core stages: (1) Pixel Embedding Disambiguation: Images are decomposed into hierarchical masks (via SAM/SAM2), which are encoded by CLIP and aggregated to yield consistent per-pixel embeddings. (2) Semantic Point Cloud Reconstruction: A feed-forward model (DUSt3R) generates 3D point clouds, which are then refined using semantic cues. (3) Global View Perception: Text embeddings are compared against the 3D point features via global similarity normalization, enabling open-vocabulary semantic localization. 2D-to-3D Perception. A parallel line of research seeks to lift 2D semantic information into 3D reconstructions. Early works such as GNeRF [ 7 ] and InPlace [ 61 ] embedded 2D masks into NeRF representations to enable 3D-aware segmentation with 2D supervision. Subsequent efforts extended NeRF to support 3D instance segmentation, using either supervised [ 26 , 4 , 42 ] or unsupervised [ 30 , 58 ] learning paradigms. The introduction of the Segment Anything Model (SAM) [ 21 ] inspired a range of SAM-guided 3D lifting pipelines, including SA3D [ 6 ] and Feature3DGS [ 62 ] , which incorporate prompt-driven segmentation into 3D scene representations. Several other methods directly project SAM masks into 3D space, such as SAGA [ 5 ] , Gaussian Grouping [ 56 ] , SAGS [ 18 ] , Click-Gaussian [ 8 ] , and FlashSplat [ 41 ] . The advent of vision-language models has further enabled open-vocabulary 3D segmentation. For example, LangSplat [ 33 ] aligns CLIP embeddings with 3D Gaussians, while GOI [ 34 ] enforces multi-view consistency through text-image alignment. The Large Spatial Model (LSM) [

Read the complete text