PLAF : Pixel-wise Language-Aligned Feature Extraction for Efficient 3D Scene Understanding

Junjie Wen, Junlin He, Fei Ma, Jinqiang Cui · 2026

Read PLAF : Pixel-wise Language-Aligned Feature Extraction for Efficient 3D Scene Understanding by Junjie Wen, Junlin He, Fei Ma, Jinqiang Cui (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-15770.txt.

Introduction

Wen, He, Ma, and Cui offer PLAF, a way to pull word-tied traits from each pixel for open-word 3D scene grasp, without losing word range. Thick pixel-to-3D spread wastes store room and slows asks in big scenes, so PLAF adds a lean store-and-ask plan that cuts repeats in both 2D and 3D. Tests show PLAF grounds true, fast open-word 3D scene grasp.

Catalog record

CREATORJunjie Wen, Junlin He, Fei Ma, Jinqiang Cui
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21531
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.15770

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PLAF : Pixel-wise Language-Aligned Feature Extraction for Efficient 3D Scene Understanding Junjie Wen, Junlin He, Fei Ma, Jinqiang Cui Abstract Accurate open-vocabulary 3D scene understanding requires semantic representations that are both language-aligned and spatially precise at the pixel level, while remaining scalable when lifted to 3D space. However, existing representations struggle to jointly satisfy these requirements, and densely propagating pixel-wise semantics to 3D often results in substantial redundancy, leading to inefficient storage and querying in large-scale scenes. To address these challenges, we present PLAF , a Pixel-wise Language-Aligned Feature extraction framework that enables dense and accurate semantic alignment in 2D without sacrificing open-vocabulary expressiveness. Building upon this representation, we further design an efficient semantic storage and querying scheme that significantly reduces redundancy across both 2D and 3D domains. Experimental results show that PLAF provides a strong semantic foundation for accurate and efficient open-vocabulary 3D scene understanding. The codes are publicly available at https://github.com/RockWenJJ/PLAF . I Introduction Open-vocabulary 3D scene understanding seeks to recognize, localize, and reason about arbitrary concepts in real-world environments, rather than being restricted to a fixed label set [ 10 ] [ 3 ] . Vision–language models (VLMs) are a natural fit for this setting because they align visual signals and natural language in a shared embedding space, enabling flexible zero-shot recognition [ 11 ] . Despite this promise, lifting language-aligned semantics from 2D images to 3D scenes remains non-trivial, particularly when both rich semantics and precise geometric correspondence are required [ 6 ] [ 14 ] . One key reason is that commonly used vision–language features are not sufficiently fine-grained for accurate 3D semantic association. CLIP [ 11 ] , for example, provides strong image-level vision–language alignment, but such global semantics are often too coarse for 3D scene understanding. Several recent works [ 6 ] [ 14 ] therefore attempt to map CLIP embeddings into 3D reconstructions; however, the mapped semantics can still be noisy and inconsistent in practice [ 12 ] . Meanwhile, patch-level alignment methods [ 1 ] improve spatial localization compared with image-level features, but their granularity remains limited, making them still coarse for precise 3D spatial semantic understanding. Thus, pixel-level semantic alignment is essential for precise and reliable 3D language-semantic association. Motivated by this observation, pixel-wise language-aligned features provide a direct way to achieve such precision, offering dense semantics with fine-grained spatial correspondence at the pixel level. However, existing pixel-wise solutions face two practical challenges. First, most approaches build upon CLIP-based VLMs, yet obtaining reliable pixel-level language alignment from such models is non-trivial and often requires careful and costly feature extraction or adaptation [ 6 ] [ 14 ] [ 15 ] . Second, even with pixel-level alignment, lifting dense features into 3D introduces substantial storage overhead: a language-aligned embedding is typically high-dimensional [ 11 ] , and storing it for every pixel across multi-view observations quickly becomes prohibitive. These limitations call for a method that ensures accurate pixel-wise semantics and scalable efficiency in storage and querying. To address these challenges, we present PLAF , a P ixel-wise L anguage- A ligned F eature extraction framework for efficient open-vocabulary 3D scene understanding. To this end, we first extract dense features with a visual foundation model and aggregate them with masks from a class-agnostic mask extractor to obtain pixel-wise language-aligned features that preserve open-vocabulary expressiveness while providing fine-grained spatial correspondence in 2D. Building upon these pixel-aligned semantics, we further propose a mask-based feature storage scheme that stores compact mask-level representations instead of dense per-pixel embeddings, enabling efficient lifting to 3D as well as scalable semantic querying. Overall, this framework substantially reduces redundancy across both 2D and 3D domains, making large-scale open-vocabulary 3D semantic mapping practical. In summary, our contributions are three-fold: • We propose PLAF , a pixel-wise language-aligned feature extraction framework that achieves dense and accurate 2D semantic alignment for open-vocabulary scene understanding. • We design an efficient semantic storage and querying scheme to reduce redundancy in multi-view dense semantic propagation and 3D lifting. • Extensive experiments demonstrate that PLAF serves as an accurate and efficient foundation for open-vocabulary 3D scene understanding. II Related Work II-A 2D Foundation Models Recent progress in 2D foundation models has provided strong priors for open-vocabulary understanding. Vision–language models such as CLIP [ 11 ] align images and text in a shared embedding space, while self-supervised backbones such as DINOv2 [ 9 ] improve local feature quality and dense transferability. More recent distilled models, e.g., RADIOv2.5 [ 4 ] , further improve robustness and reduce semantic noise [ 12 ] . For precise 2D-to-3D transfer, however, image-level semantics are often too coarse, motivating methods at finer granularities, including patch-level representations [ 1 ] , region-level grouping [ 15 ] , and mask-level priors such as SAM [ 7 ] . Among them, mask-guided representations offer a practical balance between semantic expressiveness and spatial precision, which motivates our compact mask-level design for scalable 3D mapping. Fig. 1: PLAF converts dense language-aligned pixel features into a compact mask-indexed semantic memory in 2D and extends this index-and-reference design to 3D, enabling scalable open-vocabulary 3D semantic mapping with significantly reduced redundancy. II-B Open-Vocabulary 3D Scene Understanding Open-vocabulary 3D scene understanding transfers vision–language semantics from 2D images to 3D geometry for open-set segmentation and querying [ 10 ] [ 3 ] [ 6 ] [ 14 ] [ 16 ] . Representative methods such as OpenScene [ 10 ] , ConceptFusion [ 6 ] , CLIP-FO3D [ 16 ] , OpenMask3D [ 14 ] , and ConceptGraphs [ 3 ] have shown that language-aligned 2D features can support flexible 3D understanding and interaction. However, existing methods still suffer from noisy local semantics, cross-view inconsistency, and high memory overhead when dense descriptors are attached to pixels or points. These limitations are especially severe in large indoor scenes. Our method addresses this gap by strengthening pixel-wise language alignment before 2D-to-3D lifting and improving semantic coherence in the reconstructed scene. II-C Efficient 3D Spatial Semantic Representation Building semantic 3D maps that are both accurate and scalable requires reducing redundancy in multi-view observations while supporting fast querying. Existing open-vocabulary mapping systems often store dense per-pixel or per-point embeddings [ 10 ] [ 6 ] [ 16 ] , which is memory-intensive because high-dimensional descriptors are repeatedly observed across views. To improve efficiency, prior work has explored neural compression [ 5 ] , sparsity-aware storage [ 8 ] , quantization, and higher-level abstractions such as metric-semantic maps and scene graphs [ 3 ] [ 13 ] . However, directly applying these strategies to open-vocabulary settings remains difficult, since compact representations must still preserve semantic discriminability and text-aligned geometric correspondence. Our method addresses this challenge with an index-and-reference design that is compact by construction in both 2D and 3D, substantially reducing memory usage while retaining queryability. III Proposed Method III-A Overall Framework The overall pipeline of PLAF is illustrated in Fig. 1 . Our goal is to construct a language-grounded yet scalable 3D semantic representation from multi-view RGB observations. To this end, PLAF decomposes the problem into two tightly coupled components: (i) mask-guided pixel-wise alignment in 2D, which turns generic visual features into spatially grounded semantics aligned with the language space, and (ii) efficient lifting and indexing in

Read the complete text