EPSegFZ: Efficient Point Cloud Semantic Segmentation for Few- and Zero-Shot Scenarios with Language Guidance

Jiahui Wang Haiyue Zhu Haoren Guo Abdullah Al Mamun Cheng Xiang Tong Heng Lee · 2025

Read EPSegFZ: Efficient Point Cloud Semantic Segmentation for Few- and Zero-Shot Scenarios with Language Guidance by Jiahui Wang Haiyue Zhu Haoren Guo Abdullah Al Mamun Cheng Xiang Tong Heng Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-11700.txt.

Introduction

Jiahui Wang and five co-authors present EPSegFZ, a network that labels three-dimensional point clouds when only a few examples, or none, are available for the target classes. Standard few-shot methods first pretrain, then adapt, which makes them rigid, and they mostly ignore the text annotations sitting beside the support examples.

EPSegFZ drops the pretraining stage entirely and uses language guidance to exploit support information more fully, restoring zero-shot ability that earlier designs lost. Point cloud labeling matters for robotics and mapping, where new object types appear constantly and labeling budgets are tiny. A flexible, pretraining-free route widens where few-shot segmentation can be deployed.

Catalog record

CREATORJiahui Wang Haiyue Zhu Haoren Guo Abdullah Al Mamun Cheng Xiang Tong Heng Lee
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-11700

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

EPSegFZ: Efficient Point Cloud Semantic Segmentation for Few- and Zero-Shot Scenarios with Language Guidance Jiahui Wang Haiyue Zhu Haoren Guo Abdullah Al Mamun Cheng Xiang Tong Heng Lee Abstract Recent approaches for few-shot 3D point cloud semantic segmentation typically require a two-stage learning process, i.e., a pre-training stage followed by a few-shot training stage. While effective, these methods face overreliance on pre-training, which hinders model flexibility and adaptability. Some models tried to avoid pre-training yet failed to capture ample information. In addition, current approaches focus on visual information in the support set and neglect or do not fully exploit other useful data, such as textual annotations. This inadequate utilization of support information impairs the performance of the model and restricts its zero-shot ability. To address these limitations, we present a novel pre-training-free network, named E fficient P oint Cloud Semantic Seg mentation for F ew- and Z ero-shot scenarios. Our EPSegFZ incorporates three key components. A Pro totype- E nhanced R egisters A ttention ( ProERA ) module and a D ual R elative P ositional E ncoding ( DRPE )-based cross-attention mechanism for improved feature extraction and accurate query-prototype correspondence construction without pre-training. A L anguage- G uided P rototype E mbedding ( LGPE ) module that effectively leverages textual information from the support set to improve few-shot performance and enable zero-shot inference. Extensive experiments show that our method outperforms the state-of-the-art method by 5.68% and 3.82% on the S3DIS and ScanNet benchmarks, respectively. Introduction Recently, F ew- S hot Sem antic Seg mentation ( FS-SemSeg ) for 3D point clouds has gained increasing research interest ( 53 ; 54 ; 22 ; 19 ; 47 ) , driven by its potential to efficiently learn from limited data and adapt to unseen categories. However, existing FS-SemSeg approaches heavily rely on fully-supervised pre-trained backbones ( 53 ; 4 ) , which can introduce biases due to domain differences between datasets. This is particularly problematic as 3D FS-SemSeg datasets are typically small in size and prone to overfitting. Additionally, the pre-training process is resource-intensive and time-consuming, which limits the practical adoption. Seg-PN ( 55 ) attempts to address this challenge by designing a non-parametric encoder. However, it discards high-frequency information to ensure robustness (as shown in Figure 1 ). We argue that despite potential noise, high-frequency features in point clouds carry essential information for precise object segmentation, particularly edge details. Therefore, developing a method that can both capture high-frequency information effectively and maintain robustness remains a core challenge in FS-SemSeg. To address this challenge, we developed a trainable attention module called Pro totype- E nhanced R egisters A ttention ( ProERA ). It utilizes trainable layers to progressively focus on high-frequency information by subtracting low-frequency components and the training process ( 46 ) . In addition to incorporating register and prototype tokens, ProERA tackles the foreground-background imbalance, placing greater emphasis on high-frequency details typically found in the minority foreground features. Figure 1 illustrates the frequency spectrum of prototype features processed by our method compared to Seg-PN ( 55 ) . Our prototype features exhibit richness and uniformity across various frequency bands, whereas Seg-PN features predominantly capture low-frequency information. (a) (b) Figure 1: Visualized frequency spectrum of embedded features from Seg-PN ( left ) and Ours ( right ) (both are pre-training-free method). Our latent features are rich and uniform across frequency bands, while Seg-PN overlooks high-frequency components. While high-frequency information is crucial, achieving a balance between high and low frequencies is essential for a more comprehensive representation. Therefore, low-frequency information should not be overlooked. It is well known that textual embeddings from large pre-trained encoders provide ample low-frequency information ( 32 ; 6 ; 1 ) , which can serve as an enhancement in this regard. However, existing models ( 53 ; 54 ; 22 ; 47 ; 55 ; 4 ; 29 ; 40 ) primarily rely on point labels and disregard textual information. Given the scarcity of labeled data in FS-SemSeg, leveraging textual cues not only supplements low-frequency information to enhance performance but also maximizes the utility of available data. To this end, we propose L anguage- G uided P rototype E mbedding ( LGPE ), which integrates support text annotations to update prototypes within a unified text-3D joint space. This approach enriches low-frequency representations even during the early training stages and helps the model’s learning process. Additionally, LGPE enables prototype construction using only text embeddings, introducing a zero-shot capability. Nonetheless, even with features containing rich high and low frequency information, optimal performance cannot be guaranteed. This is because FS-SemSeg prediction relies on query-prototype correspondence, necessitating additional mechanisms to effectively capture these relationships. Previous approaches have attempted to extract such correlations in various ways: COSeg ( 4 ) designed a cascade network that requires numerous training parameters, while Seg-PN ( 55 ) calculated Gram matrices for correlation analysis. However, these methods introduce significant computational overhead through additional parameters and substantial memory consumption. In response, we propose D ual R elative P ositional E ncoding ( DRPE ), which introduces no extra training parameters. It is the first method to utilize query-prototype relationships within the latent space as Relative Positional Encoding (RPE). Our DRPE process computes the spatial relationships between query and prototype features, incorporating this information as additional input for subsequent cross-attention operations. This approach efficiently captures query-prototype correlations as prior knowledge without resorting to computationally intensive methods. We term this approach DRPE-based cross-attention. The detailed process is illustrated in the Methodology section. In summary, the main contributions of this work are: • We propose EPSegFZ, a framework that does not require any pre-training and demonstrates the SOTA performance in 3D FS-SemSeg. • We developed the ProERA module to enhance extracted features with high-frequency, low-noise information, while its sampling strategies and registers mitigate foreground-background imbalance. • An LGPE module is proposed to dynamically update prototypes using support textual data, reducing reliance on perfect support point clouds and enabling zero-shot inference. • A DRPE-based cross-attention is designed as the first to use query-prototype spatial relationships as positional signals, accurately establishing correspondence without additional training burden. Related Work Point Cloud Semantic Segmentation Recent research can be generally divided into three main categories. The voxel-based, superpoint-based, and point-based approaches. As documented in ( 26 ; 9 ; 44 ; 21 ; 51 ) , partition the 3D space into regular voxels or superpoints before employing sparse convolutions on them. While these methods demonstrate reasonable performance, they are hindered by imprecise position data resulting from the partition process. The point-based methods ( 30 ; 35 ; 45 ; 17 ; 48 ) directly consider the features and position of each point as inputs, thus preserving natural information. KPConv ( 35 ) utilized carefully designed convolution kernels to capture multi-level information. PointNet++ ( 30 ) trains its Multi-Layer Perceptron (MLP) with point clouds sampled by different strategies to capture global and local information. PointTransformer ( 45 ) utilizes the attention mechanism to learn point-wise information effectively. Few-Shot Point Cloud Semantic Segmentation AttMPTI ( 53 ) , a groundbreaking approach, utilizes label propagation to uncover connections between prototypes and query points with features extracted by a pre-trained backbone. 2CBR ( 54 ) harnesses shared features

Read the complete text