RPINet: Dual-branch attention-guided fusion of remote sensing images and point clouds for urban scene segmentation Authors: Zhe Jing, Zhengguo Yan Published: 2026 ABSTRACT Semantic segmentation of large-scale urban point clouds is a fundamental yet challenging task due to the complex spatial structures, massive data volume, and irregular distribution of points. Existing methods typically rely solely on 3D geometry or naively fuse 2D and 3D features, leading to limited performance in capturing both fine-grained details and global semantic consistency. In this paper, we propose RPINet, a novel Remote-Projection and Intelligent Network that effectively integrates 2D remote sensing images with 3D point cloud data for enhanced semantic segmentation. RPINet adopts a dual-branch architecture where the 3D point branch extracts spatial features using PointNet++, transformers, and graph convolutional networks, while the 2D image branch leverages a Gaussian splatting projection and CNN-based encoding to retain texture and contour information. A hybrid attention-based fusion module dynamically weights intra-modal and inter-modal dependencies, enabling deep semantic interaction between modalities. In addition, we introduce an adaptive sampling strategy and a multi-objective loss function to optimize segmentation accuracy and geometric consistency. Extensive experiments on the challenging SensatUrban dataset demonstrate that RPINet achieves state-of-the-art performance with a mean IoU of 66.5%, outperforming existing methods by a significant margin. Our model also shows strong generalization ability on unseen datasets, confirming its robustness and practical applicability to real-world urban scenarios. 1 Introduction Urban-level point cloud segmentation task is an important basis for realizing semantic scene [1]. Urban-scale 3D point cloud semantic segmentation is fundamental for accurate scene understanding in applications such as autonomous driving, urban planning, and remote sensing. Unlike small-scale indoor environments, urban-scale point clouds exhibit extreme complexity, with billions of points and significant variation in density, occlusion, and noise [2–5]. The challenges in urban-level segmentation are twofold: (1) the irregular and sparse nature of point clouds limits the direct application of standard convolutional methods, and (2) the need to preserve fine-grained geometric features while capturing large-scale spatial context requires models to balance local detail extraction with global reasoning [6–10]. Despite the growing attention on urban-scale semantic segmentation, existing methods still face critical limitations. Traditional point-based approaches, such as PointNet++ and RandLA-Net, are effective in local geometry encoding but lack global contextual reasoning. Voxel-based and sparse convolutional methods suffer from discretization errors and high memory costs. Furthermore, multi-modal fusion techniques that combine 2D imagery with 3D point clouds often struggle with projection-induced information loss and insufficient cross-modal alignment. Recent efforts using cross-attention mechanisms have shown promise but are typically constrained to small-scale indoor settings or well-aligned RGB-D data. Thus, developing an effective and scalable fusion strategy that can fully leverage the complementary strengths of 2D and 3D modalities remains a significant open problem. To address these challenges, remote sensing data, such as aerial imagery, is often used to complement 3D point clouds with rich texture and color information [11]. However, effectively integrating multi-modal data for robust urban scene segmentation remains a critical problem, especially when projecting 3D geometry into 2D space may cause loss of structural information [12–14]. In this paper, we propose RPINet, a Remote-Projection and Intelligent Network that fully leverages the complementary strengths of 2D remote imagery and 3D point clouds for urban-scale semantic segmentation. RPINet introduces a two-branch architecture: the Remote Image (RI) branch performs adaptive denoising and feature enhancement on projected 2D data, while the Point Cloud (PI) branch applies advanced clustering, multi-scale feature extraction, and adaptive sampling directly on the 3D points. The extracted features are fused via a hybrid attention mechanism designed to capture both intra-modal and inter-modal dependencies. Furthermore, we introduce a comprehensive loss function that jointly optimizes information‌‌ preservation during projection and segmentation accuracy [15–17] (Fig 1). Motivation. From the perspective of remote sensing, using point cloud data for target detection faces numerous challenges that need to be urgently addressed, such as noise interference, feature fusion, information loss, and large computational load. Example image and point-cloud views in this figure are adapted from the SensatUrban dataset by Hu et al. [1] and used under the MIT License from the official SensatUrban repository. The contributions of this work are summarized as follows: We design an adaptive projection method using Gaussian splatting to minimize information loss when converting 3D point clouds into 2D representations. We design an adaptive projection method using Gaussian splatting to minimize information loss when converting 3D point clouds into 2D representations. We propose an advanced point cloud feature extraction pipeline combining DBSCAN-based clustering, PointNet++, transformer-based attention, and graph convolutional networks (GCN) to enhance multi-scale geometric feature learning. We propose an advanced point cloud feature extraction pipeline combining DBSCAN-based clustering, PointNet++, transformer-based attention, and graph convolutional networks (GCN) to enhance multi-scale geometric feature learning. We develop a hybrid attention-based cross-modal feature fusion strategy that effectively integrates 2D texture and 3D structural features. We develop a hybrid attention-based cross-modal feature fusion strategy that effectively integrates 2D texture and 3D structural features. We introduce a multi-term loss function that balances information preservation, segmentation accuracy, and spatial consistency. We introduce a multi-term loss function that balances information preservation, segmentation accuracy, and spatial consistency. Experimental results on the SensatUrban dataset demonstrate that RPINet achieves superior performance compared to state-of-the-art methods, particularly in preserving fine-grained details and maintaining global semantic coherence across large urban scenes. The remainder of this paper is organized as follows. Section 2 reviews related work on urban point cloud segmentation and multi-modal fusion. Section 3 introduces the proposed RPINet architecture in detail, including the dual-branch design, fusion mechanism, and loss functions. Section 4 presents experimental evaluations on the SensatUrban dataset, including performance comparisons and ablation studies. Finally, Section 5 concludes the paper and discusses future research directions. 2 Related work 2.1 Urban-scale point cloud semantic segmentation With the increasing adoption of LiDAR and UAV technologies, large-scale outdoor point cloud semantic segmentation has become an essential task in urban scene understanding. Unlike indoor scenes, urban-scale point clouds are characterized by vast spatial extents, irregular density, and complex object distributions, which present significant challenges for accurate segmentation. Early approaches relied heavily on handcrafted geometric features, such as curvature, planarity, or height histograms, combined with traditional machine learning classifiers (e.g., random forests, support vector machines). However, these methods often failed to generalize across diverse urban environments due to the limitations of manually designed descriptors. Voxelization-based methods attempted to discretize point clouds into regular grids for convolutional processing. However, such approaches suffer from high memory consumption and loss of fine geometric details due to quantization errors, especially when scaling to large outdoor scenes. To address these challenges, deep learning-based methods have emerged, with PointNet++ [18] and RandLA-Net [19] being representative. PointNet++ introduced hierarchical feature learning via local neighborhood set abstraction, capturing fine-grained geometric structures. RandLA-Net proposed random sampling with local feature aggregation to achieve efficient large-scale point cloud processing. Despite these advances, these methods still struggle to capture long-range dependencies and global contextual relationships essential for understanding complex urban environments. Recent work has explored multi-scale feature extraction and contextual modeling to enhance segmentation. For instance, KPConv [20] uses kernel point convolutions to process irregular neighborhoods, and SPVNAS [21] applies sparse convolution with network architecture search. Nevertheless, the difficulty of effectively balancing local detail preservation with global semantic understanding remains a core limitation in large-scale outdoor point cloud segmentation. 2.2 Multi-modal fusion for point cloud segmentation Given the complementary nature of different data modalities, fusing point clouds with additional information such as RGB images, aerial remote sensing data, or depth maps has proven to enhance semantic segmentation performance. Remote sensing images provide rich texture and color information, while point clouds contribute accurate 3D geometric structure, making their fusion highly desirable for comprehensive scene understanding. Early fusion strategies projected point clouds onto 2D grids or panoramic images and applied conventional convolutional neural networks (CNNs) to extract features [22]. However, such projections inevitably introduce information loss, as continuous 3D structures are collapsed into discrete 2D spaces, disrupting spatial relationships. Mid-level and late fusion methods attempt to independently extract features from both modalities and combine them at various network stages. Techniques such as feature concatenation, addition, and multi-branch architectures have been widely adopted [23]. While these approaches improve segmentation accuracy by leveraging multi-modal cues, they often lack fine-grained alignment between 2D and 3D features, leading to suboptimal fusion effectiveness. Recent studies explore cross-domain feature alignment and joint representation learning to better bridge the gap between modalities. For example, fusion strategies based on shared latent spaces, feature correlation learning, or attention-guided integration aim to dynamically weigh contributions from each modality. Despite these advances, achieving accurate point-to-pixel correspondence and mitigating the structural inconsistencies between unstructured point clouds and regular image grids remains an open problem, especially in large-scale and noisy urban scenes. 2.3 Cross-Modal attention and feature enhancement Attention mechanisms have gained prominence for their ability to model long-range dependencies and capture complex relationships in both intra-modal and cross-modal learning. Self-attention, as employed in transformer-based architectures, enables point cloud networks to consider global context by computing pairwise interactions among all points [24]. This is particularly valuable in urban scenes, where object instances may be large, and spatial dependencies extend across wide areas. Cross-modal attention further extends this idea by enabling interactive learning between different modalities. In multi-modal segmentation, cross-attention modules allow point cloud features to attend to corresponding image features and vice versa, dynamically fusing texture and geometric cues. Methods like FusionTransformer [25] have successfully applied such mechanisms in indoor environments, demonstrating that hybrid attention can significantly enhance segmentation accuracy. However, most existing cross-modal attention strategies have been developed for relatively small-scale datasets with well-aligned modalities, such as RGB-D indoor scans. When applied to urban-level outdoor scenes, these approaches face challenges from scale variation, partial data overlap, and noise in both modalities. Furthermore, projection from 3D to 2D may cause irreversible information loss, limiting the effectiveness of downstream attention mechanisms. 2.4 Motivation for RPINet Considering the limitations in current approaches, RPINet is proposed to tackle the challenges of urban-scale multi-modal semantic segmentation. First, we adopt Gaussian splatting during the 3D-to-2D projection to preserve local geometric continuity and reduce information loss. Second, we design a powerful point cloud feature extraction pipeline that combines DBSCAN-based clustering for region organization, PointNet++ for local geometric encoding, transformers for global contextual modeling, and graph convolutional networks (GCN) for relational refinement. While more recent architectures such as Point Transformer [24] offer strong standalone performance, their global self-attention mechanisms overlap significantly with the dedicated Transformer module already present in our pipeline. Consequently, PointNet++ is deliberately retained as a lightweight and well-suited local encoder, avoiding architectural redundancy and preserving inference efficiency. This allows RPINet to capture both fine details and large-scale structure across the urban scene. To fully exploit the complementary nature of 2D and 3D data, RPINet integrates a hybrid cross-modal attention mechanism, enabling interactive learning between 2D remote sensing features and 3D point cloud representations. Finally, a unified loss function is introduced, balancing information preservation, segmentation accuracy, and spatial smoothness. By holistically addressing the challenges of large-scale data handling, multi-modal feature fusion, and context-aware segmentation, RPINet sets a new direction for robust, accurate, and efficient urban-scale point cloud semantic segmentation. 3 Methodology 3.1 Overall framework RPINet employs a dual-branch architecture to integrate 3D point cloud data with 2D remote sensing imagery for semantic segmentation. Let P=(pk,ck),k=1N be the input point cloud with N points, where each point pk=(xk,yk,zk)∈ℝ3 is a 3D coordinate and ck=(rk,gk,bk) is the corresponding color (RGB) information. The goal is to predict a semantic label ℓ^k for each point pk. We formulate the segmentation task as learning a function D(P∣θ), parametrized by θ, that outputs a probability distribution over C semantic classes for each point. The predicted label for point pk is then ℓ^k=argmaxcDk,c(P∣θ), where Dk,c is the predicted probability of class c for point k (Fig 2). Method. Our method employs a dual-branch modeling approach using point cloud data. It fully utilizes the 2D features of 3D point clouds in remote sensing data and integrates the inherent information of the 3D point clouds for feature fusion. Multiple loss functions are used to ensure information alignment, ultimately achieving accurate target classification. Any illustrative point-cloud or label snippets shown in this schematic are adapted from the SensatUrban dataset by Hu et al. [1] and used under the MIT License from the official SensatUrban repository. To achieve this, RPINet processes the data through two coordinated branches: a Remote-Image branch that focuses on texture and appearance features in 2D, and a Point-Cloud branch that emphasizes 3D geometric features. The Remote-Image (RI) branch projects the point cloud onto a 2D image plane (using known sensor calibration or a top-down projection) and applies image processing techniques to enhance and extract multi-scale features. The Point-Cloud (PC) branch operates directly on the 3D points, performing clustering, local-to-global feature extraction (PointNet++ style encoding, transformer-based attention, and graph convolution), as well as adaptive sampling and multi-scale processing to capture geometric details at various scales. The high-level features from both branches are then fused via an attention-guided module that learns to highlight relevant intra-modal and cross-modal information. Finally, a segmentation head produces the per-point label probabilities. We also design a multi-term loss function to jointly optimize semantic accuracy and structural consistency. The following subsections describe each component in detail, with rigorous mathematical formulation and justification. 3.2 Remote-image branch: Gaussian splatting and 2D feature extraction 3.2.1 3D-to-2D projection with Gaussian splatting‌‌. To leverage 2D convolutional networks on irregular point clouds, we first project the 3D points onto a 2D grid, creating a pseudo-image. A naive projection can cause aliasing and information loss, especially in sparse regions. We address this by applying Gaussian splatting as a pre-processing step, which smooths the point cloud and distributes each point’s influence to its neighbors before projection. For any point pk=(xk,yk,zk)∈P and a neighboring point pj = (xj, yj, zj), we define the Gaussian weight (influence) of pk on pj as: where σ is the standard deviation of the Gaussian kernel controlling the spatial extent of the influence. This weight decays exponentially with Euclidean distance, so closer points have a larger mutual influence. Intuitively, Gaussian splatting performs a locally weighted smoothing of the point cloud: each point “spreads” a fraction of its value (position or features) to nearby points, akin to distributing mass in a continuous density map. This process reduces noise and fills small gaps by ensuring that isolated or sparse points contribute to the representation of their vicinity, thereby enhancing local continuity. Using the weights from (1), we adjust each point’s position toward its neighbors. For a given point pk and one of its neighbors pj, the neighbor can be shifted slightly toward pk by: where pj′ is the updated neighbor position. In practice, we iteratively or simultaneously apply such shifts for all points in a local neighborhood, effectively smoothing the point cloud geometry. The influence of pk on pj is strongest when pj lies very close to pk, and diminishes for distant points. As a result, Gaussian splatting preserves fine local structures while mitigating outliers: points within a tight cluster move closer together (denoising small isolated noise), and small gaps are bridged by intermediate influence, but large-scale geometry remains unchanged since distant points have negligible effect on each other. After Gaussian splatting, the point distribution is more uniform and continuous, which is beneficial for the subsequent projection. 3.2.2 Quantized projection to image plane. Once the point cloud has been smoothed, we project the 3D points onto a 2D plane to generate a rasterized representation. Given an appropriate viewpoint (e.g., an overhead orthographic view for aerial data or a perspective view with known camera intrinsics/extrinsics), we define a projection function. For simplicity, assuming an orthonormal projection, we map coordinates by quantizing the x and y values of each point to integer pixel coordinates: where (uk, vk) are the 2D pixel coordinates of point pk on the image plane, and s is a quantization factor (grid resolution) that determines the size of each pixel. This operation effectively bins points into pixels: all points falling into the same (⌊x/s⌋,⌊y/s⌋) bin are mapped to the same image pixel. In a more general camera model, π(·) would incorporate perspective projection equations using calibration parameters, but the end result is the same—a set of image coordinates corresponding to the 3D points. We retain the association between each point pk and its pixel (uk, vk) for later fusion. The projection compresses the 3D structure onto a 2D grid while, thanks to Gaussian splatting, preserving neighborhood relationships. Specifically, the Gaussian smoothing ensures that if a 3D structure is somewhat sparse, its projection will still carry a continuous intensity or feature distribution rather than isolated pixels, thereby reducing information loss during quantization. The projection yields an initial 2D image I (with channels for accumulated color or density) of size H × W pixels covering the area of interest. Each pixel in I encodes the presence (and possibly aggregated color/features) of points in the corresponding 3D column of space. This forms the input to the image processing stage of the RI branch. 3.2.3 Multi-scale CNN feature extraction. With the 2D projection I in hand, we employ a 2D convolutional neural network to extract high-level features that capture textures, edges, and contextual cues from the remote sensing image. We denote the image feature extraction as: where F(I)∈ℝH×W×C1 is the resulting feature map (with C1 channels) produced by the CNN ΦCNN. This CNN can be a standard encoder (e.g., ResNet or U-Net backbone) that progressively extracts features from low-level (edges, textures) to high-level (object-level context). We adopt a multi-scale feature extraction strategy to ensure both local details and global contexts are captured. In practice, this can be implemented by a feature pyramid network or by using convolution layers with different receptive fields. We obtain feature maps at multiple scales (levels of resolution) and combine them to form a comprehensive representation: where Fℓ(I) is the feature map extracted at scale (or level) ℓ, and wℓ is a learned weight reflecting the importance of that scale. The summation in (5) indicates a weighted fusion (or it could be a concatenation followed by a 1x1 convolution) of multi-scale feature maps, yielding a final 2D feature representation Ffusion(I) that contains information from coarse (contextual) to fine (detailed) levels. By doing so, the RI branch ensures that features like building edges or tree textures (fine details) as well as broader layout cues (e.g., a cluster of pixels forming a large roof or road, giving context) are both preserved. The output of this branch is a dense feature map covering the entire projected scene. This Remote-Image branch output will later be aligned and fused with the point cloud branch output. Crucially, the RI branch provides rich color and texture cues that complement the geometric features from the point cloud branch, helping to distinguish objects that might be geometrically similar but visually different (for instance, a tree vs. a lamp post might have similar shapes but very different image textures). 3.3 Point-cloud branch: Multi-scale geometric feature learning In parallel to the RI branch, the Point-Cloud branch processes the 3D data directly to extract robust geometric features. This branch is composed of several modules that together learn point representations encoding local geometry, global structure, and inter-point relationships. We denote the final output of this branch as F(P)=hi(3D)i=1N, where hi(3D)∈ℝC2 is the feature vector learned for point pi. Below we detail each module in this branch and how it contributes to semantic segmentation. 3.3.1 Geometric clustering for regional features. Instead of treating every point in isolation, we first organize the point cloud into meaningful clusters that reflect local surfaces or objects. Clustering reduces data complexity and provides a structural context for each point. We employ a density-based clustering (DBSCAN) to group points that are spatially close and densely distributed. Formally, for DBSCAN we define a neighborhood radius ε and minimum points m. A point pi is considered a core point if it has at least m neighbors within distance ε. DBSCAN then defines an equivalence relation on points: pi ∼ pj if there exists a chain of points pk0=pi,pk1,…,pkL=pj such that each consecutive pair lies within ε of each other. All points related by ∼ form a cluster Ck. This identifies contiguous high-density regions (e.g., points on the same facade or ground patch). To capture larger-scale groupings, we also construct a graph G=(V,E) over clusters or points, where each node represents a point (or an initial cluster) and edges e=(i,j)∈E connect points that are in proximity or have similar attributes (color, normal, etc.) even if they belong to different DBSCAN clusters. For example, we add an edge between pi and pj if ||pi−pj||<τd (distance threshold) or if their feature similarity (e.g., normal vector difference or color difference) is above a threshold τf. We can then apply graph-based clustering (such as spectral clustering or community detection on G) to merge the DBSCAN clusters into larger super-clusters or supervoxels C′u. The result of this two-stage clustering is a set 𝒞=C1,C2,…,CK of clusters (ranging from small, dense regions to broader structures). We assign each point pi a cluster label c(i)∈1,…,K indicating which cluster it belongs to. We also compute a cluster-level feature descriptor for each cluster Ck, for instance by averaging or pooling the point features in that cluster: where f(pi) could be the initial feature of point pi (such as a positional encoding or color) or a learned intermediate feature. These cluster descriptors hCk provide a compact representation of region-wise properties. The clustering step benefits segmentation by capturing regional context: points within the same cluster likely belong to the same object or surface (hence share a label), so the model can use the cluster identity as a strong cue for segmentation consistency. Moreover, clustering helps focus the later stages of the network on meaningful groups rather than isolated points, improving efficiency and robustness to noise (outliers may fall into small clusters or remain unclustered and can be handled separately). 3.3.2 Local feature encoding (PointNet++). It is important to note that PointNet++ is adopted here in a specific and limited role: it serves solely as a local neighborhood feature encoder, not as the primary backbone of the network. Global contextual modeling is explicitly delegated to the dedicated self-attention Transformer in Module 3, while relational refinement is handled by the GCN in Module 4. Replacing PointNet++ with a globally-aware architecture such as Point Transformer [24] or PointNeXt would introduce functional redundancy with the downstream Transformer module, increasing model complexity without commensurate benefit. PointNet++ is therefore deliberately chosen as a lightweight and effective component for this local encoding stage, consistent with our design goal of maintaining inference efficiency. Next, we extract point-wise features that capture fine-grained local geometry. Drawing inspiration from PointNet++’s set abstraction, we consider each point pi and its neighborhood N(i) (for example, all points within radius r of pi, or the k nearest neighbors). We then apply a small point-based neural network (MLP) on each neighbor’s relative position and initial feature (e.g., relative coordinates and color) and aggregate these to form a local descriptor for pi. Formally, let g(·) be an MLP that maps a concatenation of a neighbor’s relative coordinates and features to a latent representation. We compute: where [·,·] denotes concatenation of the geometric offset pj − pi and the neighbor’s feature f(pj) (which initially could be just color or height, and later can be replaced by intermediate learned features), and the max operator is applied element-wise to the set of neighbor feature outputs to yield an invariant summary (one could also use average pooling). The result hilocal is a fixed-length feature vector encoding the shape of the local neighborhood around pi (e.g., flat surface vs. edge vs. corner) and the distribution of any attributes. This is analogous to the local feature extraction in PointNet++ which captures fine geometry like curvature or roughness around each point. By computing hilocal, the network gains an understanding of small-scale structures, which is crucial for distinguishing adjacent classes (for instance, ground vs. curb vs. wall might differ in the curvature or normal variation in their local neighborhoods). 3.3.3 Global context encoding (self-attention transformer). While local features are important, points in large-scale scenes can be far apart yet related (e.g., two walls on opposite sides of a street are both walls and part of the same street scene). To capture long-range dependencies and global context, we feed the point features into a self-attention module inspired by transformer architectures. We treat the set of point features hilocal,i=1N as a sequence of “tokens.” For each point i, we compute query, key, and value vectors via learnable linear projections of the local feature: where WQ,WK,WV∈ℝd×din are projection matrices (with din the dimension of hlocali and d the chosen dimensionality of the queries/keys). The self-attention mechanism allows each point i to attend to all other points j through the compatibility of their query and key. We compute attention weights and updated features as: Here αij represents the attention weight that point i assigns to point j; it is essentially a learned affinity between points i and j normalized across all j. The updated feature higlobal is a weighted sum of value vectors from all points, so it integrates information from the entire point cloud, with more focus on points that are deemed relevant to i. In practice, we use multi-head self-attention: the projection in (8) and computation in (9) are done H times (with independent WQh,WKh,WVh for head h=1…H and smaller d per head), and the results are concatenated to form a rich global feature for each point. The transformer’s self-attention endows each point’s representation higlobal with awareness of the overall scene layout and long-range interactions. For example, a point on a car will receive contextual information from other points on the same car (to understand the extent of that object) and even from points on the road underneath or buildings nearby, as the model learns to attend to relevant contexts for segmentation. This global feature helps disambiguate cases where local geometry alone is insufficient (many objects might have planar surfaces locally, but their global arrangement or co-occurrence helps identify what they are). 3.3.4 Relational refinement with graph convolution. After obtaining global context features, we further refine the point representations by incorporating local relational structure explicitly, using Graph Convolutional Networks (GCNs). While the self-attention treated all point pairs with equal potential, a GCN can enforce that only points with actual connections (e.g., spatial neighbors or within the same cluster) exchange information, thus reinforcing locality and smoothing within object regions. We construct an adjacency relation Aij for points, for example Aij = 1 if point j∈N(i) (a neighbor in the point cloud within radius or KNN) or if c(j)=c(i) (points belong to the same cluster from step 1), and Aij = 0 otherwise. A simple first-order graph convolution update for each point i is then: where W0, W1 are learnable weight matrices and σ(·) is a nonlinear activation (e.g., ReLU). In this formulation, each point i updates its feature by combining its current feature higlobal with an average of the features of its neighbors j∈N(i). This is essentially a graph smoothing or averaging that promotes similarity among connected points. More advanced GCN variants could weight each neighbor in the sum by a function of hjglobal and higlobal or by edge attributes, but the essence remains: points that are spatially or semantically connected (like belonging to the same cluster or object) will align their features. This graph convolutional refinement encourages consistency: if a point is on a planar roof and its neighbors are also on that roof, their features after GCN will become more similar, making it easier for the classifier to assign them the same label. Conversely, at object boundaries, neighbors might belong to different clusters or be excluded from the adjacency, allowing discontinuities to remain where appropriate. After the GCN step, we obtain the refined feature hiGCN for each point pi. At this stage, each hiGCN encodes multi-scale information: (i) the local neighborhood geometry (from PointNet++), (ii) the global context (from transformer attention), and (iii) the relational context enforcing that points on the same structure have similar representations (from GCN). We denote this entire point feature extraction process as a function hiGCN=F3D(pi,{pj}),parameters), summarizing that for each point the branch produces a C2-dimensional feature vector. 3.3.5 Adaptive point sampling (efficiency and focus). Processing every point in a large-scale cloud (with millions or more points) is computationally expensive and sometimes redundant. We incorporate an adaptive sampling strategy to select a subset of points that are most informative for segmentation, reducing the computational burden while maintaining performance. The idea is to reduce the number of points from N to M (M < N) for certain expensive operations, focusing on critical areas (e.g., boundaries, distinct structures) and down-sampling repetitive or homogeneous regions. One approach is importance-based sampling. We assign each point pi an importance score I(pi) that reflects how much that point contributes to segmentation accuracy. The score can be hand-crafted (e.g., higher for points with large curvature, significant height difference from neighbors, or near color edges in the image) or learned (e.g., using a small network that predicts uncertainty or feature magnitude). We then choose the top M points with the highest scores. Mathematically, if S⊂1,…,N is the index set of selected points (with |S| = M), we want meaning we aim to maximize the total importance of retained points. In practice, solving (11) exactly is combinatorial, so we use greedy heuristics or thresholding on I(pi) (e.g., pick all points above a certain importance value). This tends to keep points on edges or uncommon classes (high I) while dropping points in flat, uniformly labeled areas (low I), thus preserving segmentation-critical structures. Another approach is reinforcement learning-based sampling. We can frame point selection as a sequential decision process where a policy π chooses points to include such that a reward (e.g., segmentation accuracy or IoU on a validation set) is maximized. A neural network can be trained to output a probability of selecting each point, and a reinforcement learning algorithm (like policy gradients) updates π based on the reward. For example, an agent could learn to mimic farthest point sampling (to ensure coverage) but also to focus on high-curvature areas (to ensure detail). A differentiable approximation of sampling (such as the SampleNet approach) creates a soft selection by generating new points pj~ as weighted combinations of original points: with weights wij that form a sparse selection matrix (each pj~ ideally equals one of the input points if one weight is 1 and others 0). This can be trained end-to-end by backpropagating a surrogate loss that encourages picking points that lead to good segmentation. Regardless of the mechanism, the output of adaptive sampling is a reduced set PS=pi:i∈S that is passed through the feature extraction pipeline (or one could apply sampling at multiple stages, such as at each layer of a hierarchical set abstraction, described next). By focusing computation on the most informative points, the network can allocate more capacity to complex regions. Moreover, sampling can act as a form of data augmentation and regularization: if at each epoch a slightly different subset of points is chosen, the model is exposed to varied inputs and must rely on robust features rather than overfit to every single point. Our adaptive sampling thus strikes a balance between efficiency and accuracy, ensuring that minimal critical information is lost when reducing point density. 3.3.6 Multi-scale feature pyramid. Urban scenes contain structures of vastly different scales, from small objects like street signs to large expanses like roads and buildings. To achieve scale-invariant segmentation, we adopt a multi-scale processing strategy for the point cloud akin to an encoder-decoder pyramid. We generate a hierarchy of point sets with decreasing resolution: P(0) = P is the original set of N points, and for each level ℓ=0,1,…,L−1 we sample a subset P(ℓ+1)⊂P(ℓ) such that |P(ℓ+1)|<|P(ℓ)|. The sampling can be random, via farthest point sampling (FPS), or using the aforementioned adaptive sampling. Points in P(ℓ+1) can be thought of as representative points (or centroids) for neighborhoods in P(ℓ). For each representative point q∈P(ℓ+1), we define its receptive field in the lower level as the set of points within a radius rℓ: p∈P(ℓ):|p−q|