MFSeg: Efficient Multi-frame 3D Semantic Segmentation

Chengjie Huang, Krzysztof Czarnecki · 2025

Read MFSeg: Efficient Multi-frame 3D Semantic Segmentation by Chengjie Huang, Krzysztof Czarnecki (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-04408.txt.

Introduction

Chengjie Huang and Krzysztof Czarnecki propose MFSeg, an efficient way to label 3D point clouds across multiple frames at once, published in 2025. Semantic segmentation of lidar sweeps guides self-driving systems, and using several frames adds context but also cost. Their framework merges the sequence at the feature level rather than the raw point level.

Regularizing how features are drawn and combined keeps accuracy high while trimming computation. A light decoder based on a small multilayer network removes the need to upsample stale points from earlier frames. On two standard driving data sets, the method beat existing approaches on both quality and speed, the authors report. The design aims at systems that must label their surroundings in real time without server-class hardware.

Catalog record

CREATORChengjie Huang, Krzysztof Czarnecki
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-04408

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MFSeg: Efficient Multi-frame 3D Semantic Segmentation Chengjie Huang, Krzysztof Czarnecki Abstract We propose MFSeg, an efficient multi-frame 3D semantic segmentation framework. By aggregating point cloud sequences at the feature level and regularizing the feature extraction and aggregation process, MFSeg reduces computational overhead while maintaining high accuracy. Moreover, by employing a lightweight MLP-based point decoder, our method eliminates the need to upsample redundant points from past frames. Experiments on the nuScenes and Waymo datasets show that MFSeg outperforms existing methods, demonstrating its effectiveness and efficiency. I Introduction LiDAR-based 3D semantic segmentation is an essential task for 3D scene understanding in robotic applications such as autonomous driving. Accurate segmentation of LiDAR point clouds allows robotic systems to identify and classify objects in their environment, ensuring safe and reliable navigation. LiDAR point clouds are typically captured sequentially, and utilizing this temporal nature has been shown to enhance various 3D perception tasks. In the context of 3D semantic segmentation, recent work has explored feature-level fusion across consecutive frames [ 18 , 20 , 17 ] . While these approaches have shown promise, they rely on structured representations of point clouds such as range image to align intermediate features across frames or are extensions of older architectures. This limits their applicability to the current state-of-the-art (SOTA) methods in 3D perception such as point transformers [ 31 , 24 , 23 , 9 ] . An alternative approach involves aggregating multi-frame point clouds at the input level via concatenation. This is widely adopted in 3D object detection as the multi-frame point clouds contain richer geometric information and can improve detection accuracy [ 1 , 29 , 2 , 8 ] . In 3D semantic segmentation, we find this approach to be also beneficial. As shown in Fig. 1 , multi-frame point clouds provide a modest overall improvement (2.3%) in mean Intersection over Union (mIoU) performance, and a noticeably higher gain (up to 5.6%) in detecting vulnerable road users such as pedestrians and cyclists. This highlights the benefits of multi-frame point clouds for 3D semantic segmentation task in real-world applications. Fig. 1: Relative mIoU improvement between various levels of multi-frame point cloud concatenation and single-frame input. Performance are evaluated using a SparseUNet [ 5 ] on the Waymo dataset [ 19 ] . Fig. 2: Inference time of PTv3 [ 23 ] and MFSeg under a range of point cloud sequence length on the nuScenes dataset [ 1 ] (left) and Waymo dataset [ 19 ] (right). PTv3-ds denotes that the point cloud is downsampled using a voxel-grid with size 0.05 \mathrm{m}^{3} , while PTv3 is evaluated with minimum input downsampling. Despite the benefits, multi-frame point clouds pose unique challenges for the SOTA point-based methods. Firstly, the increase in the number of points introduces computational challenges. For instance, a 64-beam LiDAR sensor can produce an average of 177k points per frame [ 19 ] , and concatenating six frames can result in over one million points. Processing these dense point clouds requires significant computation, as illustrated by the latency of a SOTA method PTv3 [ 23 ] in Fig. 2 . Secondly, current methods follow the U-Net design and require upsampling point features in the decoder stage. This introduces unnecessary computational overhead for multi-frame point clouds, as predictions are typically only needed for points in the current frame. In this work, we propose MFSeg, an efficient multi-frame 3D semantic segmentation framework. MFSeg aggregates point cloud sequences at the feature level while retaining the properties of input-level concatenation via a novel aggregation pipeline and regularization strategy. Unlike prior work, our method does not assume a structured feature representation and thus can be applied to SOTA point-based backbones such as point transformers. To address the computational challenges, we employ a local feature extractor to reduce the input density, allowing MFSeg to efficiently process multi-frame inputs and features. Furthermore, we introduce a simple MLP-based point decoder to replace the traditional symmetrical decoder in the U-Net design, avoiding the unnecessary upsampling of point features. These components enable MFSeg to achieve SOTA performance with low latency, as illustrated in Fig. 2 . Experiments on two large-scale semantic segmentation datasets, nuScenes [ 1 ] and Waymo [ 19 ] , demonstrate that MFSeg can outperform existing methods. II Related Work 3D semantic segmentation methods typically fall into three categories: projection-based, voxel-based, and point-based, depending on the point cloud representation that is being used. Projection-based methods [ 21 , 22 , 14 , 25 , 30 ] use either range-image or bird’s-eye view (BEV) projections of point clouds, allowing traditional 2D convolutional neural networks (CNNs) to be used. However, these representations sacrifice the detailed 3D geometric information, limiting their performance. Voxel-based methods [ 3 , 5 ] voxelize the point clouds into a structured representation, allowing 3D CNNs to be applied. Although 3D CNNs are computationally demanding, the development of sparse convolutional networks [ 6 ] has greatly improved their efficiency. Point-based methods directly process unstructured point clouds without the need for voxelization or projection. Early works such as PointNet [ 15 ] and PointNet++ [ 16 ] laid the foundation for this approach, but their high computational cost limited their application in large outdoor scenes. With the recent advent of efficient transformers [ 4 ] and the introduction of transformer-based segmentation methods [ 31 , 24 , 23 , 9 ] , point-based methods have become the current SOTA. Many approaches have sought to improve 3D segmentation performance by leveraging additional sources of information. One line of work focuses on multi-view fusion, which involves combining multiple representations of the point cloud [ 12 , 32 , 26 ] or integrating other sensory modalities [ 27 , 11 , 10 ] to create richer feature representations. Another line of work aims to exploit temporal information by aggregating sequential point cloud data. Using the range-image representation, SpSequenceNet [ 18 ] proposes a cross-frame global attention and local interpolation to fuse CNN features from the previous frame. Meta-RangeSeg [ 20 ] instead computes a residual range image representation to capture the temporal difference between frames. However, range image in the multi-frame context can suffer from misalignment due to ego movement. Alternatively, TemporalLatticeNet [ 17 ] extends LatticeNet [ 13 ] and fuses temporal features using recurrent neural networks and a novel abstract flow module. MSeg3D [ 10 ] , while focusing on multi-modality fusion, uses concatenated point clouds for their point cloud branch to improve segmentation performance. In contrast, MFSeg uses a novel formulation of feature space aggregation to retain the benefits of concatenated point cloud. Moreover, unlike voxelized or range-image representations, MFSeg does not rely on structured features, which allows it to utilize SOTA point-based backbones. III Method MFSeg consists of four key components: (i) a local feature extractor (LFE) for capturing local point cloud features, (ii) a novel feature aggregator that fuses features from multiple frames, (iii) an off-the-shelf backbone encoder for processing the aggregated features, and (iv) an efficient point decoder that generates per-point predictions directly from the encoder output. An overview of the MFSeg architecture is provided in Fig. 3 . In the following sections, we describe each component in detail. III-A Local Feature Extraction Directly processing dense LiDAR point clouds or

Read the complete text