ESAM++: Efficient Online 3D Perception on the Edge

Qin Liu Lavisha Aggarwal Saptarashmi Bandyopadhyay Vikas Bahirwani, Marc Niethammer Ehsan Adeli Andrea Colaco · 2026

Read ESAM++: Efficient Online 3D Perception on the Edge by Qin Liu Lavisha Aggarwal Saptarashmi Bandyopadhyay Vikas Bahirwani, Marc Niethammer Ehsan Adeli Andrea Colaco (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-29505.txt.

Introduction

ESAM++, from Qin Liu and six colleagues, lightens online three-dimensional scene perception on edge devices without graphics acceleration. Its predecessor used a costly sparse neural network for point-cloud features, consuming most of the three-dimensional inference time and blocking practical deployment. ESAM++ swaps in a sparse feature pyramid network gathering geometric detail at several scales from streaming point clouds with far less compute and a smaller model. On four segmentation benchmarks it matches the earlier accuracy while running up to three times faster at half the model size.

Catalog record

CREATORQin Liu Lavisha Aggarwal Saptarashmi Bandyopadhyay Vikas Bahirwani, Marc Niethammer Ehsan Adeli Andrea Colaco
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32360
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.29505

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ESAM++: Efficient Online 3D Perception on the Edge Qin Liu Lavisha Aggarwal Saptarashmi Bandyopadhyay Vikas Bahirwani, Marc Niethammer Ehsan Adeli Andrea Colaco Abstract Online 3D scene perception in real time is essential for robotics, AR/VR, and autonomous systems, particularly in edge computing scenarios where computational resources are limited and privacy is crucial. Recent state-of-the-art methods like EmbodiedSAM (ESAM) demonstrate the promise of online 3D perception by leveraging the Segment Anything Model (SAM) for real-time, fine-grained, and generalized 3D instance segmentation. However, ESAM still relies on a computationally expensive 3D sparse UNet for point cloud feature extraction, which accounts for the majority of the 3D inference time, hindering its practicality on resource-constrained devices. In this paper, we propose ESAM++ , a lightweight and scalable alternative for online 3D scene perception tailored to edge devices without GPU acceleration. Our method introduces a 3D Sparse Feature Pyramid Network (SFPN) that efficiently captures multi-scale geometric features from streaming 3D point clouds while significantly reducing computational overhead and model size. We evaluate our approach on four challenging segmentation benchmarks, namely ScanNet, ScanNet200, SceneNN, and 3RScan, demonstrating that our model achieves competitive accuracy with up to 3 × \times faster inference with a 2 × \times smaller model size compared to ESAM, enabling practical deployment on edge devices. 1 Introduction Real-time online 3D scene perception plays a central role in a wide range of applications, including robotics 18 ; 2 ; 41 , augmented and extended reality (AR/VR) 5 ; 23 , and autonomous navigation 1 ; 10 ; 6 . In these settings, the ability to perform accurate and low-latency segmentation directly from streaming RGB-D videos is critical. This is especially important for applications requiring on-device inference, where computational resources are limited and data privacy is a primary concern. However, most existing 3D perception approaches 38 ; 37 ; 16 are designed with server-grade hardware in mind and fall short of meeting the stringent requirements for efficiency, latency, and privacy. Recent work has made significant strides toward bridging the gap. Notably, EmbodiedSAM (ESAM) 36 introduces an online 3D perception method that leverages the Segment Anything Model (SAM) 11 ; 44 for real-time, fine-grained, and generalized 3D instance segmentation. By encoding 2D segmentation masks as 3D geometric-aware queries, ESAM enables efficient mask merging across frames and achieves strong performance on multiple benchmarks. This design not only leverages the powerful generalization capabilities of SAM but also offers a promising way to integrate 2D and 3D information in an online setting. However, the framework still relies on a computationally expensive 3D sparse UNet 3 for point cloud feature extraction, which accounts for the majority of the 3D inference time, as shown in Figure 1 . This significant computational cost hinders its practicality on resource-constrained devices. Therefore, the 3D sparse UNet is identified as the primary efficiency bottleneck in ESAM. Developing a more efficient alternative to the 3D sparse UNet for point cloud feature extraction remains an urgent and open challenge. Despite its effectiveness in capturing spatial context, the 3D sparse UNet suffers from significant efficiency limitations for point cloud feature extraction. First , the 3D sparse UNet produces the final output exclusively at the last layer that operates on data with the same resolution as the input. As a result, when the input resolution is high or the number of output channels is large, these high-resolution layers incur substantial computational costs. This is confirmed by our per-layer computational analysis in Section 3 . Second , as the input resolution progressively decreases throughout the encoder, the number of channels typically increases, resulting in a larger number of model parameters, which can pose a bottleneck in terms of model size. This is also confirmed by our per-layer computational analysis in Section 3 . Finally , the decoder of the 3D sparse UNet progressively upsamples hierarchical features to the original resolution, yielding rich multi-scale representations. However, predictions are made only at the highest resolution, leaving lower-resolution features underutilized. This design choice limits efficiency and raises a key question: can multi-scale predictions be leveraged to better exploit these intermediate layers? Architectures such as Feature Pyramid Networks (FPN) 15 , which enable multi-scale predictions, are well-established in 2D computer vision but remain underexplored in the domain of online 3D scene perception. Figure 1 : Overview of ESAM and ESAM++ . ESAM 36 (right figure) is the state-of-the-art approach for online 3D scene perception. We identify two key efficiency bottlenecks in its design: (1) the use of a frozen Visual Foundation Model (VFM) based on FastSAM 44 , and (2) a point cloud encoder built upon a 3D sparse UNet. This work focuses on optimizing the latter, while improvements to the VFM component are left for future research. The time cost is evaluated using an Intel Xeon Silver 4314 [email protected]. To address these limitations, we propose ESAM++, a lightweight and scalable online 3D scene perception framework that incorporates a 3D sparse feature pyramid (SFPN) for efficient point cloud feature extraction. SFPN reduces computational cost by limiting the number of output channels in high-resolution layers. To compensate for this reduction, it fully leverages the decoder’s feature hierarchy and employs efficient multi-scale feature aggregation. Each level of the pyramid is carefully optimized to balance spatial resolution and computational latency, making SFPN particularly well-suited for latency-sensitive applications. In addition, we further reduce the model size by removing intermediate layers that demand an excessive number of channels. We evaluate our method on four challenging 3D instance segmentation benchmarks—ScanNet, ScanNet200, SceneNN, and 3RScan—and demonstrate that our model achieves competitive performance while running up to 3 \times faster inference with 2 \times smaller model size compared to ESAM. Moreover, ESAM++ demonstrates strong potential in data-efficient settings, maintaining competitive performance even when trained on limited data. It also exhibits robustness to camera noise, highlighting its promise for RGB-only online 3D scene perception when integrated with RGB-only online reconstruction models such as CUT3R 32 . Our contributions are summarized as follows: • We analyze the computational inefficiencies of ESAM’s 3D sparse UNet and identify it as the primary bottleneck in online 3D perception. This analysis serves as a foundation for designing a more efficient backbone for point cloud feature extraction. • We introduce a novel 3D sparse feature pyramid (SFPN) for efficient multi-scale point cloud representation, specifically designed for online 3D scene perception on edge devices without GPU acceleration. • We demonstrate that ESAM++ delivers up to 3 \times faster inference and 2 \times smaller model size than ESAM, while maintaining competitive accuracy across challenging benchmarks. It also excels in low-data regimes and noisy conditions, enabling future extension for RGB-only online 3D scene perception. 2 Related Work Efficient Point Cloud Representation. PointNet 24 ; 25 pioneered direct point cloud processing. Recent advances in efficient point cloud encoders for online 3D scene perception have focused on balancing computational efficiency with high-quality feature extraction, enabling real-time understanding on resource-constrained platforms. Lightweight architectures, including voxel-based networks with sparse convolutions (e.g., PointPWC-Net 33 ), point-based models using local neighborhood aggregation (e.g., PointNet++ variants 25 ), and hybrid designs with attention mechanisms, have demonstrated improved performance with reduced latency. Innovations like KPConv 30 and PointTransformer 43 , alongside recent quantization-aware and CPU-optimized models, have further accelerated inference

Read the complete text