EfficientPENet: Real-Time Depth Completion from Sparse LiDAR via Lightweight Multi-Modal Fusion

Johny J. Lopez, Md Meftahul Ferdaus, Mahdi Abdelguerfi, Anton Netchaev, Steven Sloan, Ken Pathak, and Kendall N. Niles · 2026

Read EfficientPENet: Real-Time Depth Completion from Sparse LiDAR via Lightweight Multi-Modal Fusion by Johny J. Lopez, Md Meftahul Ferdaus, Mahdi Abdelguerfi, Anton Netchaev, Steven Sloan, Ken Pathak, and Kendall N. Niles (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-18790.txt.

Introduction

Lopez and six colleagues present EfficientPENet, a two branch network for real time depth completion from sparse LiDAR and RGB images. Existing methods reach high accuracy but rely on heavy backbones that block real time use on embedded hardware. EfficientPENet replaces the usual ResNet encoder with a modernized ConvNeXt backbone, adds sparsity invariant convolutions for the depth stream, and refines predictions with a convolutional spatial propagation network. On the KITTI benchmark it uses about 4 times fewer parameters and runs 23 times faster than BP-Net, while keeping competitive accuracy.

Catalog record

CREATORJohny J. Lopez, Md Meftahul Ferdaus, Mahdi Abdelguerfi, Anton Netchaev, Steven Sloan, Ken Pathak, and Kendall N. Niles
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49106
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.18790

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

EfficientPENet: Real-Time Depth Completion from Sparse LiDAR via Lightweight Multi-Modal Fusion Johny J. Lopez, Md Meftahul Ferdaus, Mahdi Abdelguerfi, Anton Netchaev, Steven Sloan, Ken Pathak, and Kendall N. Niles Abstract Depth completion from sparse LiDAR measurements and corresponding RGB images is a prerequisite for accurate 3D perception in robotic systems. Existing methods achieve high accuracy on standard benchmarks but rely on heavy backbone architectures that preclude real-time deployment on embedded hardware. We present EfficientPENet, a two-branch depth completion network that replaces the conventional ResNet encoder with a modernized ConvNeXt backbone, introduces sparsity-invariant convolutions for the depth stream, and refines predictions through a Convolutional Spatial Propagation Network (CSPN). The RGB branch leverages ImageNet-pretrained ConvNeXt blocks with Layer Normalization, 7 × 7 7\times 7 depthwise convolutions, and stochastic depth regularization. Features from both branches are merged via late fusion and decoded through a multi-scale deep supervision strategy. We further introduce a position-aware test-time augmentation scheme that corrects coordinate tensors during horizontal flipping, yielding consistent error reduction at inference. On the KITTI depth completion benchmark, EfficientPENet achieves an RMSE of 631.94 mm with 36.24M parameters and a latency of 20.51 ms, operating at 48.76 FPS. This represents a 3.7 × 3.7\times reduction in parameters and a 23 × 23\times speedup relative to BP-Net, while maintaining competitive accuracy. These results establish EfficientPENet as a practical solution for real-time depth completion on resource-constrained edge platforms such as the NVIDIA Jetson. I Introduction Underground civil infrastructure in the United States is aging at a rate that far outpaces current maintenance capacity. The American Society of Civil Engineers assigned a cumulative grade of C - to the nation’s infrastructure in its most recent assessment [ 1 ] , with wastewater and stormwater systems receiving particularly low marks. Municipal agencies responsible for maintaining these networks increasingly rely on robotic inspection platforms equipped with cameras and range sensors to assess structural conditions without excavation [ 2 , 3 ] . However, two-dimensional images and semantic segmentation masks, while providing dense textural information and categorical defect classifications [ 4 ] , fundamentally lack the absolute geometric context required to assess structural severity [ 5 ] . A monocular 2D camera cannot reliably differentiate between a superficial surface stain on a concrete wall and a deep, structurally compromising void. Metrics required by municipal engineers, such as measuring the volume of sediment buildup or calculating the cross-sectional ovality of a pipe under active soil load, demand precise distance measurements. Light Detection and Ranging (LiDAR) sensors address this limitation by emitting laser pulses to capture time-of-flight distance measurements. However, high-density, survey-grade LiDAR scanners are physically large, expensive, and power-hungry, precluding their use on small, battery-powered inspection crawlers. Low-cost, lightweight LiDAR sensors suitable for edge robotics produce sparse point clouds: when projected onto the high-resolution camera plane, valid LiDAR returns often cover less than 5% of the total visual field [ 6 ] . This sparsity renders raw LiDAR data insufficient for detailed micro-defect analysis. Depth completion , the task of recovering a dense, continuous depth map from a highly sparse set of initial depth measurements by utilizing a corresponding high-resolution RGB image as semantic guidance, therefore becomes a necessary enabling technology for advanced 3D robotic perception [ 7 , 8 ] . The problem has attracted sustained attention in the autonomous driving community, where the KITTI depth completion benchmark [ 9 ] provides a standardized evaluation protocol with over 80,000 training pairs of sparse Velodyne HDL-64E LiDAR projections and semi-dense ground-truth depth maps. Early learning-based approaches to depth completion employed simple encoder-decoder architectures to directly regress dense depth from sparse input [ 7 , 10 ] . Subsequent work introduced two-branch fusion architectures that independently encode RGB and depth modalities before merging them at intermediate or late stages. PENet [ 11 ] established the efficacy of this late-fusion paradigm using ResNet-34 [ 12 ] variants, while BP-Net [ 13 ] proposed bilateral propagation to handle sparsity at the earliest network stage. More recently, Transformer-based architectures have been applied to depth completion: CompletionFormer [ 14 ] combines convolutional and vision transformer branches, achieving state-of-the-art KITTI results but at substantial computational cost ( \sim 1/3 FLOPs reduction relative to pure Transformers, yet still exceeding 100M parameters). Concurrently, large-scale monocular depth foundation models such as Depth Anything [ 15 ] have demonstrated that pre-trained geometric priors can transfer across domains, and knowledge distillation from such models into task-specific networks has emerged as a promising direction [ 16 ] . Despite this progress, a persistent gap remains between benchmark accuracy and deployability. The highest-performing methods on KITTI rely on heavy backbone architectures (ResNet-101/152, Swin Transformers) with parameter counts exceeding 90M and per-frame latencies above 350 ms. These computational requirements are incompatible with real-time operation on embedded GPU platforms such as the NVIDIA Jetson AGX, which imposes strict power, memory, and thermal constraints [ 17 , 18 ] . For robotic inspection systems that must process sensor data onboard without cloud connectivity, this gap between accuracy and efficiency represents a fundamental barrier to deployment. The need for efficient depth perception on edge devices is not unique to infrastructure inspection. Autonomous navigation, augmented reality, and warehouse robotics all require dense 3D understanding under similar resource constraints [ 19 ] . However, the underground inspection domain introduces additional challenges that compound the efficiency problem: zero natural lighting, highly reflective wet concrete surfaces that scatter LiDAR pulses, and repetitive cylindrical geometry that lacks the texture diversity found in outdoor driving scenes. In this paper, we present EfficientPENet, a depth completion architecture that directly addresses the accuracy-efficiency trade-off through three architectural modernizations applied to the established PENet framework. Our contributions are as follows: 1. We replace the standard ResNet-34 RGB encoder with a ConvNeXt [ 20 ] backbone that integrates Layer Normalization, 7\times 7 depthwise separable convolutions, GELU activations, and stochastic depth regularization [ 21 ] , yielding improved representational capacity at reduced computational cost. ImageNet-pretrained weights are transferred to the 6-channel input via a structured weight replication strategy. 2. We incorporate sparsity-invariant convolutions [ 6 ] in the depth encoding branch, maintaining a parallel binary validity mask to prevent zero-valued missing measurements from corrupting learned feature maps through successive network layers. 3. We introduce a position-aware test-time augmentation (TTA) strategy that applies coordinate corrections during horizontal flipping, maintaining geometric consistency for networks that receive explicit pixel position encodings. This targeted correction yields a consistent \sim 12 mm RMSE reduction without retraining. 4. We demonstrate that the proposed architecture achieves 48.76 FPS on an NVIDIA Jetson platform with 36.24M parameters, representing a 3.7\times parameter reduction compared to PENet, a 2.5\times reduction compared to BP-Net, and a 23\times latency speedup compared to BP-Net, while achieving the lowest RMSE (631.94 mm) among all compared methods on the KITTI depth completion benchmark. II Related Work II-A Sparse Depth Completion The depth completion literature has evolved from simple interpolation schemes to complex multi-branch neural architectures. Uhrig et al. [ 6 ] introduced sparsity-invariant convolutions that normalize filter outputs by the count of valid input pixels, providing a principled mechanism for handling the irregular sampling pattern of projected LiDAR data. Ma and Karaman [ 7 ] demonstrated that a single encoder-decoder network can learn the sparse-to-dense mapping

Read the complete text