LiFT: Lightweight, FPGA-tailored 3D object detection based on LiDAR data

Konrad Lis, Tomasz Kryjak, Marek Gorgoń · 2025

Read LiFT: Lightweight, FPGA-tailored 3D object detection based on LiDAR data by Konrad Lis, Tomasz Kryjak, Marek Gorgoń (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-11159.txt.

Introduction

Self-driving hardware in cheap vehicles cannot carry a data-center chip, so object detection must shrink to fit an FPGA. Konrad Lis, Tomasz Kryjak, and Marek Gorgoń designed LiFT around the chip's hard limits from the start. The limits were strict: an operations budget, eight-bit arithmetic throughout, flat cell-based processing instead of 3D voxels, and few skip connections.

Novel pillar features and a careful input quantization scheme clawed back accuracy within that budget. The finished detector ran at about twenty-one billion operations, among the leanest in its class. On the nuScenes benchmark it ranked first among comparable minimal methods. The authors will release the code.

Catalog record

CREATORKonrad Lis, Tomasz Kryjak, Marek Gorgoń
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-11159

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LiFT: Lightweight, FPGA-tailored 3D object detection based on LiDAR data Konrad Lis, Tomasz Kryjak, Marek Gorgoń Abstract This paper presents LiFT, a lightweight, fully quantized 3D object detection algorithm for LiDAR data, optimized for real-time inference on FPGA platforms. Through an in-depth analysis of FPGA-specific limitations, we identify a set of FPGA-induced constraints that shape the algorithm’s design. These include a computational complexity limit of 30 GMACs (billion multiply-accumulate operations), INT8 quantization for weights and activations, 2D cell-based processing instead of 3D voxels, and minimal use of skip connections. To meet these constraints while maximizing performance, LiFT combines novel mechanisms with state-of-the-art techniques such as reparameterizable convolutions and fully sparse architecture. Key innovations include the Dual-bound Pillar Feature Net, which boosts performance without increasing complexity, and an efficient scheme for INT8 quantization of input features. With a computational cost of just 20.73 GMACs, LiFT stands out as one of the few algorithms targeting minimal-complexity 3D object detection. Among comparable methods, LiFT ranks first, achieving an mAP of 51.84% and an NDS of 61.01% on the challenging NuScenes validation dataset. The code will be available at https://github.com/vision-agh/lift . 1 Introduction With the rapid development of autonomous vehicle technology, one of the key challenges becomes ensuring reliable perception of the surroundings. In particular, real-time, high-precision 3D objects detection is the foundation for the safety and efficiency of autonomous systems. Among the various sensors used in autonomous vehicles, LiDAR (Light Detection and Ranging) has gained considerable popularity due to its ability to generate accurate 3D maps of the environment, allowing for precise object detection and classification. Thanks to its independence from lighting conditions and its high resolution and accuracy, the technology offers advantages over other sensors such as cameras and radars. Modern 3D object detection algorithms based on LiDAR data typically use Deep Convolutional Neural Networks (DCNNs), which offer high performance but come with significant computational and memory demands. As a result, they are usually implemented on high-performance computers with GPU (Graphics Processing Unit) cards for efficient training and inference. However, the overarching goal of these algorithms is real-time operation in autonomous vehicles or advanced driver assistance systems (ADAS). For application in an actual, mass-produced vehicle, a reliable, power-efficient and low-cost computing platform is required – for example a modern SoC FPGA. The real-time reimplementation of SoTA (State of The Art) algorithms on such platforms is a major challenge and often requires a significant redesign of the algorithm to take full advantage of the capabilities of the platform under consideration (so-called hardware aware algorithm design). In this paper, we present LiFT, a lightweight 3D detector based on LiDAR sensor data, carefully designed to run in real-time on low-power FPGA or ASIC platforms while providing high detection performance on the demanding NuScenes dataset by combining a number of novel solutions with SoTA mechanisms. The main contributions of this paper are: • a set of constraints on a 3D detector architecture in the context of implementation on an FPGA platform, • an efficient way to quantize initial features, • Dual-Bound Pillar Feature Net – a Pillar Feature Net extension to increase detection performance without added complexity, • a 3D detection algorithm adapted for implementation in FPGAs, with the best detection efficiency among comparable methods on the NuScenes dataset. The reminder of this paper is organised as follows. In Section 2 we discuss issues related to our work: most commonly used datasets, DCNN approaches to object detection in LiDAR data and FPGA/ASIC implementations of such algorithms. Next, in Section 3 we elaborate on the LiFT design, focusing on hardware induced constraints on the algorithm and novel mechanisms. The results obtained are summarised in Section 4 . The paper ends with a short summary with conclusions and discussion of possible future work. 2 Related work Datasets The most commonly used datasets for object detection based on LiDAR data are KITTI [ 6 ] (2012), NuScenes [ 3 ] (2019) and Waymo [ 14 ] (2019). The latter two sets are much larger and more challenging than KITTI. NuScenes contains 1,000 sequences that add up to 1.4 million images, 390,000 LiDAR scans and 1.4 million tagged objects. Of the 390k LiDAR scans, only 40k are labelled – 28310 are used for training, 6019 for validation, and 6008 for testing. 3D object detection on the NuScenes is evaluated using the standard mAP metric (mean Average Precision) and a metric called NDS (nuScenes detection score). It includes mAP and several error measures, e.g. orientation error or scale error. Methods There are several approaches to 3D object detection from LiDAR data, the most popular of which are point-based and cell-based methods (the later most common in SoTA solutions). In the first step, the point cloud is reduced to a regular grid of 2D or 3D cells (pillars or voxels).The cells are assigned a feature vector using the Pillar Feature Encoder (PFE) in the 2D case or the Voxel Feature Encoder (VFE) in the 3D case. Due to the rationale described in Section 3.1 , the review is limited to detectors using 2D cells. The first and the simplest solution is PointPillars [ 7 ] , with the PFE called Pillar Feature Net (PFN), consisting of a linear layer, Batch Normalization and ReLU, which are applied to each point individually. Points from each pillar are reduced to a feature vector using pointwise Max pooling . Another 2D detector is CenterPoint-Pillar [ 15 ] , which uses PointPillars as a backbone and a novel CenterHead. In the one-stage version, it predicts a heatmap with the centres of objects and regression maps specifying their location, shape and orientation. Both PointPillars and CenterPoint-Pillar are based on regular convolutional neural networks. However, usually 90% of the input pillars are empty and are implicitly assigned a zeroed feature vector, what can be exploited using sparse convolutional neural networks to significantly reduce computations. In addition to standard sparse layers, which function similarly to regular convolutions by gradually expanding the area of non-zero, so-called active, pixels, there are also submanifold versions of these layers, which maintain the active pixels structure intact. Other popular 2D detectors, representing the SoTA on datasets such as NuScenes and Waymo, include PillarNet [ 12 ] and its derivatives VoxelNeXt-2D [ 4 ] and FastPillars [ 16 ] . VoxelNeXt-2D is particularly notable as it incorporates only sparse convolutions, even in its head. FPGA/ASIC-based implementations To date, there have been few implementations of LiDAR sensor-based object detectors on FPGA platforms, all of which are based on the original or modified version of PointPillars [ 1 , 13 , 10 , 8 , 2 ] . Most of them operate in real-time, defined as processing 10 point clouds per second, which corresponds to the typical LiDAR rotation frequency. All of them are evaluated on the KITTI dataset. However, methods capable of performing well on more challenging datasets, such as the NuScenes are currently missing. The authors of SPADE [ 9 ] and SPADE+ [ 11 ] undertook the task of developing a 2D sparse convolution accelerator targeting an ASIC, which was evaluated in simulation. In addition, they implemented PointPillars, CenterPoint-Pillar and PillarNet in several different sparse versions and tested the accelerator performance. PointPillars was evaluated on

Read the complete text