3D perception algorithm of unstructured environment based on point cloud enhanced pixel fusion Authors: Guo Chen, Liming Wan, Jingjing He, Shuhao Jiang, Lin Song Published: 2026 ABSTRACT Based on the complementary and enhanced fusion of 3D point clouds and 2D RGB images, this paper designs an end-to-end learning framework—Point Cloud Enhanced Depth Pixel Fusion Network (PEPF-Net), aimed at enabling robots to achieve accurate 3D perception of unstructured environments. In the process, we address four key problems in 3D perception tasks: enhancing RGB representation using the reflection intensity and depth information of point clouds to generate Depth-RGB Pixel (D-Pixel); proposing Point-by-Point Vector Attention (PVA-Net) to model the vector relationships of point clouds, to obtain deep-level point cloud features, and to achieve direct and effective fusion of heterogeneous data; designing a Layered-Transformer (L-TsfmNet) feature extractor to hierarchically extract D-Pixel features; proposing Variable Window Self-attention (VS-a) to focus on the relationships between local “window tokens” and avoid the complexity of global computation. Extensive experiments on the KITTI dataset demonstrate that PEPF-Net outperforms the currently common advanced environmental 3D perception algorithms. Introduction 3D perception of unstructured environmentally manifests in the ability to quickly detect and identify environmental targets in three-dimensional space, and to accurately predict their position, size, and direction of movement1. For intelligent robots, unstructured environments are ambiguous and extensive, referring to unknown dynamic environments—including urban roads, residential roads, and other unstructured scenarios as well as specific work scenes. At present, research of 3D perception based on multiple sensors has made splendid achievements, among which LiDAR and cameras have been widely favored due to their inherent advantages. This article based on “target point enhanced fusion target pixel”, the end-to-end learning framework PEPF-Net is designed, which enables robots to achieve efficient 3D perception of unstructured environments. That is while recognizing environmental targets with high precision, it accurately acquires their 3D position and geometric morphology. Although the 3D point clouds output by LiDAR are sparse, disordered, and non-uniform distributed, they contain depth information and reflection intensity information required for 3D Perception2–4. RGB images output by cameras as rich in dense texture and color information, which can depict environmental targets and scenes in detail. However, perspective projection leads to the loss of depth information. When facing scenes such as target deformation, scale change, overlapping occlusion, and underexposure, they fail to ensure the accuracy of depth estimation, making it difficult to achieve high-precision 3D positioning5–7. Enhancing RGB semantic information using the depth information and reflection intensity of point clouds is crucial for the precise positioning of the 3D environmental perception (with Pixel as the main input data). Since point clouds and RGB are heterogeneous data with significantly different data characteristics, fusing their intrinsic information is extremely challenging! Secondly, point cloud data have characteristics of sparsity, disorder, and non-uniform distribution. Improper data fusion methods may introduce harmful (interfering) information into 3D object detection tasks. In the task of 3D perception of unstructured environments, generating 3D bounding boxes and class labels for targets of interest is the key for the perception system to perceive environmental targets. Existing multi-sensor 3D perception research commonly fuses point clouds and RGB images through image annotations (annotated 2D bounding boxes), which belongs to data-level fusion. Fusion methods can be roughly divided into two categories: One is cascade fusion algorithms that use different sensors at different stages8–10, this type of method does not fully utilize the complementarity between multiple sensors, and its fusion results are limited by the performance of different sensors; the other is joint inference fusion algorithm that take multi-sensor as input11,12, this type of method cannot avoid information loss in the process of generating Bird’s Eye View (BEV) through perspective projection or voxelization, and the fusion process only establishes a rough correspondence between point cloud voxel features and image semantic features. These two types of fusion algorithms usually process different view features in a proportional and rigid manner, but the outputs of point cloud and RGB are parallel—their weights should be adaptive rather than fixed when facing unstructured environments. For example, when the environment targets are small size or have similar shapes, 3D detection should depend slightly on RGB texture information due to the sparsity of point clouds. Conversely, when the point cloud is sufficient to characterize the target, additional RGB data will interfere with 3D detection. To address the above problems, based on the BiProDet13 network framework, we integrated the D-Pixel generation module, the PEP-F module, PVA-Net, and L-TsfmNet to propose a novel 3D detection framework — PEPF-Net (algorithm). It aims to enable robots to accurately perform 3D detection of environmental targets in unstructured environments, i.e., while ensuring improved target recognition accuracy, it achieves accurate estimation of the 3D positions and geometric morphologies of environmental targets. This algorithm realizes high-precision 3D detection of multi-targets on the KITTI benchmark dataset14 in complex unstructured environments, delivering advanced perceptual performance. Among them, PEP-F belongs to feature-level fusion, which establishes the corresponding relationship between point clouds and D-Pixels in a point-by-point manner, achieving adaptive point features enhancing pixel semantics and suppressing interfering pixels. Compared with previous fusion methods, PEP-F has the following advantages: ①Avoids the complex process of BEV generation—by simply combining our proposed Point-by-Point Vector Attention Network (PVA-Net) with Sets of Abstraction (SA), multi-scale point cloud features can be obtained, realizing fine-grained point-by-point correspondence between point clouds and D-Pixels. ②Avoids the loss of point cloud reflection intensity and geometric structure information. ③Effectively suppresses interfering information in D-Pixel images that disturbs 3D perception. ④Eliminates the need for 2D bounding box annotation (image annotation). Experiments demonstrate that this algorithm successfully achieves the direct fusion of heterogeneous data; effectively avoided (Caused by data conversion) data loss; efficiently adapts to changes in environmental targets; and effectively addresses the problems of insufficient unidirectional connection fusion, full connection data redundancy and excessive computational complexity. The main contributions of this paper are as follows: (1) Propose source data augmentation fusion method capable of generating D-Pixel feature maps. We encode the reflectance intensity and depth information of the point cloud into the three channels of RGB. Specifically, the reflectance intensity is utilized to enrich the spectral information of the RGB, while the depth information is employed to enhance its geometric information. This approach successfully improves algorithm performance by leveraging raw data fusion. (2) Propose PEP-F fusion subnet. It is based on the step-by-step enhanced fusion manner1, using multi-scale point cloud feature points to enhance D-Pixel feature in a point-by-point a manner. Without image annotation, it avoids data loss. Its step-by-step enhanced fusion approach addresses the problem of insufficient unidirectional connection fusion, full connection data redundancy, and excessive computational complexity enabling the direct fusion of heterogeneous data. (3) Propose PVA-Net as the feature point selector for SA. It models the relationship between point elements in the multi-scale abstract sets, adjusts element weights, and plays a role in highlighting key feature points, suppressing useless interference, and self-adapting to changes in environmental targets, which renders the algorithm more robust. This attention mechanism adopts a lightweight design with negligible computational overhead. (4) Propose the L-TsfmNet feature extractor, which extracts multi-layer and multi-scale features of D-Pixel. Through the step-by-step enhanced fusion of multi-scale point features, the semantic and spatial representations of D-Pixel are effectively enriched, significantly enhancing the 3D perception accuracy of the algorithm. Related work In 3D perception tasks in unstructured environment, cameras and LiDAR are the mainstream sensors commonly used. At present, 3D perception algorithm based on camera and LiDAR have made remarkable achievements. 3D perception based on camera data In past research on environmental perception, 3D perception research based on camera data has been highly active. Among them, most studies based on monocular camera data adopt the idea of first generating 2D bounding boxes via CNN object detectors and then inferring 3D bounding boxes of environmental target based on semantic information, context and shape information, such as References5,15–19 . Most studies based on stereo camera data adopt the research idea of “using projected geometric constraints to directly estimate the 3D boundary box, position and direction of environmental targets” such as References20–22 et al. In short, due to the lack of depth information in camera data and its vulnerability to factors such as illumination and occlusion (see Fig. 1), the perception accuracy of this type of 3D perception algorithm is not high (see Fig. 2(A)). Fig. 1Strong interference based (environment target) camera (RGB) data. Strong interference based (environment target) camera (RGB) data. 3D perception based on LiDAR data The physical characteristics of the cameras determine that the 3D perception accuracy based on camera data is not high, which has prompted many scholars to dedicate themselves to the research of 3D perception based on LiDAR data, such as References4,23,24 . Among them, the Voxel-Net al.gorithm2 effectively solves the problem of information loss of the traditional manual raster method25 by combining point cloud feature extraction and boundary box prediction; the SECOND algorithm26 uses sparse convolution operation to significantly improve the speed of network training and prediction and realize real-time performance (0.05s); the Point-net algorithm27 pioneeringly takes the original Point cloud as the input of CNNs to construct a unified neural network model, successfully avoiding the complexity of gridding and irregular combinations; the PointRCNN algorithm28 innovatively utilizes a two-stage detection group (weak detection subnet and strong detection subnet) to segment point clouds into foreground and background, generate a small number of high-quality 3D candidate boxes, which are then refined by a refinement network, effectively avoiding the high computational overhead of searching for a large number of 3D anchor boxes. However, since LiDAR data is usually very sparse, the positioning accuracy of this type of 3D perception algorithm remains limited (see Fig. 2B). 3D perception based on multi-sensor data fusion At present, research on 3D perception based on multi-sensor data fusion has received extensive attention, and among these studies, sensor selection mainly uses camera and LiDAR as the mainstream. In most studies, the 3D perception accuracy based on multi-sensor data is significantly superior to that based on single-sensor data (see Fig. 2C). According to the perception method, the existing research achievements can be roughly summarized into the following two categories. 3D perception algorithm based on 2D data drive For example, the F-PointNet algorithm first generates 2D candidate boxes from RGB images, then performs 3D bounding box regression based on point clouds. This algorithm can still estimate 3D boundary boxes even in scenarios with strong occlusion or sparse points8. This type of “cascade structure” algorithm requires 2D annotation, and the 3D sensing accuracy is limited by the performance of the anterior layer 2D detector. Its phased prediction prevents the network from conducting joint inference on multi-modal data, resulting in insufficient data utilization. 3D perception algorithm based on multi-view fusion For instance: the MV3D algorithm first projects 3D candidate boxes generated based on the aerial view onto feature maps of three perspectives, generates corresponding regional features respectively, then deeply fuses regional features to perform bounding box regression and classification29; The AVOD algorithm first projects prior anchor boxes onto the (BEV and RGB) feature maps to extract regional features, then fuses regional features to generate 3D candidate boxes; the 3D candidate boxes are the reprojected, resampled and re-fused (repeating the anchor boxes processing process); finally, bounding box regression and category classification are executed30; the Con-Fuse algorithm proposes a new continuous fusion layer to achieve voxel alignment between BEV and RGB feature maps, achieving more accurate 3D detection results based on “RGB + BEV” fusion data12. This type of algorithms realizes 3D perception of environmental targets by a joint inference approach, but there is still room for improvement in perception accuracy. Unlike pure image processing algorithms31–33 and image quality assessment algorithms34–37, our proposed PEPF-Net algorithm framework directly performs fusion operation on a raw 3D point cloud and RGB data (direct fusion of heterogeneous data, establishing elaborate point-by-point correspondence between features) with step-by-step enhancement, featuring simplicity, efficiency, and no data loss. Fig. 23D object detection based on different data. 3D object detection based on different data. Description of the algorithm Effectively realizing complementary fusion of multi-sensor heterogeneous data is crucial for robot 3D perception of unstructured environment. This paper proposed a 3D perception algorithm (framework) named PEPF-Net, which can improve robots’ 3D perception performance of unstructured environment from three aspects. PEPF-Net is composed of a raw data enhancement fusion subnet, a D-Pixel feature generator (main data stream), a point cloud target feature generator (enhanced data stream), a PEP-F enhanced fusion module, and a classification and 3D position estimation subnet. It is an end-to-end learnable multi-object 3D perception network for unstructured environments. The raw data enhancement fusion subnet encodes the 2D point cloud feature map (generated by 2D mapping of point clouds) into the RGB image, enhancing its representation and obtaining the D-Pixel feature map. The PEP-F enhanced fusion module uses multi-scale point cloud features to progressively enhance the multi-scale features of D-Pixel. In addition, we integrate PVA-Net into the point cloud feature extraction process to ensure that the feature data and predicted 3D bounding boxes are not lost. In the following sections: Section 3.1 detailed the architecture, constituent parts, and functions of the PEPF-Net algorithm; Section 3.2 elaborates on PVA-Net; Section 3.3 described the L-TsfmNet feature extractor; Section 3.4 described Variable Window Self-attention (VS-a); Section 3.5 elaborated on multitask loss function. PEPF-Net algorithm framework PEPF-Net takes D-Pixel image (mainstream data) and 3D point cloud (enhanced data) as inputs and embeds the L-TsfmNet feature extractor and PVA-Net. These components are used to reinforce the discriminative information and location information during the feature extraction process and implicitly leverage the depth information and reflection intensity of the point cloud to enhance the skeleton pixels of the image targets. It combines traditional algorithms with deep neural networks, adopting a joint knowledge-and-data-driven approach to construct an end-to-end framework for performing 3D perception (recognition and positioning) tasks. It consists of a source data enhancement fusion module, a D-Pixel image feature extraction module, a point cloud feature extraction module, a PEP-F enhancement fusion module, and a classification and 3D position estimation subnet, as shown in Fig. 3. H and W represent the height and width of the image, respectively; N denotes the number of points in the point cloud. Fig. 3PEPF-Net network framework. PEPF-Net network framework. The source data enhancement fusion Source data enhancement fusion refers to the data-level enhancement fusion that takes the original data output by the sensor (data modal transformation not performed) as input. In PEPF-Net, source data enhancement fusion achieves the enhancement of discriminative representation of environmental targets in RGB data, generating the D-Pixel feature map containing point cloud features (reflection intensity and position information). Its enhancement fusion process is shown in Fig. 4. Fig. 4Source data enhanced fusion model. Source data enhanced fusion model. Based on the selectively enhanced fusion method, first, 2D mapping on the 3D point cloud is performed, resulting in a 2D point cloud feature map that retains the reflection intensity of the point cloud and the depth information of the target. Then, it is encoded and fused into the three channels of RGB data to enhance the spectral and skeleton information representation of each channel, thereby generate D-Pixel feature data. 2D mapping refers to converting the point cloud from the LiDAR coordinate system to the camera coordinate system based on the external parameter matrix (positional relationship) of LiDAR and camera, achievving the unification of the coordinate systems between the 2D point cloud feature map and the RGB image. Encoding refers to projecting the 2D point cloud feature map in the camera coordinate system onto the three channels of the RGB image according to the internal parameter matrix. The enhanced fusion process can be summarized as Eq. (1) (2):1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_{{lidar}}^{{camera}} \left( {\begin{array}{*{20}c} x \\ y \\ z \\ \end{array} } \right) \times P_{{rect}} = \left( {\begin{array}{*{20}c} u \\ v \\ \end{array} } \right)$$\end{document}2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{T}_{lidar}^{camera}=\left(\begin{array}{c}{R}_{lidar}^{camera}\\\:0\end{array}\right.\left.\begin{array}{c}{t}_{lidar}^{camera}\\\:1\end{array}\right)$$\end{document} In Eq. (1),\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{T}_{lidar}^{camera}$$\end{document}is the external parameter matrix for transforming the point cloud target point from the LiDAR coordinate system to the camera coordinate system; \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\left(x,y,z\right)$$\end{document} are the coordinate of the point cloud target point in the LiDAR coordinate system; \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{P}_{rect}$$\end{document}is the projection matrix that transforms the 3D point cloud convert to ; \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\left(u,v\right)$$\end{document} is the coordinate of the target pixel in the camera coordinate system. The external parameter matrix \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{T}_{lidar}^{camera}$$\end{document} consists of a rotation matrix \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{R}_{lidar}^{camera}$$\end{document} and a translation matrix \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{t}_{lidar}^{camera}$$\end{document}. Point cloud feature extraction Point cloud feature extraction takes the raw point cloud as input, including three PVA-Nets, three SA38, and three feature propagation (FP)38 layers. The outputs of SA and FP are expressed as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:S{A}_{i}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:F{P}_{i}\left(i=\mathrm{1,2},3\right)$$\end{document} respectively. We add PVA-Net before each Set of Abstraction layer to highlight the target feature points of the SA layer, enhancing the discriminability of target features. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:S{A}_{i}$$\end{document} refers to the target feature point set and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:F{P}_{i}$$\end{document} refers to the target feature set, used for correspondingly enhancing D-Pixel features of different scales. At the back of the third layer of FP, all the target point features contained in \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:F{P}_{i(i=1,2,3)}$$\end{document} are fused to obtain the multi-scale target point feature \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:F{P}_{U}$$\end{document}. Subsequently, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:F{P}_{U}$$\end{document} is combined with the regional candidate boxes generated by \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:S{A}_{3}$$\end{document} through RPN (Region Proposal Network). After Rol (Region of Interest) grid pooling, is further enhanced and fused with the D-Pixel feature map to generate more accurate 3D candidate boxes. D-Pixel feature extraction D-Pixel is a feature pixel obtained by encoding and fusing the reflection intensity and position information of raw point clouds into RGB images. D-Pixel feature extraction takes D-Pixel data as input, and semantic feature information is extracted through convolution operation, its outputs multi-scale feature maps render target features more discriminative. The D-Pixel image feature extraction network consists of a scattered partition module, three linear embedding modules, three L-TsfmNet, and two sets of convolutional operation modules. Among them, each convolution operation module contains one 3 × 3 convolution layer, one batch normalized layers39 and a ReLU activation function. We use scattered partition module to segment D-Pixel hierarchically by channel (D/R/G/B), generating pixel matrix (feature block) as inputs to the first set of convolutional networks. The purpose of adding the linear embedding module is to reduce network complexity through hierarchical computational self-concern40. The goal of integrating L-TsfmNet is to hierarchically extract skeleton points and discriminative information from D-Pixel. The second set of convolution restores the feature map to the input size by up-sampling, obtaining a compact and differentiae feature representation (i.e., multi-scale features containing rich semantic information and different receptive fields) for foreground point segmentation and 3D candidate box generation. PEP-F fusion Enhancing non-target pixels in the image will complicate feature extraction and lead to redundancy; therefore, the pixel density enhancement of D-Pixel should be targeted. The PEP-F fusion module consists of two parts: point-by-point image feature generation1 and target pixel enhancement, as shown in Fig. 5. Fig. 5PEP-F network framework. PEP-F network framework. Point-by-point image feature generation. That is, a point-by-point correspondence is established to form a point-by-point semantic feature representation and generate point-by-point image features. Point-by-point correspondence relation established1: That is, based on the mapping matrix and grid generator, the target point set is projected onto the D-Pixel feature maps to establish a point-by-point correspondence. Specific process: First, the point cloud and mapping matrix (M) are input into the grid generator to obtain the target point set (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:S{A}_{i}$$\end{document}); then the target point set is mapped to the D-Pixel characteristic picture (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:I{F}_{i}$$\end{document}) at different resolutions, and the point-by-point correspondence between the two between is obtained. The point cloud target point \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{P}_{i}\left(x,y,z\right)$$\end{document} is projected to the corresponding position in the D-Pixel feature map to generate the enhanced target point \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{p}_{i}^{{\prime\:}}\left({x}^{{\prime\:}},{y}^{{\prime\:}}\right)$$\end{document}, and its projection process can be expressed by mathematical model as Eq. (3)1:3\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{P}_{i}\times\:M={p}_{i}^{{\prime\:}}$$\end{document} In Eq. (3), \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{p}_{i}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{p}_{i}^{{\prime\:}}$$\end{document} in the homogeneous coordinates are respectively 3-dimensional and 4-dimensional vectors, and the dimension of M is 3 × 4. Point-by-point semantic feature representation: That is, based on the point-by-point correspondence relationship, the process of generating Point-by-point image features (PIF) using an image sampler based on the point-by-point correspondence relationship. Specific process: The enhanced target point set \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{p}_{i}^{{\prime\:}}$$\end{document} and multi-scale feature maps \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:I{F}_{i}$$\end{document} are simultaneously input into the image sampler to generate point-wise image features (PIF). (The enhanced target points set (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{p}_{i}^{{\prime\:}}$$\end{document}) is generated by mapping the multi-scale target points set (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:S{A}_{i}$$\end{document}) to the multi-scale feature points set (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:F{P}_{i}$$\end{document})). In order avoid \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{p}_{i}^{{\prime\:}}$$\end{document} (possibly) falling into adjacent pixels, we use bilinear interpolation to acquire continuous coordinate image features, which can be represented by the mathematical model as Eq. (4):4\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$PIF^{{(P_{i} )}} = \kappa \left( {F^{{\left( {N\left( {P_{i}^{'} } \right)} \right)}} } \right)~~~~~$$\end{document} In Eq. (4), \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:PI{F}^{\left({p}_{i}\right)}\:$$\end{document}is the point-by-point image feature corresponding to point set \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{p}_{i}$$\end{document}; \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\kappa\:\:$$\end{document}is a bilinear interpolation function; \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F^{{(N(P_{i} ^{'} ))}}$$\end{document} is the adjacent pixel image feature of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{p}_{i}^{{\prime\:}}$$\end{document}. Target pixel enhancement. That is to enhance (fuse) D-Pixel feature pixels using point-by-point image features of point clouds. The specific process is as follows: First, the multi-layer image features of D-Pixel and point-by-point image features of point cloud are summed to form a compact feature representation and mapped to the same channel. Then, the hyperbolic tangent function is used to compress it into a single channel feature graph. Subsequently, the sigmoid activation function is applied to normalize the mapping weightings to the range [0,1], yielding the weight map W. Finally, the image semantic featured \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:I{F}_{i}\:$$\end{document}and point by-point image feature (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:PI{F}_{i}$$\end{document}) are combined in a concatenated manner to obtain the enhanced target feature pixel set \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:P{S}_{ETF}$$\end{document}. The applicable mathematical models are expressed respectively as Eqs. (5) and (6):5\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:W=\sigma\:(w{tanh}(u{F}_{I{F}_{i}}+v{F}_{PIF}))$$\end{document}6\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$PS_{{ETF}} = IF_{i} w\parallel{PIF_{i}}$$\end{document} In equation, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\sigma\:$$\end{document} is the sigmoid activation function, and w,u,v are all is in learning weightings matrix in PEP-F module. Classification and 3D position estimation The classification and 3D position estimation network is mainly used to extract compact global feature descriptions and consists of three SA layers; both subnetworks each contain a 1 × 1 of cascade convolutional layer for classification and regression, respectively. First, a semantic segmentation branch is designed to provide accurate semantic labels for the feature pixels in the candidate box based on the candidate box features; then, a region of Interest (ROI) feature extractor is designed to capture the detailed geometric structure within the region with the help of attention mechanisms. Meanwhile, Non-Maximum Suppression (NMS) is applied to preserve high-quality 3D candidate boxes, which are then input into the refinement network. Finally, the extracted global feature descriptions are fed into two specific branches to perform category classification41 and 3D bounding box optimization, outputting high quality 3D inspection results. Point by point vector attention In past 3D inspection studies, point cloud feature extraction is commonly implemented using raster method or graph annotations. This process is complicated with serious data loss and an unsatisfactory result. In order overcome the problem of data loss in the process of point cloud feature extraction, we build on the Point-Net algorithm and directly feed raw point clouds into CNNs to construct a unified neural network model. To improve the robustness of the algorithm and enable it to flexibly adapt to environmental changes, we introduce an attention mechanism to weigh important feature information, to enhance network performance and strengthen the network’s ability to generate candidate boxes. The point cloud feature extraction in PEPF-Net is shown in Fig. 6. Fig. 6Point cloud feature extraction network. Point cloud feature extraction network. A Point cloud consists of a series of points, all each containing location and property information. Points cloud feature extraction needs to consider the relationship between points and points, because they do not have an unequivocal pixel arrangement structure like image data. The application of attention mechanism to point cloud feature extraction can effectively capture the correlation between points and improve the characterization ability of point cloud. The classic attention module SE-Net42 uses fully connected layer (FC) to capture nonlinear cross-channel interactive information capture; but its efficiency in capturing inter-channel dependencies is low, and it increases the computation overhead and network complexity43,44. While guaranteeing attention superior performance, to balance the network complexity, we based on the inherent characteristics of the point cloud propose PVT-Net, use for extract point cloud features by combining 2D convolutions. The PVA-Net module consists of three components: attention vector generation, global information tensor generation, and output processing, as shown in Fig. 7. Fig. 7Point by point vector attention network framework. Point by point vector attention network framework. In PVA-Net, we introduce two 1 × 1 convolution layers (query convolution and keyed convolution) for query calculation and generating learnable weight matrix; integrate a Soft-max layer, which is used to normalize and calculate attention coefficients; employ two FC layers and one Soft-max layer to further acquire global information of point features; and introduce a learnable parameter Alpha to balance the weight ratio between the attention vector in the output data and the input. Attention vector generation. First, the query convolution (1D convolution) is used to query and compute the feature points of the input point cloud. At the same time, based on different weights matrices, the keyed convolution layer (1D convolution) is used to perform convolution transformation for each feature point, and a learnable weight matrix is generated. In this process, we convert the query and keyed convolution layer into the format “batch size, −1, num points” to calculate attention scores, and set the number of output channels to a quarter of the number of input channels. Second, we use Torch bmm function to normalized the query and keyed convolution, got the “energy” tensor, and multiply the “energy” tensor with the Soft-max function to generate the attention vector. Global information tensor generation. First, the input tensor is processed through adaptive average pooling. Then, two FC layers and one Soft-max layer are used to obtain a global attention vector that maintains the same format as the input tensor. Finally, each element of the input tensor is multiplied by the corresponding global attention vector to generate a tensor containing global information. Processing of PVA-Net Output. The output of PVA-Net is obtained by multiplying the attention vector with a tensor containing global information. To prevent PVA-Net from learning degradation, is necessary to incorporate the input features into the PVA-Net output data. Meanwhile, a learnable parameter α, is introduced to scale the PVA-Net output, aiming to balance the weight ratio between the PVA-Net output and the input data. Alpha (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\alpha\:$$\end{document}) is a learnable parameter equipped with an adaptive optimization and updating mechanism, which eliminates the need for specific human intervention during the training process. Layered-transformers network Transformer networks stem from sequence modeling and transduction tasks, renowned for focusing on dependencies in data, they are currently the most popular architecture in natural language processing (NLP)45. Their outstanding performance in the language domain has attracted many scholars continuously explore applying them to the field of computer vision. Computer vision modeling is mainly dominated by Convolutional Neural Networks (CNN) for domination. At present Transformer-based computer vision research has achieved some ideal results, especially in image classification46 and joint visual language modeling47, with particularly outstanding performance. Previous vision Transformer (ViT) generally output a single low-resolution feature map and rely on self-attention global computation, where the computational complexity is the quadratic of the image size46. Existing research indicates that transferring the high performance of Transformer networks from the language domain to the visual domain can be achieved by explaining the differences between the two-modal data. In the existing research results on object detection, “scale” differences have received widespread attention48–50. The differences in “scale” between language and vision are manifested as follows: in the language domain, only text symbols are used as the basic elements for linguistic transformation processing, while visual elements may undergo significant changes in scale. Inspired by the Swin-Transformer algorithm51, the paper proposes L-TsfmNet based on differences in image pixel resolution. The network effectively realizes hierarchical (D/R/G/B) segmentation and feature extraction of D-Pixel, significantly improving the algorithm’s modeling capability and detection accuracy, as shown in Fig. 8. Fig. 8Layered-transformer network. Layered-transformer network. L-TsfmNet process of extracting image features consists of four stages. The first stage, first, the scattered partition module is used to hierarchically segment the input data by channels (D/R/G/B), single channel non-overlapping data block with the size of H/4×W/4 × 64 is obtained; each data block is then assigned a tag, its feature is set to original pixel RGB value the connect series. In the experiment, we set the size of the data block to 4 × 4, so the feature size of each data block is 4 × 4 × 4 = 64. The second stage, First, the linear embedding layer is used to concatenate the feature blocks and project them to an arbitrary dimension (denoted as C) to reduce the complexity of self-attention computational. Then, the Swin-Transformer layer is adopted for feature transformation, yielding data blocks of size H/4×W/4×C. Finally, the variable window self-attention module is used to realize window boundary crossing and connection. The third stage, first, the feature block merge layer connects adjacent feature blocks (2 × 2 per group), and a linear layer is applied to the concatenated features to reduce the number of data blocks by 4 times (2× resolution down-sampling). Then, the output dimension is set to 2 C, and feature transformation is carried out in combination with Swin-Transformer layer to obtain H/8×W/8 × 2 C data blocks to achieve hierarchical representation. Finally, the variable window self-attention module is used to realize window boundary crossing and connection. The fourth stage, first, the feature extraction process of the third stage is repeated. Then, the dimension reduction layer is used to restore the output data to the same size as the input data. Variable window self-attention The improved Transformer network used for image classification are almost all continue to use of standard Transformer network of the Multi-head Self-attention (MS-a)45 to calculate the relationship between global window labels (as shown in Fig. 9a). Its computational complexity is the square of the image size, making it unsuitable for visual tasks that require many labels for dense prediction or representing high-resolution images46. To avoid the complexity of global computation in multi-head Self-attention, we proposed Variable window Self-attention (VS-a), which is used to calculate the relationship between local “window labels” (as shown in Fig. 9b). Experimental results show that VS-a realizes window boundary crossing and connection improve the autonomous modeling ability of the algorithm. The comparison of computational complexity between MS-a and VS-a is shown in Fig. 9. VS-a hierarchically segments the input image data in a non-overlapping manner and randomly generates multiple variable local windows, effectively reducing computational complexity. Given that each window contains ‘M×M’ data blocks, the global computational complexity of MS-a can be expressed as Eq. (7), and the local computational complexity of VS-a can be expressed as Eq. (8).7\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\varOmega\:(MS-A)=4hw{C}^{2}+2(hw{)}^{2}C$$\end{document}8\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\varOmega\:(VS-A)=4hw{C}^{2}+2{M}^{2}hwC$$\end{document} In Eqs. (7) and (8), C stands for dimension. In Eq. (7) the data block label hw is quadratic element input, when facing large model computation, hardware usually can’t be afforded. In Eq. (8), the value of M is random (M = 1, 2, 3…). When the randomly generated variable M window is 7 (M = 7), the data block label hw is input linearly. Therefore, simply setting M to a fixed value of 7 (defaulting to 7) allows for scalability to accommodate large model computations. Fig. 9Comparison of MS-a and VS-a calculations. Comparison of MS-a and VS-a calculations. Multitask loss function In PEPF-Net, we utilize a multitask loss function to optimize tasks such as point cloud target feature extraction, D-Pixel feature extraction, classification, and 3D position estimation, expressed as (Eq. 9):9\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{L}_{multi}=({L}_{rpn}+{L}_{rcnn}+{L}_{ace})$$\end{document} In the equation: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{L}_{rpn}$$\end{document} is the target feature extraction loss function, and its mathematical model is (Eq. 10)1; \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{L}_{rcnn}$$\end{document} is the target feature training loss function, which is a similarity to the optimization method of the former; \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{L}_{ace}$$\end{document}(attention consistency enforcement loss function) is an NMS loss function, used to retain high-quality 3D bounding box, its mathematical model is (Eq. 13)1. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{L}_{rpn}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{L}_{rcnn}$$\end{document} both stem from the baseline framework BiProDet, as common loss functions their effectiveness and superiority have been well-established in extensive literature (such as reference13 etc.). The \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{L}_{ace}$$\end{document} is derived from the REPF-Net algorithm and is specifically designed for the NMS process. NMS refers to the process of retaining bounding boxes based on the maximum value of either the classification confidence or the localization confidence. Due to the inverse trend between classification confidence and localization confidence, the commonly used Consistency Execution loss (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{L}_{ce}$$\end{document}) tends to filter out some high-fidelity data during the NMS process, which leads to suboptimal performance. However, the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{L}_{ace}$$\end{document} can fit those with high localization confidence and those with high classification confidence level two categories of 3D candidate boxes, to generate where both localization and classification confidences are all maximum values 3D bounding boxes. References1 have already demonstrated the effectiveness and superiority of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{L}_{ace}$$\end{document} through ablation experiment. In the classification and 3D position estimation network, we adopt the focus loss function52 as the classification loss function to balance positive and negative samples. 3D position estimation requires network regression boundary box center point (x,y,z), size\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\:(l,h,w)$$\end{document} and direction \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\theta\:$$\end{document}. We use smoothing loss function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\left(L1\right)$$\end{document}53 to optimize the size (l,h,w) of the bounding box and calculate the offset between the center point and the ground truth. The bin-based regression loss function (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{L}_{reg}$$\end{document})8,28 to constrain the x axis, z axis and direction \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\theta\:$$\end{document}. Bin-based regression first divides the adjacent area of each foreground point into several bins, then predicts which bin (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{b}_{u}$$\end{document}) the center point will fall into, and finally regresses the intra-bin residual offset (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{r}_{u}$$\end{document}), its mathematical model is Eq. (12)10\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{L}_{rpn}={L}_{cls}+{L}_{reg}+\lambda\:{L}_{cf}$$\end{document}11\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{L}_{cls}=-\alpha\:(1-{c}_{t}{)}^{\gamma\:}{log}{c}_{t}$$\end{document}12\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{L}_{reg}={\sum\:}_{u\in\:x,z,\theta\:}E({b}_{u},{\widehat{b}}_{u})+{\sum\:}_{u\in\:x,y,z,h,w,l,\theta\:}S({r}_{u},{\widehat{r}}_{u})$$\end{document}13\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{{ace}} = - \log (NMS \times D_{i} ((\left| {C_{{\max }}^{c} } \right|,\left| {P_{{ - \max }}^{c} } \right|) \cdot (\left| {C_{{ - \max }}^{c} } \right|,\left| {P_{{\max }}^{c} } \right|)))$$\end{document} In Eq. (11), \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{c}_{t}$$\end{document} represents the probability that the target point belongs to ground truth data. In Eq. (12), E is the cross-entropy loss function, S is the smoothness loss function L1, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\widehat{b}}_{u}$$\end{document} represents the ground truth data of bin, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\widehat{r}}_{u}$$\end{document} represents the residual error offset quantity of bin. Experiments KITTI is the dataset based on an outdoors unstructured environment, its environmental targets exhibit randomness and complexity. Our research objective is to enhance the 3D perception capabilities the robot in unstructured environments. Therefore, we chose KITTI as the validation dataset. In the following sections: Section. 4.1 briefly introduces the KITTI dataset; Section. 4.2 details the experimental process and parameters; Section. 4.3 comprehensively analyzes the effectiveness of the paper’s innovative components; Section. 4.4 presents the experimental results based on the KITTI dataset and a comparison with advanced algorithms; Section. 4.5 presents the feasibility experimental results based on sample data. KITTI data set and evaluation index KITTI Dataset. KITTI Dataset is a standard benchmark environment awareness dataset1 constructed based on autonomous driving perception system, consisting of 3D point clouds and 2D RGB images. Its dataset splitting protocol is consistent with that in References1,8,13, including training frames 3,712, test frames 7,518 and verification frames 3,769. In the experiment, based on the size, occlusion, and truncation of environmental targets, the test dataset and validation dataset are respectively divided into three subsets of different difficulty levels (Easy/Moderate/Hard). Evaluation Index. Currently, the algorithm experiments based on KITTI dataset all adopt the new evaluation protocol54, it used 40 recall locations, more equitable than an evaluation protocol that uses 11 recall locations. In the experiment, in accordance with the official new evaluation protocol of the KITTI dataset, we use mean average precision mAP as the metric to verify and compare the proposed method with common advanced methods. Experimental process Network Settings. PEPF-Net takes D-Pixel images (mainstream data) and 3D point clouds (enhanced data) as inputs. Following the practice of the Point-RCNN algorithm28, the resolution of the D-Pixel image data stream is set to 1280 × 384, and the 3D point cloud data stream is confided to extract 16,384 points from the original point cloud. Based on outdoor unstructured environments, the perception range of the LiDAR is set as follows: the distance along the x (right) axis in the camera coordinates is [−40,40] meters; the distance of y (bottom) axis is1,3 meters; the distance of z (front) axis is [0,70.4] meters; orientation of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\theta\:$$\end{document} is in the range of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{{[ - \pi ,\pi ]}}$$\end{document}.The performance test sets the IoU thresholds of cars/pedestrians/bicycles to 0.7/0.5/0.5/respectively. Target Feature Extraction. Point cloud target feature extraction process: we use three PVA-Nets combined with three abstraction set layers (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:S{A}_{i}$$\end{document}) to sub-sample the input point clouds and three feature propagation layers (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:F{P}_{i}$$\end{document}) to restore the size of the point cloud, in order to enhance the multi-scale D-Pixel. D-Pixel target feature extraction process: we used three convolutional blocks with steps of 4/8/16/respectively, combined with three linear embedding modules and three L−TsfmNet, to up-sample the input images and generate feature maps of different scales. Three deconvolutional layers with strides of 16/8/4 are used in combination with down sampling modules to restore the resolution and size of the feature maps. In the NMS process: we used the attention consistency execution loss (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{L}_{ace}$$\end{document}) to filter out the redundant box with an NMS threshold of 0.8 from the first 8000 3D box output by the FPU, and the remaining 128 candidate box hand over the classification and 3D position estimation network. Data Enhancement. Data enhancement needs to balance applicability and universality. Based on the outdoor unstructured environment, to prevent data enhancement generate over-fitting, the enhancement mechanisms used in the PEPF-Net framework include three common strategies: rotation, flipping and scaling. The enhancement process is: first, the point cloud is rotated randomly along the vertical axis in the range of [-π/18, π/18], then flipped randomly along the front axis until all the ground truth boxes are evenly distributed at [0.95,1.05], finally, random scaling transformation is performed. Training loss Function. PEPF-Net is an end-to-end trainable network. During training, the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{L}_{reg}$$\end{document} loss function and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{L}_{ace}$$\end{document} loss function are only applicable to positive proposals generated by foreground point and the proposals with an IoU greater than 0.55 during the CNN convolution stage. Parameter Optimization. We use adaptive moment estimation55 for network optimization, its initial training parameters are set to universal reference values (learning rate 0.002, weight decay 0.001, momentum factor 0.9)56–58. On the NVIDIA RTX 3090ti(24G) GPU, the end-to-end training model is 80 epochs, its batch size is 2. The balance weighted value λ in the loss function is set to 5. This setting stems from the common practice of multi-task loss in 3D detection frameworks, aiming to enhance bounding box regression and proposal constraints, thereby avoiding excessive weakening of classification optimization. The focal loss parameters in the classification branch are set to α = 0.25 and γ = 2.0. This configuration follows the standard practice of focal loss in object detection tasks: α = 0.25 is used to down-weight the dominant impact of numerous background samples on the classification loss, while γ = 2.0 reduces the loss contribution from easy-to-classify samples, directing the model to focus more on foreground points and hard samples. To ensure comparability of experimental results, all comparative and ablation experiments in this paper adopt identical training configurations. This ensures that performance variations primarily reflect the impact of structural modules such as D-Pixel, PEP-F, and PVA-Net on the model. Ablation study We used KITTI to evaluate the effectiveness of proposed D-Pixel module, PEP-F module, L-TsfmNet module and PVA-Net module, proceed effectiveness evaluated. Superiority analysis of D-Pixel We verified the effectiveness of D-Pixel by replacing it with RGB. The experimental results are shown in Table 1, D-Pixel improves the 3DmAP of Car by 0.53% points. This fully demonstrates that enhancing RGB target feature representation with point cloud reflection intensity and position information to generate D-Pixel has a positive effect on 3D detection tasks. Table 1Ablation experiments based on KITTI validation data set (Car IoU = 0.7).PEP-FD-PixelPVA-NetL-TsfmNetEasyModerateHard3DmAPGain✓✓✓✓90.3784.3880.9885.24—✓×(RGB)✓✓89.7683.7280.6584.71↓0.53×(SF)✓✓✓88.1282.1678.0582.77↓2.47✓✓×✓89.8683.8480.7084.80↓0.44✓✓✓×89.9483.7780.6384.78↓0.46 Table 1Ablation experiments based on KITTI validation data set (Car IoU = 0.7).PEP-FD-PixelPVA-NetL-TsfmNetEasyModerateHard3DmAPGain✓✓✓✓90.3784.3880.9885.24—✓×(RGB)✓✓89.7683.7280.6584.71↓0.53×(SF)✓✓✓88.1282.1678.0582.77↓2.47✓✓×✓89.8683.8480.7084.80↓0.44✓✓✓×89.9483.7780.6384.78↓0.46 Ablation experiments based on KITTI validation data set (Car IoU = 0.7). Effectiveness analysis of PEP-F First, simple fusion (SF) was used to replace PEP-F to verify its effectiveness, and the experimental results are shown in Table 1. The PEP-F module improves the 3DmAP of car by 2.47% points. The comparison results demonstrate that PEP-F is effective in enhancing image semantic features using point cloud target features in a point-by-point manner, exerting a positive effect on 3D detection tasks. Second, single connect (SC) and multi-scale(M) SF were usage to replace multi scale(M) PEP-F, with the experimental results presented in Table 2. Compared with the baseline (which has no D-Pixel data input), the multi-objective 3DmAP of SC is improved by 1.03% points. This indicates that enhancing and fusing D-Pixel target features with point cloud target features yields a remarkable 3D perception effect. Compared with SC, the multi-objective 3DmAP of (M)SF increases by 1.13% points, proving the effectiveness of the multi-scale incremental enhancement fusion method. When comparing (M)SF and (M)PEP-F, the multi-objective 3DmAP of (M)PEP-F is improved by 1.44% points, confirming the high efficiency of the proposed PEP-F module. Table 2Analysis of fusion mechanism based on KITTI validation data set (IoU:Car 0.7/Pedestrian 0.5/Cyclist 0.5).SC(M)SF(M)PEP-FCarPedestrianCyclist3DmAPGain×××80.8946.9472.8866.90—✓××81.5848.3473.8767.93↑1.03—×✓×82.7749.4574.9569.06↑1.13—××✓85.2450.3675.9070.50↑1.44 Table 2Analysis of fusion mechanism based on KITTI validation data set (IoU:Car 0.7/Pedestrian 0.5/Cyclist 0.5).SC(M)SF(M)PEP-FCarPedestrianCyclist3DmAPGain×××80.8946.9472.8866.90—✓××81.5848.3473.8767.93↑1.03—×✓×82.7749.4574.9569.06↑1.13—××✓85.2450.3675.9070.50↑1.44 Analysis of fusion mechanism based on KITTI validation data set (IoU:Car 0.7/Pedestrian 0.5/Cyclist 0.5). The SF used in this paper refers to attaching the spatial coordinates of point cloud target feature points to D-Pixel feature using a single convolution layer. SC means that the fusion connection layer in PEPF-Net only retains the FPu layer (see Fig. 3), removing the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:S{A}_{1-3}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:F{P}_{1-3}$$\end{document}F layers. Effectiveness analysis of PVA-net PVA-Net is specially designed for PEPF-Net to extract multi-scale features of point clouds. Its purposes are to weigh vital feature points, improve the network’s ability to capture correlations between points, enhance the network’s capacity to generate 3D candidate boxes, and optimize point cloud feature extraction. To verify its effectiveness in PEPF-Net, the PVA-Net module was removed. As shown in Fig. 10, the 3DmAP of Car decreases by 0.44% points after removing PVA-Net, indicating that the PVA-Net module is actively effective for point cloud feature extraction. Effectiveness analysis of L-TsfmNet L-TsfmNet is specially designed for PEPF-Net to extract multi-scale features of images. Its purposes are to weight important feature information, improve the performance of convolutional neural networks, enhance the networks’ learning ability for 3D candidate boxes, and optimize image feature extraction. To verify its effectiveness in PEPF-Net, the L-TsfmNet module was removed. As shown in Fig. 11, the 3DmAP of Car decreases by 0.46% points after removing L-TsfmNet, indicating that the L-TsfmNet module is actively effective for extracting semantic features from D-Pixel images. Fig. 10Effectiveness analysis of PVA-Net in point cloud feature extraction process. Effectiveness analysis of PVA-Net in point cloud feature extraction process. Fig. 11Effectiveness analysis of L-TsfmNet in D-Pixel feature extraction process. Effectiveness analysis of L-TsfmNet in D-Pixel feature extraction process. Experimental results based on KITTI dataset The quantitative experimental results based on the KITTI test dataset are compared in Table 3. It can be seen from the comparison that the multi-objective (Car/Pedestrian/Cyclist) 3DmAP of the proposed PEPF-Net al.gorithm outperforms common advanced 3D detection algorithms. For example: It is 9.72% points higher than PointRCNN28 algorithm, 5.31% points higher than HotSpotNet59 algorithm, 2.47% points higher than EO-PVRCNN60 algorithm, 13% points higher than AVOD-FPN30 algorithm, 0.37% points higher than BiProDet13, the most advanced algorithm proposed in 2023. Among them, the 3DmAP of the target Car is improved by 1.19% points, and the 3DmAP of the target Pedestrian is improved by 0.2% points. Further analysis shows that the multi-objective 3DmAP of PEPF-Net is significantly superior to cascade algorithms8 as well as joint inference fusion algorithms based on RoI29,30,61 and voxel11,12. It is 5 to 9% points higher than 3D perception algorithms based on single-sensor (LiDAR) data, and 1 to 13% points higher than those based on multi-sensor (LiDAR + camera) data. The PEPF-Net algorithm is based on the study of raw data and its features. Taking “point cloud + pixel” as input, it is categorized as a “raw data-based 3D detection algorithm.” In recent years, significant progress has been made in 3D detection research leveraging large model-generated data and its features. These algorithms, which take “point cloud + pixel + generated pseudo-data” as input, can be defined as “generated data-based 3D detection algorithms. Although most generative data-based 3D detection algorithms achieve a slightly higher 3DmAP compared to those based on raw data, they generally lack feasibility validation for the actual sample data. Therefore, Table 3 of this paper focuses solely on comparing existing common actual advanced 3D detection algorithms based on raw data, excluding generative data-based 3D detection algorithms that emerged between 2024 and 2026 from the comparison. Table 3Comparison of advanced methods based on KITTI test data sets (The blue data is from the original text).AlgorithmInput3D Car(IoU = 0.7)3D Pedestrian (IoU = 0.5)3D Cyclist (IoU = 0.5)Multiple Target 3DmAPEasyModerateHard3D mAPEasyModerateHard3D mAPEasyModerateHard3D mAPPointRCNN28(2019) L 85.9475.7668.3276.6749.4341.7838.6343.2873.9359.6053.5962.3760.78PointGNN62(2019) L 88.3379.4772.2980.0351.9243.7740.1445.2878.6063.4857.0866.3963.90PV-RCNN63(2020) L 90.2581.4376.8282.8352.1743.2940.2945.2578.6063.7157.6566.6564.913DSSD64(2020) L 88.3679.5774.5580.8354.6444.2740.2346.3882.4864.1056.9067.8365.01HotSpotNet59(2020) L 87.6078.3173.3479.7553.1045.3741.4746.6582.5965.9559.0069.1865.19MV3D29(2017) L + I 71.2962.6856.5663.51 — — — — — — — — —AVOD-FPN30(2018) L + I 81.9471.8866.3873.4050.8042.8140.8844.8364.0052.1846.6154.2657.50F-PointNet8(2018) L + I 81.2070.3962.1971.2651.2144.8940.2345.4471.9656.7750.3959.7158.80F-ConvNet65(2019) L + I 85.8876.5168.0876.8252.3745.6141.4946.4979.5864.6857.0367.1063.47EP-Net66(2020) L + I 89.8179.2874.5981.23 — — — — — — — — —CAT-Det67(2022) L + I 89.8781.3276.6882.6254.2645.4441.9447.2183.6868.8161.4571.3167.05EO-PVRCNN60(2022) L + I 90.1382.0177.5383.2255.8447.0242.9448.6085.4169.1062.3072.2768.03REPF-Net1(2023) L + I 91.0979.8877.7882.92 — — — — — — — — —BiProDet13(2023) L + I 89.1382.9780.0584.0555.5948.7746.1250.1686.7474.3267.4576.1770.13 PEPF-Net (Ours) L + I 90.37 84.38 80.98 85.24 55.94 48.89 46.24 50.36 87.22 73.68 66.81 75.90 70.50 Table 3Comparison of advanced methods based on KITTI test data sets (The blue data is from the original text).AlgorithmInput3D Car(IoU = 0.7)3D Pedestrian (IoU = 0.5)3D Cyclist (IoU = 0.5)Multiple Target 3DmAPEasyModerateHard3D mAPEasyModerateHard3D mAPEasyModerateHard3D mAPPointRCNN28(2019) L 85.9475.7668.3276.6749.4341.7838.6343.2873.9359.6053.5962.3760.78PointGNN62(2019) L 88.3379.4772.2980.0351.9243.7740.1445.2878.6063.4857.0866.3963.90PV-RCNN63(2020) L 90.2581.4376.8282.8352.1743.2940.2945.2578.6063.7157.6566.6564.913DSSD64(2020) L 88.3679.5774.5580.8354.6444.2740.2346.3882.4864.1056.9067.8365.01HotSpotNet59(2020) L 87.6078.3173.3479.7553.1045.3741.4746.6582.5965.9559.0069.1865.19MV3D29(2017) L + I 71.2962.6856.5663.51 — — — — — — — — —AVOD-FPN30(2018) L + I 81.9471.8866.3873.4050.8042.8140.8844.8364.0052.1846.6154.2657.50F-PointNet8(2018) L + I 81.2070.3962.1971.2651.2144.8940.2345.4471.9656.7750.3959.7158.80F-ConvNet65(2019) L + I 85.8876.5168.0876.8252.3745.6141.4946.4979.5864.6857.0367.1063.47EP-Net66(2020) L + I 89.8179.2874.5981.23 — — — — — — — — —CAT-Det67(2022) L + I 89.8781.3276.6882.6254.2645.4441.9447.2183.6868.8161.4571.3167.05EO-PVRCNN60(2022) L + I 90.1382.0177.5383.2255.8447.0242.9448.6085.4169.1062.3072.2768.03REPF-Net1(2023) L + I 91.0979.8877.7882.92 — — — — — — — — —BiProDet13(2023) L + I 89.1382.9780.0584.0555.5948.7746.1250.1686.7474.3267.4576.1770.13 PEPF-Net (Ours) L + I 90.37 84.38 80.98 85.24 55.94 48.89 46.24 50.36 87.22 73.68 66.81 75.90 70.50 Comparison of advanced methods based on KITTI test data sets (The blue data is from the original text). PointRCNN28 (2019) PointGNN62 (2019) PV-RCNN63 (2020) 3DSSD64 (2020) HotSpotNet59 (2020) MV3D29 (2017) AVOD-FPN30 (2018) F-PointNet8 (2018) F-ConvNet65 (2019) EP-Net66 (2020) CAT-Det67 (2022) EO-PVRCNN60 (2022) REPF-Net1 (2023) BiProDet13 (2023) PEPF-Net (Ours) Feasibility experiments based on sample data To verify the engineering application value of the proposed PEPF-Net algorithm, visualization experiments were conducted based on KITTI sample data and actual sample data, with the results shown in Fig. 12. In the Fig, the red 3D boxes are calibration boxes, while the green 3D boxes are the estimation results of the PEPF-Net algorithm. In practical engineering applications, 3D annotation of measured data is not required.In Fig. 12 (a), (b), and (c) are experimental results based on KITTI sample data; (d), (e), and (f) are experimental results based on actual sample data (unstructured environmental data collected by integrating an Intel Realsense-D435i camera and an Ouster-OS1-64-line LiDAR). The experiments demonstrate that: (1) PEPF-Net algorithm has engineering application feasibility and effectiveness. (2) In the experiment, the PEPF-Net algorithm demonstrates high-precision 3D perception (classification, 3D positions, and orientation) of multi-category targets on the two types of sample data. (3) the missed detection and false detection rates of the PEPF-Net algorithm for far small targets in the dataset samples are better than those for far small targets in the actual sample data. Fig. 12Feasibility experiment based on sample data. Feasibility experiment based on sample data. Based on actual sample data, the PEPF-Net algorithm exhibited missed detections and false detections when detecting far small targets. The core reasons for this phenomenon mainly include: (1) It is difficult to achieve high-precision calibration for actual sample data in advance. (2) Dataset sample data usually maintains high distributional consistency with the training data. (3) The domain Offset between actual sample data and training data leads to a decline in the algorithm’s generalization capability in real-world scenarios. Compared to the training data, actual sample data exhibits near-out-of-distribution (near-OOD) and out-of-distribution (OOD) features. (4) PEPF-Net is a single-frame supervised learning model that relies on the current frame’s visual and point cloud spatial features to complete detection tasks. It lacks the capability emphasized by world models to “infer target state changes using historical observations.” To overcome missed and false detections, Subsequent work could deeply integrate semi-supervised or self-supervised learning paradigms based on world models to build a detection framework combining real-time perception with historical temporal reasoning, aiming to improve the robustness and generalization of 3D object detection in sparse, highly dynamic, and complex open environments through the full exploitation of spatiotemporal contextual information. Conclusion The PEPF-Net framework proposed in this paper is a 3D perception algorithm for unstructured environments. It consists of a source data enhancement fusion module, a D-Pixel feature generator (main data stream), a point cloud feature generator (enhanced data stream), a PEP-F enhancement fusion module, classification and 3D position estimation network. In the D-Pixel feature generator, the proposed L-TsfmNet realizes hierarchical segmentation and feature extraction, yielding multi-scale feature maps containing rich semantic information and different receptive fields. In the point cloud feature generator, the proposed PVA-Net enables the generation of global information tensors, obtaining multi-scale point cloud target features with abundant skeleton and position information. The PEP-F module then effectively enhances D-Pixel features using these point cloud features. Finally, numerous experiments verify the effectiveness of the D-Pixel, PVA-Net, L-TsfmNet, and PEP-F modules, as well as the superiority of PEPF-Net and its feasibility for engineering applications. Despite the superior performance and engineering value demonstrated by PEPF-Net, targeted optimization and adaptation are still necessary for its deployment on resource-constrained mobile robot platforms. Future work will focus on the in-depth processing of ‘fused images’ by incorporating advanced pure image processing algorithms68[,69 and image quality assessment algorithms70,72,72, to facilitate lightweight network design, model compression, and robust sensor calibration.