DDRN: DETR with dual refinement networks for autonomous vehicle object detection Authors: Jiayao Li, Chak Fong Cheang, Zhaolong Du, Xiaoyuan Yu, Suigu Tang, Qianxiang Cheng Published: 2025 ABSTRACT AbstractObject detection is an essential part of autonomous driving perception, as its effectiveness directly affects subsequent decision making and control. However, the difficulty of detecting small objects and the complexity of the driving environment poses challenges that prevent existing detectors from fully meeting the safety and efficiency requirements of autonomous driving systems. To address these challenges, we propose three components: Feature Pyramid Network- (FPN-), Classification Refinement Networks (CRN), and Localization Refinement Networks (LRN), and organize them into DETR with Dual Refinement Networks (DDRN). FPN- determines to what degree deep layers in FPN participate in the learning of shallow layers by reweighting during gradient back propagation, which addresses background misdetection in small object detection. CRN and LRN integrate FPN- features with predictions from advanced end-to-end detectors to further optimize classification and localization, respectively. Compared to several advanced autonomous driving object detectors, our method achieved an improvement of 1.4% and 1.1% in average precision (AP) on two public autonomous driving datasets, respectively. Extensive experiments have demonstrated the effectiveness, robustness, and applicability of DDRN.Keywords: Autonomous driving, Object detection, Refinement networks, Background misdetectionSubject terms: Engineering, Mathematics and computing Introduction The use of autonomous vehicle technology, according to studies, will reduce traffic accidents by 50–80%, increase traffic efficiency by 10–30%, and significantly improve driving comfort level1. Autonomous vehicles also have significant advantages in terms of enhancing road safety and traffic utilization2,3. Deep learning technology is driving the development of computer vision, enabling the gradual realization of autonomous driving4,5. Perception is a critical first step in autonomous driving and advanced driver assistance systems, and its effectiveness directly influences subsequent decision making and control6,7. Object detection is the most essential task in the autonomous driving perception process, which still faces significant challenges8. In the context of autonomous driving scenes, there is considerable room for improvement in detector performance due to the difficulty of detecting small objects and the complexity and variability of the driving environment. Real-time autonomous driving systems face many challenges. Figure 1 illustrates the typical detection challenges in three autonomous driving scenes. (1) In road traffic scenes, most of the detected objects are small. According to the statistical results of the BDD100K9 dataset, there are up to 77260 images containing small objects (area pixels). The proportion of small objects in the BDD100K is 55.2%. Due to their low resolution, blurred images, and limited information, small objects have weak feature representation capabilities. This means that very few features can be extracted during the feature extraction process. The detection of small objects remains a challenge to be overcome. (2) In the nighttime environment, many classes have similar characteristics, leading to classification errors in the detectors. Eliminating classification errors can greatly improve 10. (3) Rainy and snowy weather directly affects sensor capture, making object localization difficult. Eliminating localization errors can greatly improve 10. Autonomous vehicle object detectors face many challenges. (a) The objects are small. (b) At night, car taillights resemble traffic lights and traffic signs resemble billboards. This makes classification difficult for the detectors. (c) Rain or snow can cause blurry visual effects. This makes localization difficult for the detectors. To address the aforementioned autonomous vehicle object detection challenges, we propose DETR with Dual Refinement Networks (DDRN), which contributes three main components: FPN-, CRN, and LRN. The major contributions of this work are listed below. We proposed FPN-, which enhances multi-scale feature learning in FPN by introducing a learnable fusion factor in top-down connections and dynamically controls the degree to which deep-layer features influence shallow-layer learning, thereby improving small object detection in traffic scenes where such objects dominate. We proposed FPN-, which enhances multi-scale feature learning in FPN by introducing a learnable fusion factor in top-down connections and dynamically controls the degree to which deep-layer features influence shallow-layer learning, thereby improving small object detection in traffic scenes where such objects dominate. We proposed the Dual Refinement Networks (CRN and LRN), which integrate and learn from multi-scale features and predictions of an end-to-end high-performance object detector, further improving the performance of the autonomous driving detector in terms of classification and localization. We proposed the Dual Refinement Networks (CRN and LRN), which integrate and learn from multi-scale features and predictions of an end-to-end high-performance object detector, further improving the performance of the autonomous driving detector in terms of classification and localization. We proposed DDRN by integrating a state-of-the-art end-to-end object detector, FPN-, CRN, and LRN. We performed fair comparisons between DDRN and several advanced detectors on two public autonomous driving datasets. DDRN achieved 1.4% and 1.1% improvement in AP, demonstrating its efficiency, robustness, and applicability. We proposed DDRN by integrating a state-of-the-art end-to-end object detector, FPN-, CRN, and LRN. We performed fair comparisons between DDRN and several advanced detectors on two public autonomous driving datasets. DDRN achieved 1.4% and 1.1% improvement in AP, demonstrating its efficiency, robustness, and applicability. Related work Autonomous driving object detection techniques have evolved from universal object detection techniques. This section will first introduce the enlightening universal object detection techniques and small object detection that pose difficulties for autonomous driving perception, and finally focus on the recent autonomous driving object detectors. Universal object detectors Traditional object detectors based on deep learning can be divided into two categories: two-stage detectors11,12 and one-stage detectors13,14. Two-stage detectors typically involve two steps: first, generating region proposals from the image, and then refining these proposals to produce the final object bounding boxes15,16. Fast R-CNN17 first runs the image through a CNN to extract features and returns region of interest (ROI). It then applies pooling layers to the ROI to ensure that each region has the same size. Finally, the features of these regions are passed through a fully connected network and the results are returned using Softmax and linear regression. FPN18 proposed a top-down network architecture with lateral connections to build high-level semantic information across all layers with different scales. The introduction of FPN greatly promoted the improvement of detection network accuracy. One-stage object detection algorithms do not require a region proposal stage19,20. They directly generate the probability of object classes and position coordinates, thereby obtaining the final detection results in a single stage, resulting in faster detection speed. YOLOX14 proposed a unique decoupled head processing method and anchor-free design, which distinguishes it among the YOLO series algorithms21. Overall, two-stage detectors generally achieve higher accuracy than one-stage detectors. Although two-stage detectors perform well, they rely heavily on many manually designed components, such as non-maximum suppression (NMS) or anchor box generation. The emergence of DETR22 aims to address the manual dependency problem in two-stage detectors and to achieve a more flexible end-to-end detector23. DETR treats object detection as a set prediction problem and introduces a one-to-one set matching method based on Transformer24. DINO25 adopts the advanced query formulation of DAB-DETR26 and combines it with improved contrastive denoising techniques to achieve excellent performance. Chen et al.10 proposed RefineBox, a module that uses a lightweight refinement network to optimize the localization performance of DETR-like detectors. DDQ27 first arranges dense queries similar to traditional detectors, then selects different queries for one-to-one mapping. DDQ integrates the advantages of traditional and end-to-end detectors, significantly improving the performance of detectors. CO-DETR28 adopts a collaborative hybrid assignment training architecture, the key of which is to use versatile one-to-many label assignment to improve the training efficiency and effectiveness of the encoder and decoder. Currently, CO-DETR is the state-of-the-art (SOTA) on COCO, with its highest AP reaching 66%. DETR-like models have significantly improved the performance of detectors, even outperforming classical convolutional models. For the foreseeable future, DETR-like models will remain a research hotspot in the field of object detection. Small object detectors The prosperity of universal object detectors cannot hide the slow development in the field of small object detection29,30. The inherent structural characteristics of small objects, coupled with their poor visual appearance and noisy representations, make small object detection an extremely challenging task. C3Det31 relates the full image context with annotator inputs in a local and global manner via late-fusion and feature-correlation, respectively. Gong et al.32 used fusion factors to control the information passed from deep layers to shallow layers, enabling FPN to adapt to the detection of small objects. Chen et al.33 pointed out that feature prediction heads of different scales can also remain effective in small object detection, using multi-scale predictions is more effective than single-scale predictions. Zhu et al.34 proposed a new Dynamic Sparse Vision Transformer, which achieves more flexible computation allocation through dual-layer connections and achieves very significant results on small objects, especially in dense prediction tasks. These methods improve small object detection from different perspectives, but do not consider the negative impact of complex scenes in road traffic. Autonomous vehicle object detectors MFR-CNN35 proposed a new convolutional neural network model for autonomous vehicle object detection by using multi-scale local and global feature representations. The model proposed by MFR-CNN consists of two parts: a region proposal network that generates candidate object regions, and an object detection network that integrates multi-scale features and global information. Zhang et al.36 divide the image into patches and select patches containing objects of different scales, this way is based on a transformer-based network and integrates diffusion models to improve the detection accuracy. DRMNet37 can be applied to autonomous driving scenarios, which can simultaneously complete the tasks of vehicle detection, lane detection, and drivable area detection. Firstly, the backbone of the model is designed as a two-pathway structure, which is used to extract shallow detail information and deep semantic information, respectively. Secondly, a multi-scale feature fusion module is designed to fuse the extracted shallow detail and deep semantic information. Then, different detection branches are designed according to the different characteristics of each detection task. CTAFFNet38 employs dual branches of CNN and Transformer to extract local and global features from images, which are then adaptively fused. Subsequently, a novel adaptive feature fusion strategy is designed to integrate local high-frequency features with global low-frequency features, thereby obtaining comprehensive feature representations. Finally, CTAFFNet adopts an encoder-decoder architecture to facilitate the propagation of fused local-global information across different stages, ensuring the model’s generalization capability. C2FDA39 proposed a novel category-induced coarse-to-fine domain adaptation approach, which consists of three pivotal components: (1) Attention-induced coarse-grained alignment module; (2) Attention-induced feature selection module and (3) Category-induced fine-grained alignment module. The aforementioned methods have improved the field of autonomous vehicle object detection from various perspectives and have achieved promising results. However, there is still a considerable gap between current autonomous vehicle detectors and the safe and efficient implementation of autonomous driving systems, leaving significant room for improvement. Inspired by the enlightening universal detectors mentioned above and the recent autonomous driving object detectors, we propose DDRN, which consists of CO-DETR, FPN-, CRN, and LRN. CO-DETR is the state-of-the-art end-to-end universal detector; FPN- balances the feature learning of objects with different sizes by incorporating fusion factors, thus addressing the problem of background false positives in small object detection; CRN and LRN fuse pyramid features with the predictions of CO-DETR and further optimize classification and localization, respectively. Unlike existing one-stage and two-stage object detectors, DDRN does not adhere to a strict stage-wise architecture but shares similarities with two-stage detectors. It divides the detection into two stages: first, end-to-end detection, and then feature fusion and prediction. In the first stage, the end-to-end detection model acts as the region proposal network. In the second stage, the parameters of CO-DETR are frozen, and only the weights of FPN- and Dual Refinement Network are updated. DDRN addresses two key aspects: (1) In FPN, each layer must handle objects of varying scales (shallow layers focus on small objects, deep layers on large ones). However, autonomous driving scenarios predominantly feature small objects, necessitating a fusion factor to balance learning between shallow and deep layers. DDRN adaptively learns to control the degree to which deep layers reweight their contribution to shallow layer learning during backpropagation, thereby improving small object detection performance in traffic scenes; (2) For the two detection subtasks: classification and localization, DDRN proposes Dual Refinement Networks to specifically enhance performance. There is inconsistency between the well-trained model and the randomly initialized refinement network. Therefore, refinement networks only need to fit the output features of the frozen detector. DETR-like models are difficult to learn multiscale features due to the quadratic complexity of self-attention. DDRN integrates and learns multi-scale features with the predictions of CO-DETR, further increasing the potential for performance improvement of the autonomous vehicle object detector. Method Overview The framework of the DDRN designed for autonomous driving scenarios is shown in Fig. 2. It consists of four components: the frozen CO-DETR, FPN-, CRN, and LRN. CRN and LRN are dedicated to improving the prediction classification and localization, respectively. We will provide a detailed description of the components and offer insights into why it can work well. The framework of DDRN. FPN- balances the feature learning of objects with different sizes by incorporating fusion factors, thus addressing the problem of background false positives in small object detection. CRN and LRN fuse pyramid features with the predictions of CO-DETR and further optimize classification and localization, respectively. Method details The Dual Refinement Networks perform two main steps. First, the refinement networks use the feature pyramid network FPN- to extract features from the well-trained backbone of the CO-DETR detection model. Second, through a series of refinement modules, they effectively refine the classifications and localizations using multi-scale features predicted by the detector. During the training process, we freeze the parameters of the pre-trained object detector and only update the weights of FPN- and the Dual Refinement Networks. FPN- The backbone of the detector serves as the input to FPN-, which reduces the number of channels to a certain value C, C is considered as the model dimension. Fig. 3 illustrates the specific details of FPN-, where is set based on the ratio of the number of objects between adjacent layers in FPN. This procedure can be formulated as: where is the fusion factor; is the number of objects in the layer, where the number of objects is computed from the entire dataset; is a stability constant used to prevent the denominator from being zero. Our formula design is based on the fact that when detecting small objects, each layer finds it difficult to capture representative features for the detection task, which intensifies the competition between layers. More specifically, all layers aim to use their shared parameters to learn appropriate features for their respective detection tasks. Unfortunately, some layers may have significantly fewer training samples than others, resulting in gradients for these layers being at a disadvantage compared to others when updating shared parameters. Therefore, when is small or is large, this method sets a smaller to reduce the gradients generated by the detection tasks in , and vice versa, encouraging the network to learn the detection tasks of each layer equally. This improves the learning efficiency of small objects. The framework of FPN-. FPN- controls the degree of involvement of deep layers in shallow layer learning in FPN by the fusion factor . As an example, the gradient of the layer can be mathematically formulated as: here, represents the classification and regression losses corresponding to the i-th layer. The layer is influenced by two distinct tasks: object detection in deeper layers (, ) and object detection in shallower layers (, ). When applying a larger , the layer will acquire more information for shallow detection tasks while losing more data relevant to deep detection tasks, and vice versa. Furthermore, the concepts of “deep” and “shallow” are relative. For instance, acts as a deeper layer relative to but becomes a shallower layer relative to . In Feature Pyramid Networks, the detection responsibilities across different layers can be characterized as: shallow-layer features are utilized for small object detection, while deep-layer features are employed for large object detection. Therefore, the fusion factor is needed to control the priority of these two requirements and achieve a balance between them. determines to what degree deep layers in FPN participate in the learning of shallow layers by reweighting during gradient back propagation. If there are fewer objects in the deep layers or more objects in the shallow layers, the deep layers cannot help the shallow layers learn. In this case, each layer must focus more on its own learning. FPN- will set a smaller to reduce the fusion between adjacent layers, thereby improving the learning efficiency of small objects in traffic images. Dual refinement network Once DDRN obtains multi-scale features, DDRN applies m Classification Refinement Networks to improve class prediction and n Localization Refinement Networks to improve bounding box prediction. CRN and LRN have the same network structure, but they are distinguished by the purposes of classification prediction and localization prediction respectively. Dual Refinement Networks take the predictions from CO-DETR and the feature pyramid as input. DDRN corrects classes and bounding boxes by predicting the deltas between predictions and ground truth. Each refinement network consists of two ROI align layers40, two residual modules41 (as shown in Fig. 4), a gate unit that generates weights to select useful information, and a multilayer perceptron (MLP). The residual module of refinement networks consists of several bottleneck blocks. Each block stacks a 11 convolutional layer, a 33 convolutional layer, and a 11 convolutional layer. The bottleneck channels correspond to the input channels of the 33 convolutional layer. We use L1 loss and GIOU loss42 as regression losses, and Focal Loss43 as classification loss. We conducted ablation experiments to study the number of Classification Refinement Networks and Localization Refinement Networks. The mathematical expressions of CRN and LRN are shown below: here, RB represents Residual Block, RA denotes ROI Align, and indicates dot product. Details of the residual module used in Dual Refinement Networks. Training and inference During training, we select predictions that match the ground truth and improve the corresponding classification and localization. Other predictions that do not match the ground truth are ignored. For inference, we refine the top k predictions based on confidence scores. Since the ground truth is not accessible, the value of k is set to the maximum number of objects in a single image in the dataset, which is set to 100 in DDRN. The rationale for setting the value of k is provided in the ablation experiments. Why DDRN works Because of the fusion operations between adjacent layers, FPN is indeed multitask learning. If the top-down connections were removed from FPN, each layer would only need to focus on objects relevant to that layer, so that shallow layers learn small objects and deep layers learn large objects. In FPN, almost every layer must learn objects of all sizes. However, for autonomous vehicle object detection, a significant portion of the objects in the dataset are small objects. In addition, each layer not only needs to focus on objects of its corresponding size, but also needs help from other layers to obtain more training samples. Therefore, the fusion factor is needed to control the priority of these two requirements and achieve a balance between them. determines to what degree deep layers in FPN participate in the learning of shallow layers by reweighting during gradient back propagation. If there are fewer objects in the deep layers or more objects in the shallow layers, the deep layers cannot help the shallow layers learn. In this case, each layer must focus more on its own learning. According to (1), DDRN will set a smaller to reduce the fusion between adjacent layers, thereby improving the learning efficiency of small objects in traffic images. Recent DETR-like models generate a set of predictions and use a one-to-one mapping strategy to find positive samples for training. Reducing the error in positive samples can significantly improve the performance of DETR-like models10. For the two subtasks involved in the detection task: classification and localization, we propose Dual Refinement Networks each to improve the performance in a targeted manner. There is inconsistency between the well-trained model and the randomly initialized refinement network. Therefore, refinement networks only need to fit the output features of the frozen detector. DETR-like models are difficult to learn multiscale features due to the quadratic complexity of self-attention. DDRN integrates and learns multi-scale features with the predictions of CO-DETR, further increasing the potential for performance improvement of the autonomous vehicle object detector. Results Experiments settings The experimental platform’s operating system is Ubuntu 18.04. CUDA 11.3 is used for GPU acceleration. For fair comparisons, all experiments are implemented on PyTorch 1.12 and MMDetection 3.3.044. ResNet-5041 is the default backbone in this study if not specified. Most models adopt the 1x (12 epochs) training protocol in MMDetection44. We consider CO-DETR28 as our baseline. All hyperparameters and the optimization algorithm follow the settings in MMDetection44 if not specifically noted. Datasets The Berkeley Deep Drive 100k (BDD100K)9 is the largest open driving video dataset with 100K videos to evaluate progress of image recognition algorithms in autonomous driving. The dataset has geographic, environmental, and weather diversity. The road object recognition dataset we use has a total of 100,000 photos with an original size of 1280 720, and the dataset is randomly partitioned into training, validation, and test sets in the ratio of 6:2:2. The road object recognition dataset includes ten classes: car, bus, person, bike, truck, motor, train, rider, traffic sign, and traffic light. The KITTI dataset51, co-founded by the Karlsruhe Institute of Technology (KIT) and the Toyota Technological Institute at Chicago (TTIC), is one of the commonly used datasets for autonomous driving scenarios internationally. KITTI contains real-world image data collected from various scenes such as urban areas, rural areas, and highways. In this paper, KITTI is randomly partitioned into training, validation, and test sets in a ratio of 6:2:2. The object detection dataset includes six categories: Car, Van, Truck, Pedestrian, Cyclist, and Tram. Evaluation metrics All reported results follow standard COCO-style metrics, including AP (averaged AP over IoU thresholds ranging from 0.5 to 0.95 with an interval of 0.05), (AP for IoU threshold 0.5), (AP for IoU threshold 0.75), (AP for small objects: area pixels), (AP for medium objects: pixels < area pixels) and (AP for large objects: area pixels). The formula for AP is as follows: here, P(R) represents the precision value corresponding to recall R. Performance comparison We compare DDRN with state-of-the-art conventional detectors14,45,48–50,52 and recent end-to-end detectors25,27,28,46,47 on BDD100K and KITTI. Tables 1 and 2 show the results of the performance comparison. Table 1Performance comparison of models on BDD100K.MethodBackboneAPDynamic R-CNN (2020)45RN5022.642.220.79.128.738.4YOLOX (2021)14CSPDarknet5312.626.211.13.315.728.8ViTDet (2022)46ViT-B21.139.019.45.227.846.4DINO (2022)25RN5025.249.821.511.830.843.2CO-DETR (2023)28RN5028.852.427.214.133.750.5DiffusionDet (2023)47RN5026.250.523.114.030.541.9DDQ (2023)27RN5025.349.522.111.630.145.5SES-YOLOv8n (2024)48C2f22.341.9––––LSFM (2024)49ConvMLP-Pin28.255.724.4–––DGSS-YOLOv8s (2025)50C2f24.051.3––––DDRN (Ours)RN5030.254.928.314.635.653.1Dynamic R-CNN45RN10124.942.422.813.230.838.5DINO25RN10127.052.723.512.432.345.6CO-DETR28RN10131.555.130.113.537.157.1DiffusionDet47RN10127.953.824.914.232.246.5DDQ27RN10126.450.024.811.731.348.1DDRN (Ours)RN10133.157.431.914.939.357.7Significant values are in bold. Performance comparison of models on BDD100K. Significant values are in bold. Table 2Performance comparison of models on KITTI.MethodBackboneAPDynamic R-CNN (2020)45RN5064.890.173.759.466.768.0YOLOX (2021)14CSPDarknet5341.974.642.026.342.449.9ViTDet (2022)46ViT-B60.388.669.447.362.866.0DINO (2022)25RN5061.986.772.556.063.467.8CO-DETR (2023)28RN5065.190.376.054.866.072.4DiffusionDet (2023)47RN5057.385.164.355.657.962.1DDQ (2023)27RN5052.882.858.545.252.062.3HRYNet (2024)52C2f67.190.676.9–––DGSS-YOLOv8s (2025)50C2f65.486.9––––DDRN (Ours)RN5068.290.978.162.669.871.1Dynamic R-CNN45RN10166.090.376.261.066.869.9DINO25RN10165.389.274.858.665.271.9CO-DETR28RN10165.689.774.361.467.068.8DiffusionDet47RN10159.487.166.054.159.766.4DDQ27RN10155.180.863.247.256.262.4DDRN (Ours)RN10167.791.276.861.869.370.5Significant values are in bold. Performance comparison of models on KITTI. Significant values are in bold. To validate DDRN’s detection performance in autonomous driving scenarios, this paper conducts comparative experiments with several mainstream detection frameworks under identical experimental conditions. Based on the ResNet50 backbone, DDRN achieves 30.2% AP, significantly surpassing CO-DETR’s 28.8% (AP = 1.4%), while improve to 28.3%, demonstrating the synergistic enhancement of dual-branch optimization networks on classification accuracy and localization reliability. Although DDRN’s is slightly lower than LSFM’s, DDRN attains optimal performance across small ( = 14.6%), medium ( = 35.6%), and large ( = 53.1%) object detection dimensions, demonstrating that FPN- effectively mitigates background misclassification through gradient reweighting. With ResNet101 backbone, DDRN further boosts AP to 33.1%, achieving 57.4%/31.9% on / and 14.9%/39.3%/57.7% on // metrics, comprehensively outperforming existing methods. These results fully validate the robustness and generalization capability of DDRN’s feature-enhanced architecture with dual refinement networks in complex driving environments, providing an optimal solution balancing detection accuracy and computational efficiency for autonomous driving systems. This study also conducted systematic comparative experiments on the KITTI dataset, using ResNet50 and ResNet101 as backbone networks for performance evaluation. Experimental results show that under the ResNet50 configuration, DDRN achieves 68.2% average precision (AP), surpassing the second-place HRYNet by 1.1%, with and reaching 90.9% and 78.1% respectively, significantly improving detection accuracy. Particularly in small object detection (), DDRN leads existing state-of-the-art model Dynamic R-CNN (59.4%) by 3.2% with 62.6% value, validating the effectiveness of FPN- in reducing background false detections. When using ResNet101 as the backbone, DDRN maintains 67.7% AP, outperforming all other methods, while demonstrating balanced advantages across (61.8%), (69.3%), and (70.5%) metrics. These experimental results fully demonstrate DDRN’s robustness in complex driving scenarios, particularly validating its innovative design for small object detection and complex background environments, providing a safety-efficient solution for autonomous driving perception systems. Figure 5 shows the Precision-Recall (PR) curve graphs of baseline and DDRN when facing small objects on BDD100K. We can see that the of DDRN has increased by 1.7% when detecting small objects. In addition, the area of the purple region in Fig. 5b is reduced by 1.8 percentage points compared to that in Fig. 5a, indicating a significant reduction in the number of small objects falsely detected by DDRN in the background region. PR curve graph of models facing small objects on BDD100K. (a) Baseline. (b) DDRN. The curves are as follows: C75: PR at IoU = 0.75. C50: PR at IoU = 0.50. Loc: PR at IoU = 0.10 (localization errors ignored, but not duplicate detections). All remaining settings use IoU = 0.10. Sim: PR after supercategory false positives are removed. Oth: PR after all class confusions are removed. BG: PR after all background (and class confusion) false positives are removed. FN: PR after all remaining errors are removed. The area of the purple region in (b) is reduced by 1.8 percentage points compared to that in (a), indicating a significant reduction in the number of small objects falsely detected by DDRN in the background region. Table 3 presents a comprehensive comparison between the proposed DDRN and several state-of-the-art object detection methods in terms of parameter count (Params), Frames Per Second (FPS), and average precision (AP). DDRN, built upon a ResNet-50 backbone, achieves an AP of 30.2 while maintaining a competitive inference speed of 7.1 FPS and a parameter count of 47.9M. Compared to other methods with similar backbones, DDRN outperforms Dynamic R-CNN, DINO, DiffusionDet, and DDQ in both accuracy and speed. Notably, it surpasses the recent CO-DETR (28.8 AP, 7.8 FPS) with a higher AP and a marginally lower FPS. Although some lightweight models like DGSS-YOLOv8s exhibit significantly higher FPS (72.0), they lag behind DDRN in accuracy by a substantial margin (24.0 AP versus 30.2 AP). These results indicate that DDRN effectively balances detection performance and computational efficiency, making it highly promising for application in autonomous driving. Table 3Comparison of the proposed model with several state-of-the-art object detection methods in terms of parameter count and inference speed.MethodBackboneParams (M)FPSAPDynamic R-CNNRN5041.54.122.6YOLOXCSPDarknet5354.93.312.6ViTDetViT-B124.21.421.1DINORN5049.74.525.2CO-DETRRN5037.37.828.8DiffusionDetRN5053.14.426.2DDQRN5042.02.525.3SES-YOLOv8nC2f3.536.222.3LSFMConvMLP-Pin12.121.928.2DGSS-YOLOv8sC2f15.472.024.0DDRNRN5047.97.130.2Significant values are in bold. Comparison of the proposed model with several state-of-the-art object detection methods in terms of parameter count and inference speed. Significant values are in bold. Ablation experiments To investigate the effects of different components in the DDRN and the influence of multiple parameters, we performed numerous ablation experiments on BDD100K, which has a large data sample and is therefore more representative. Components To validate the contribution of DDRN components to detection performance, this paper designs component ablation experiments and evaluates the average precision metric. As shown in Table 4, the experimental results demonstrate that introducing FPN- increases AP from 28.8 to 29.4% (AP = +0.6%), proving its effectiveness in mitigating background misclassification in small object detection through gradient reweighting mechanisms; when CRN or LRN is independently integrated with FPN-, AP reaches 29.9% (AP = +1.1%) and 29.7% (AP = +0.9%) respectively, validating the independent efficacy of classification refinement and localization refinement modules; simultaneous integration of CRN and LRN further improves AP to 30% (AP = +1.2%), revealing synergistic gains from dual-branch optimization. Notably, increasing CRN/LRN stacking layers from 1 to 2 sustains AP improvement to 30.1% (AP = +1.3%), but performance regresses at 4 layers (AP = +1.0%), revealing nonlinear relationships between network depth and detection performance. This experiment comprehensively demonstrates FPN-’s feature enhancement capabilities and CRN/LRN’s fine-grained optimization for detection tasks, ultimately adopting the optimal configuration of 1-layer CRN and 2-layer LRN (AP = +1.4%) as the critical support for DDRN’s overall performance advancement. Table 4Results of different number of components in DDRN.FPN-CRNLRNAPAP0028.8–0029.4+ 0.61029.9+ 1.10129.7+ 0.91130.0+ 1.22130.2+ 1.41230.0+ 1.22230.1+ 1.34429.8+ 1Significant values are in bold. Results of different number of components in DDRN. Significant values are in bold. Fusion factors To evaluate the effectiveness of the proposed FPN- module, we conduct an ablation study on the BDD100K dataset to analyze the impact of different values. As shown in Table 5, when using the default setting ( = 1), the baseline AP is 29.3. Adjusting to values such as 1.25 and 0.75 leads to a decrease in performance, with AP dropping to 28.6 and 29, respectively. However, when is set to 0.5 and 0.25, the AP increases to 29.7 and 29.4, indicating that reducing the fusion factor can significantly improve detection performance on this dataset. Notably, the proposed FPN-, which automatically determines the fusion factor based on the statistical distribution of objects across FPN layers, achieves the best performance with an AP of 30.2, surpassing the baseline by 0.9 AP. These results demonstrate that adaptively adjusting the fusion factor is beneficial for object detection. Table 5Results of different in DDRN.APAP129.3–1.2528.6-0.70.7529.0-0.30.529.70.40.2529.40.1FPN-30.20.9Significant values are in bold. Results of different in DDRN. Significant values are in bold. Training epochs To validate the impact of training epochs on model performance, this paper conducts incremental training experiments based on the baseline model. The experimental results are shown in Table 6. Results demonstrate that adopting DDRN architecture with extended training cycles consistently improves average precision (AP): when increasing to 24 epochs (from baseline 12), AP rises from 28.8 to 30.2% (AP = +1.4%); further extending to 24 and 36 epochs achieves AP of 30.5% (AP = +1.7%) and 30.6% (AP = +1.8%) respectively, though improvement margins gradually narrow. Precision analysis reveals that significantly increases from the baseline 52.4 to 55.3%, indicating enhanced classification accuracy with prolonged training; improves from 27.2 to 28.5% before plateauing; reflecting the performance boundary in maintaining high-confidence detection consistency. The experiment demonstrates that extended training effectively boosts model performance while exhibiting diminishing marginal returns, providing empirical evidence for DDRN’s convergence stability. Table 6Results of DDRN with different epochs.ModelEpochsAPBaseline1228.852.427.2DDRN+ 1230.254.928.3DDRN+ 2430.555.228.5DDRN+ 3630.655.328.5Significant values are in bold. Results of DDRN with different epochs. Significant values are in bold. Top k To validate the impact of proposal box quantity (k value) on detection performance, this paper conducts ablation experiments with varying proposal scales under the DDRN framework, as shown in Table 7. Experimental results demonstrate that the AP metric exhibits a gradual improvement trend as k increases: when k grows from 10 to 30, AP improves from 29.2 to 29.9% (AP = +0.7%), with / rising from 53.5%/27.6% to 54.4%/28.1%; further increasing k to 100 achieves AP = 30.2% (AP = +1.0%) and synchronized / optimization to 54.9%/28.3%; however, performance plateaus at k = 300 (AP = 30.2%), indicating the model has fully exploited detection potential in feature space when proposal count reaches 100. This experiment reveals the nonlinear relationship between proposal scale and detection performance, where selecting k = 100 avoids redundant computation while maintaining synchronized accuracy improvement, providing a performance-efficiency trade-off basis for DDRN’s parameter configuration. Table 7Results of DDRN with different values of k.kAP1029.253.527.63029.954.428.110030.254.928.330030.254.928.3 Results of DDRN with different values of k. Discussion To evaluate the efficiency and applicability of DDRN in autonomous vehicle object detection, we compared the results of different models in complex scenarios, as shown in Fig. 6. Dashed lines in the figures indicate misdetections or missed detections, and the number adjacent to each box represents confidence scores. Results of different models on BDD100K. (a) Ground truth. (b) Dynamic R-CNN. (c) YOLOX. (d) ViTDet (Since ViTDet uses Mask R-CNN40, there are masks in the image). (e) DINO. (f) CO-DETR. (g) DiffusionDet. (h) DDQ. (i) DDRN (Ours). Dashed lines in the figures indicate misdetections or missed detections, and the number adjacent to each box represents confidence scores. Compared to the ground truth in Fig. 6a, Dynamic R-CNN, CO-DETR, DiffusionDet, and DDQ misidentified a reflection of a car’s brake light as a traffic light; YOLOX missed a right turn sign; ViTDet misidentified an extra car; DINO misidentified both a traffic light and an extra car. Consistent with Fig. 5, DDRN optimized the detection performance of small objects in the background region. DDRN achieved detection of all five objects with high confidence scores of 99.9%, 99.6%, 92.5%, 90.7%, and 78.5%. We attribute this to DDRN’s use of FPN- to emphasize balanced learning between objects with different sizes, and the further enhancement of classification and localization effects using CRN and LRN. Beyond the results shown in Fig. 6, the consistent state-of-the-art performance on the BDD100K (Table 1) and KITTI (Table 2) datasets further demonstrates DDRN’s strong generalization ability. Notably, DDRN achieves the highest small object detection performance () on both benchmarks: 14.6 on BDD100K (outperforming CO-DETR by 0.5) and 62.6 on KITTI (surpassing CO-DETR by 7.8). This robustness stems from two synergistic innovations: (1) The adaptive FPN fusion factor alone contributes 0.6 AP in ablation studies (Table 4), dynamically balancing scale-specific feature learning to handle diverse object scales; (2) The dual refinement network jointly optimizes classification and localization, delivering a total AP improvement of 1.4 when combined with . This architecture ensures consistent detection accuracy under varying environmental conditions and domain shifts, which is critical for real-world deployment where adverse weather, occlusions, and scale variations are inevitable. Limitations of the study This paper improves the performance of autonomous driving object detectors by addressing small object detection and environmental diversity. However, for those large objects that can be easily detected, the performance improvement of DDRN is quite limited and needs to be further improved in the future. Scenario mining from extensive autonomous driving datasets is crucial for the validation of self-driving systems53; therefore, the robustness of DDRN requires further evaluation across a broader set of datasets. Additionally, the modules added in this paper increase computational complexity, inevitably reducing inference speed, which is another direction for future improvement. Several interesting directions not discussed in this paper can be further explored. For example, the fusion factor can balance the degree to which deep layers contribute to shallow layer learning, which helps in learning objects with different sizes. Dual Refinement Networks further increase the potential for improving the performance of the detector. These features can be applied to other tasks, such as semantic segmentation, to improve the model’s performance on objects with different sizes. Conclusion In this work, we analyzed the challenges faced by current autonomous vehicle object detectors: the difficulty of detecting small objects and the complexity and variability of traffic environments. To address these challenges, we proposed three components: FPN-, CRN, and LRN, and organized them into DDRN accordingly. FPN- balances feature learning from objects with different sizes through fusion factors, addressing the problem of background false positives in small object detection. CRN and LRN fuse FPN- features with predictions from state-of-the-art end-to-end detectors, further optimizing classification and localization, respectively. DDRN outperforms several advanced detectors on two public autonomous driving datasets, and extensive experiments demonstrate the efficiency, robustness, and applicability of our approach. In the future, we will extend DDRN to multimodal fusion technology to further enhance detection robustness; we will also apply DDRN to additional driving scenarios, such as those in the Argoverse 2 dataset54, which includes high-definition maps and geometric representations of objects, to explore potential improvements; additionally, improving DDRN through knowledge distillation techniques or lightweight design is a promising direction worth exploring. Author contributions Jiayao Li was accountable for the development of the study, execution of experiments, analysis of the findings, and preparation of the manuscript. Chak Fong Cheang supervised the project and offered a critical evaluation of the written document. Zhaolong Du contributed to mathematical modeling and experimental data analysis. Xiaoyuan Yu and Suigu Tang supervised the project and contributed to mathematical modeling. Qianxiang Cheng contributed to experimental data analysis. All authors participated in the review of the manuscript. Funding This work was supported by the Science and Technology Development Fund, Macau SAR, under Grant 0067/2023/RIB3 and Grant 0089/2022/A. Data availability The data that support the findings of this study are openly available at http://bdd-data.berkeley.edu/ and https://www.cvlibs.net/datasets/kitti/. Declarations Competing interests The authors declare no competing interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References Associated Data Data Availability Statement The data that support the findings of this study are openly available at http://bdd-data.berkeley.edu/ and https://www.cvlibs.net/datasets/kitti/.