Enhanced real-time 6d pose estimation for automatic recovery of in-flight UAVs using distance-aware keypoint heatmaps Authors: Mingyu Jeong, Andrew Jaeyong Choi Published: 2025 ABSTRACT AbstractUnmanned Aerial Vehicles (UAVs) play crucial roles across various fields but face significant challenges when control is lost or securing a safe runway is not feasible. Under these circumstances, reliable UAV recovery systems are essential for safe retrieval, requiring precise and real-time localization. This paper proposes an automatic in-fight UAV recovery system using an unmanned ground vehicle when securing a safe runway is challenging such as unstructured road, and presents a practical approach for real-time six-degree-of-freedom (6-DOF) pose estimation of an in-flight UAV using monocular RGB images and deep learning-based heatmap keypoint detection. To enhance the accuracy of the pose estimation, we propose an Adaptive Sigma technique for keypoint detection, which adjusts the sigma values of the keypoint heatmap based on the distance between the camera and the UAV. Thus, the proposed method can robustly adapt to changes in the distance of UAV to improve the keypoint localization performance. By utilizing the predicted keypoints in a Perspective-n-Point algorithm, the 6-DOF pose information of the UAV can be obtained in real time. The proposed Adaptive Sigma to heatmap-based keypoint detection improves the Percentage of Correct Keypoints by up to 2.5%, with consistently high performance across various state-of-the-art backbone architectures. The proposed method qualitatively evaluated in challenging scenarios such as various altitudes, significant tilt, and motion blur.Subject terms: Engineering, Mathematics and computing, Optics and photonics Introduction Unmanned Aerial Vehicles (UAVs) are aircraft operated remotely or autonomously without an onboard human pilot. With advancements in UAV technology, their applications have significantly expanded, particularly in military contexts for reconnaissance, surveillance, and tactical strikes. However, UAVs such as fixed-wing models inherently require a runway for safe landing. To address this limitation, recent research has investigated alternative recovery methods that operate without the need for a runway1–3. But, they often risk causing structural damage or the loss of sensitive data. To address these issues, a potential solution to these drawbacks is an autonomous in-flight recovery system using a ground vehicle to capture the UAV, as conceptually illustrated in Fig. 1. For such a system to be feasible, the ground vehicle must be capable of robust and accurate real-time 6-DOF pose estimation of the UAV. This paper specifically focuses on the final docking stage (Phase 3) of this concept, where precise localization is crucial for a successful capture. Conceptual overview of the in-flight UAVs automatic recovery system using a mobile manipulator robot platform with 6D pose estimation. While the figure illustrates the overall system-level goal, this paper focuses on evaluating the 6D pose estimation module, with experimental validation conducted using a quadcopter for practical lab-scale implementation. From the perspective of the recovery UGV, obtaining the UAV’s accurate and real-time 6-DOF (six-degree-of-freedom) pose is a key requirement. To address this in a practical and cost-effective manner, we propose a method for real-time 6-DOF pose estimation using only a monocular RGB camera, avoiding the need for expensive external localization equipment. Our approach is a two-stage process: we first detect the 2D keypoints of the in-flight UAV using a heatmap-based deep learning approach, and then solve the Perspective-n-Point (PnP) problem to compute its 6D pose. Unlike conventional heatmap-based keypoint detection methods, this paper introduces the concept of Adaptive Sigma to achieve more effective precision in UAV localization. The Adaptive Sigma technique adjusts the Gaussian distribution’s sigma value according to the distance between the UAV and the camera. This adjustment is crucial as it allows the model to account for variations in the apparent size of the UAV in the captured image, enhancing the accuracy of keypoint localization. The performance of this approach was evaluated through extensive experiments using a custom dataset labeled with UAV keypoints. The experiments included comparisons with various state-of-the-art backbone architectures. The effectiveness of the Adaptive Sigma method was demonstrated through improvements in the Percentage of Correct Keypoints (PCK) metric, frame rates (FPS), and model parameters. In summary, the main contributions of this paper are: We propose a distance-aware Adaptive Sigma technique for keypoint heatmap generation, which improves 2D localization accuracy by explicitly modeling the UAV’s scale variations in monocular images. We propose a distance-aware Adaptive Sigma technique for keypoint heatmap generation, which improves 2D localization accuracy by explicitly modeling the UAV’s scale variations in monocular images. We conduct a comprehensive empirical study showing that Adaptive Sigma consistently boosts keypoint detection performance (up to 2.5% in PCK) across diverse modern backbone architectures. We conduct a comprehensive empirical study showing that Adaptive Sigma consistently boosts keypoint detection performance (up to 2.5% in PCK) across diverse modern backbone architectures. We present a practical and real-time UAV 6D pose estimation framework on an embedded platform, demonstrating robustness under challenging conditions such as severe tilt and motion blur in in-flight recovery scenarios. We present a practical and real-time UAV 6D pose estimation framework on an embedded platform, demonstrating robustness under challenging conditions such as severe tilt and motion blur in in-flight recovery scenarios. Related works UAVs automatic recovery systems Unmanned Aerial Vehicles (UAVs) are increasingly utilized in various applications, especially in military operations for reconnaissance, surveillance, and tactical strikes. Given the high costs associated with UAVs and the critical information they may carry, ensuring their safe recovery after missions is imperative. Efficient recovery and redeployment of UAVs not only enhance their operational range but also improve cost-effectiveness. However, reliable recovery of UAVs poses significant challenges, particularly when control is lost or suitable landing zones are unavailable. Such scenarios introduce substantial risks, including the potential loss of valuable equipment and sensitive data. Recent studies have explored UAV recovery in different environments-marine, road, and aerial spaces-to address these challenges4. One notable approach involves using two multirotor UAVs equipped with nets to intercept an incoming UAV mid-air5. While this aerial recovery technique showcases innovative solutions, our research adopts a different strategy. We focus on tracking and capturing descending UAVs from the ground using Unmanned Ground Vehicles (UGVs) equipped with robotic manipulators. This ground-based approach aims to provide a practical and efficient solution for UAV recovery operations. By obtaining precise 6-DOF pose information of the UAV from a ground perspective, our method enhances the accuracy and reliability of the recovery process. This approach not only mitigates the risks associated with uncontrolled UAV descents but also contributes to the overall safety and effectiveness of UAV deployments. In summary, effective UAV recovery methods are crucial due to the high value of UAVs and the potential risks involved in their operations. Our research contributes to this field by proposing a ground-based tracking and capture system that ensures the safe and efficient recovery of UAVs, addressing the challenges posed by loss of control and lack of suitable landing zones. 6D rigid body pose estimation 6D pose estimation for rigid objects aims to determine the 3D rotation and 3D translation of an object relative to a camera. Modern approaches can be broadly categorized into correspondence-based methods and direct regression methods. We review these two mainstreams and then focus on heatmap-based techniques, which our work is built upon. Correspondence-based methods: These methods follow a two-stage pipeline: first, they establish correspondences between 2D image pixels and the object’s 3D coordinates, and second, they solve for the 6D pose using a Perspective-n-Point (PnP) algorithm. While early works relied on hand-crafted features like SIFT, recent deep learning-based methods have shown significant improvements. For instance, PVNet6 pioneered a pixel-wise voting approach, where each pixel votes for the direction of 3D keypoints, making it robust to heavy occlusion. Another notable method, GDRNet7, learns an intermediate geometry-aware representation to establish dense correspondences, proving effective even for texture-less objects. These methods demonstrate that high-quality correspondences are key to accurate pose estimation. Correspondence-based methods: These methods follow a two-stage pipeline: first, they establish correspondences between 2D image pixels and the object’s 3D coordinates, and second, they solve for the 6D pose using a Perspective-n-Point (PnP) algorithm. While early works relied on hand-crafted features like SIFT, recent deep learning-based methods have shown significant improvements. For instance, PVNet6 pioneered a pixel-wise voting approach, where each pixel votes for the direction of 3D keypoints, making it robust to heavy occlusion. Another notable method, GDRNet7, learns an intermediate geometry-aware representation to establish dense correspondences, proving effective even for texture-less objects. These methods demonstrate that high-quality correspondences are key to accurate pose estimation. Direct Regression Methods: In contrast to correspondence-based approaches, direct regression methods aim to predict the 6D pose from the input image in an end-to-end fashion. PoseCNN8 was an early attempt that regressed pose parameters directly, but its accuracy was limited, partly due to the challenges in regressing discontinuous rotation representations. More recent works have tried to overcome this by designing better loss functions or employing iterative refinement schemes9,10. However, these methods often exhibit lower accuracy compared to correspondence-based approaches, as they do not explicitly leverage geometric constraints during the pose calculation. Direct Regression Methods: In contrast to correspondence-based approaches, direct regression methods aim to predict the 6D pose from the input image in an end-to-end fashion. PoseCNN8 was an early attempt that regressed pose parameters directly, but its accuracy was limited, partly due to the challenges in regressing discontinuous rotation representations. More recent works have tried to overcome this by designing better loss functions or employing iterative refinement schemes9,10. However, these methods often exhibit lower accuracy compared to correspondence-based approaches, as they do not explicitly leverage geometric constraints during the pose calculation. Recent foundation models, such as FoundationPose11, achieve strong generalization to novel objects using a complex render-and-compare framework. However, this approach’s computational overhead from rendering numerous pose hypotheses makes it challenging for real-time, embedded applications. Additionally, their optimal performance often relies on RGB-D input, which is impractical for outdoor scenarios due to sunlight interference with depth sensors. In contrast, our work employs a direct, keypoint-based method using only RGB images, targeting a lightweight and robust solution for real-time outdoor operations. Our research builds on the correspondence-based paradigm, leveraging heatmap-based keypoint detection, a technique proven effective in human pose estimation (HPE)12. This method involves two main steps: first, predicting a probabilistic heatmap for each predefined keypoint on the object; second, using these detected 2D keypoints and their known 3D model coordinates to solve for the object’s 6D pose via a Perspective-n-Point (PnP) algorithm13. As illustrated in Fig. 1, this corresponds to Phase 3 of the UAV recovery process, where reliable 6D pose estimation is essential for successful docking. During this phase, the UAV captured by the ground camera may appear at varying distances and scales-sometimes fully visible, and other times only partially within the field of view. To ensure consistent pose estimation under such conditions, we formulate the task as a Perspective-n-Point (PnP) problem that remains solvable even when only a subset of keypoints is visible. While this heatmap-PnP pipeline is robust, a common limitation in existing works is the use of a fixed-sigma Gaussian kernel to generate ground-truth heatmaps for training. This approach overlooks the significant scale variations of an object as its distance to the camera changes, potentially degrading localization accuracy. To address this gap, we introduce a Distance-Aware Adaptive Sigma technique. Unlike these previous works, our key idea is to dynamically adjust the sigma value of the Gaussian kernel based on the object’s distance, allowing the model to learn scale-adaptive heatmaps and leading to more accurate keypoint localization across varying scales. This simple modification enhances keypoint precision across various distances, thereby improving the accuracy of the final 6D pose estimated by the PnP solver, making our framework more suitable for dynamic, real-world operations. Problem formulation As shown in Fig. 1, in the context of an Unmanned Aerial Vehicle (UAV) recovery system, we consider a scenario where a UAV needs to land in an unstructured environment without access to a conventional runway. The recovery system comprises an Unmanned Ground Vehicle (UGV) equipped with a robotic manipulator designed to capture the descending UAV using its end effector. The successful execution of this task necessitates precise localization of the UAV relative to the UGV. The primary objective is to determine the 6D pose of the UAV as it descends, using an RGB camera mounted on the UGV. This camera is oriented upwards, ensuring continuous monitoring of the UAV’s underside during its descent. The task is to predict the UAV’s pose ( orientation and position) in real-time using RGB Image captured by the camera. This estimated pose is then utilized within a predefined transformation (TF) system to guide the UGV’s manipulation system for capturing the UAV. We replaced the fixed-wing UAV with a quadcopter (multirotor UAV) due to limitations related to the flying speed and to conduct the lab-scale experiments with validation 6-DoF pose estimation To address the challenge of 6-DOF pose estimation for UAVs, we adopt a correspondence-based method. Directly acquiring ground truth pose data for training a model to regress 6-DOF pose is prohibitively expensive. Therefore, we employ a heatmap-based approach to predict keypoints on the UAV, leveraging these predictions as input to a PnP solver to determine the final 6-DOF pose. 6-DoF pose estimation pipeline Our proposed 6D pose estimation pipeline is illustrated in Fig. 2. Given a monocular RGB image of the UAV captured by the camera, we first extract image features using a backbone network. These features are then passed to a decoder, which generates heatmaps representing the likelihood of each keypoint’s presence for each predefined class. Keypoint locations are extracted from these heatmaps through post-processing. Finally, by establishing correspondences between the extracted 2D keypoints and a priori defined 3D points on the UAV, we utilize a PnP solver, along with camera intrinsics, to compute the 6D pose of the UAV. Our 6D pose estimation flow for UAV localization. Distance-aware sigma adaptation for heatmap generation In heatmap-based keypoint detection tasks, keypoints within an image are typically represented using heatmaps generated from 2D Gaussian distributions. These distributions are centered at the keypoints’ 2D coordinates and characterized by a predefined sigma (). However, applying a uniform sigma fails to account for variations in the object’s apparent size due to changing distance from the camera, which can limit the model’s localization precision. As shown in Fig. 3, We propose a method to adaptively adjust the sigma of the Gaussian distribution based on the actual distance between the object and the camera when generating heatmaps. The apparent size of the object in the image diminishes as it moves further from the camera, altering the image’s informational characteristics. By leveraging these variations, the model can more easily make accurate predictions. For instance, when an object is distant and appears smaller in the image, using a smaller sigma provides more precise positional information. The heatmaps generated using this method are utilized for training the Backbone and Decoder, as illustrated in Fig. 2a. Overview of adaptive sigma assignment for keypoint heatmap generation during model training: (a) illustrates heatmap creation process based on the object in all images, and (b) illustrates 2d heatmap generation by determining sigma based on the distance between objects and the camera. Variations in the object’s apparent size in the image also impact existing models. In transformer-based models like Vision Transformers (ViT)14, the number of image patches that contain the object varies depending on the object’s apparent size in the image. In convolutional neural network (CNN) models, the amount of information captured by the receptive field varies depending on the object’s dimensions within the image. By exploiting these characteristics, we can enhance the linearity of the model’s localization capabilities. By adjusting sigma to align with the object’s actual distance and its apparent size in the image, we reduce the complexity of heatmap generation for the model and achieve more accurate keypoint localization. To calculate the distance between the object and the camera, various methods are available. In this study, we employ the Perspective-n-Point (PnP) algorithm using pre-labeled keypoints to estimate this distance. It is crucial to clarify that this distance calculation is performed as a one-time, offline preprocessing step on the training dataset. We use the manually annotated ground-truth (GT) 2D keypoints to compute a ground-truth distance for each training image. This pre-computed distance is then saved and used to assign the corresponding adaptive sigma value during model training. This approach ensures that the training process relies only on pre-established ground-truth information. While manual annotation of 2D keypoints is labor-intensive, this method provides the significant advantage of generating the necessary pose ground-truth data, including distance, without requiring expensive external sensors, thus making the dataset construction process more practical. We begin by computing the distances between the object and the camera for all images in the dataset. Based on these distances, we define a range for sigma values and decide on the number of stages to divide this range. The calculated object distances are then partitioned into the predetermined number of stages (bins), and a specific sigma value is assigned to each distance bin. The sigma values are calculated using the following equations: Where: and are the minimum and maximum sigma values, respectively. and are the minimum and maximum sigma values, respectively. stages is the number of bins the distances are divided into. stages is the number of bins the distances are divided into. i represents the stage index. i represents the stage index. An example of Heatmap with Adaptive Sigma is introduced in Fig. 5. The generated heatmaps using this adaptive sigma can be seamlessly integrated into existing training methods without any additional modifications to the model. By implementing this adaptive sigma method, we enhance the model’s ability to localize keypoints accurately by aligning the heatmap generation process with the varying apparent sizes of objects within the images. Examples of heatmaps generated using adaptive sigma based on object distance: (a) 0.78m distance with 0.85 sigma, (b) 0.47m distance with 1.06 sigma, and (c) 0.11m distance with 1.28 sigma. Heatmap prediction We adopted ViTPose15 as our baseline for heatmap-based pose estimation. As shown in Fig 2a, the ViTPose-based architecture. ViTPose has shown excellent performance in human pose estimation by leveraging a basic ViT structure and simple deconvolution. We applied various state-of-the-art backbones to this structure to identify the optimal one that could meet the specific requirements of our experiments, including parameters, FPS, and other metrics. The decoder consists of two consecutive deconvolution layers, each followed by Batch Normalization and ReLU activation functions. All deconvolution layers have an output channel size of 256, and their kernel sizes vary depending on the backbone used, as shown in Table 1. Specifically, when the kernel size is 4, we set the stride and padding to 2 and 1, respectively; when the kernel size is 8, we set them to 4 and 2. With these configurations, we aim to increase the resolution of the input feature maps by factors of two and four, respectively. Here, is the reshaped output of the Backbone’s embedding vector into an image form. The reshaping form is as follows: where P is the patch size, H and W are the height and width of the input image respectively, is the size of the embedding vector, and is the total number of tokens. Table 1Comparison of different state-of-the-art Backbones on Drone Keypoints Estimation. denotes the model hyperparameter where is set to 512, Depth to 8, to 6, and MLP size to 2048. FPS was measured on Jetson AGX Orin 64GB Development Kit using FP32. Decoder (Kernels) means the size of each kernel in Deconv layer. For instance, an entry of ’8,8’ indicates two Deconv layers, each using an 8x8 kernel filter.BackboneDecoderInputHeatmapParamsFLOPsFPS (Kernels)SizeSize(M)(G)[Jetson]EfficientViT-M188, 822422464649.41.212684.293.791.887.464.0 +Adaptive8, 822422464649.41.2126 86.1 (+1.9) 95.3 93.7 88.7 66.7 PoolFormer-S1198, 4224224565615.62.957590.796.495.894.076.5 +Adaptive8, 4224224565615.62.9575 92.3 (+1.6) 97.4 96.8 95.1 79.7 EfficientNet-B2208, 4256192644817.05.172694.998.1 98.0 96.587.0 +Adaptive8, 4256192644817.05.1726 95.2 (+0.3) 98.2 97.8 96.8 88.0 FastViT-S12218, 4224224646421.52.528591.4 97.1 96.6 95.077.0 +Adaptive8, 4224224646421.52.5285 92.6 (+1.2) 97.096.5 95.3 81.5 ViT-S/16144, 4256192644828.86.429794.198.097.8 96.9 83.5 +Adaptive4, 4256192644828.86.4297 94.0 (-0.1) 98.4 98.1 96.782.9CMT-T228, 4224224565630.43.162293.198.097.6 96.1 80.6 +Adaptive8, 4224224565630.43.1622 94.0 (+0.9) 98.6 98.2 96.0 83.0 CoAtNet-0238, 4224224565630.64.844892.097.897.395.777.0 +Adaptive8, 4224224565630.64.8448 94.2 (+2.2) 98.1 97.7 96.4 84.6 Swin-T248, 4224224565633.55.954489.8 97.4 96.7 93.871.3 +Adaptive8, 4224224565633.55.9544 90.9 (+1.1) 97.296.5 94.4 75.6 MobileOne-S3258, 4256192644834.64.296092.496.495.994.5 82.9 +Adaptive8, 4256192644834.64.2960 91.5 (-0.9) 97.1 96.5 94.7 77.7LiTV2-S pan2022fast8, 4224224565640.75.01-92.597.797.095.280.1 +Adaptive8, 4224224565640.75.01- 95 (+2.5) 98.7 98.2 96.6 86.3 ConvNeXt-V2-T278, 4224224565641.55.915489.894.694.092.678.1 +Adaptive8, 4224224565641.55.9154 92.1 (+2.3) 95.6 95.2 93.9 83.6 MambaVision-T288, 4224224565642.65.814293.3 98.6 98.3 96.979.4 +Adaptive8, 4224224565642.65.8142 95 (+1.7) 98.498.2 97.3 85.9 RegNetY-3.2G298, 4224224565645.25.223594.198.097.696.684.1 +Adaptive8, 4224224565645.25.2235 95.7 (+1.6) 98.6 98.3 97.3 88.5 Comparison of different state-of-the-art Backbones on Drone Keypoints Estimation. denotes the model hyperparameter where is set to 512, Depth to 8, to 6, and MLP size to 2048. FPS was measured on Jetson AGX Orin 64GB Development Kit using FP32. Decoder (Kernels) means the size of each kernel in Deconv layer. For instance, an entry of ’8,8’ indicates two Deconv layers, each using an 8x8 kernel filter. After the first deconvolutional layer, the resolution of the image is doubled: This upsampling process continues through the second deconvolution layer, ultimately producing heatmaps with the desired output resolution for keypoint prediction. Finally, we generate the required class heatmaps through a convolution layer. If the input passes through two deconvolution layers with a kernel size of 4, the final heatmap size is as follows: Keypoint extraction & PnP for pose estimation As shown in Fig. 2b, to extract 2D keypoints from the generated heatmaps for use in the PnP algorithm, we utilize a post-processing step involving an argmax operation followed by an unbiased decoding method. First, the initial keypoint predictions are obtained by finding the pixel location with the highest confidence score in each heatmap channel. This can be expressed as: where represents the predicted 2D coordinates of the k-th keypoint, and denotes the heatmap value for the k-th keypoint at pixel location (x, y). To mitigate potential bias introduced by the argmax operation, we apply an unbiased decoding method proposed by Zhang et al.16. This method applies a Gaussian distribution modulation to the heatmaps before extracting the keypoints, resulting in more accurate predictions. After obtaining the initial keypoint predictions using the argmax operation, the heatmaps undergo Gaussian blurring. This step smooths the heatmap values, which helps in better capturing the true peak of the distribution around the keypoint. The Gaussian blur adjusts the heatmap such that the influence of surrounding pixels is considered, reducing the bias caused by simply selecting the maximum value. This can be expressed as: where represents the smoothed heatmap for the k-th keypoint, is the original heatmap value at pixel location (x, y), and G(x, y) denotes the Gaussian kernel. Subsequently, a Taylor expansion is applied around the peak location identified by the argmax. This mathematical technique refines the keypoint position by considering the gradient and curvature of the heatmap at the peak. This can be expressed as: where is the refined 2D coordinates of the k-th keypoint, is the initial keypoint position obtained from the argmax, is the gradient of the smoothed heatmap at the initial peak location, and is a small adjustment vector based on the gradient and curvature. As shown in Fig. 2c, by incorporating this keypoint extraction process, we obtain accurate 2D keypoint locations from the heatmaps, which are then used in conjunction with the PnP algorithm to estimate the 6D pose of the UAV. Given a set of 2D-3D correspondences, , where represents the predicted 2D projection of the -th 3D point on the UAV, the PnP algorithm aims to find the rotation matrix and translation vector that best project the 3D points onto the corresponding 2D points. This can be formulated as a minimization problem: where represents the camera projection function. By solving this optimization problem, we obtain the 6D pose of the UAV, represented by the rotation and translation . Experiment In this section, we evaluate the effectiveness of our proposed heatmap-based keypoint detection method for UAV 6-DOF pose estimation. Our primary objective is to determine whether accurate keypoint positions can be obtained to establish reliable 2D-3D correspondences, which are essential for solving the PnP problem and, ultimately, determining the UAV’s pose. Another important aspect of our evaluation is to assess the real-time capability of the proposed method, which is critical for its practical application in UAV recovery operations. To evaluate keypoint detection accuracy, we use the Percentage of Correct Keypoints (PCK)17, a metric widely used in human pose estimation. We also aim to verify the impact of incorporating the Adaptive Sigma technique on keypoint detection accuracy and to conduct a qualitative evaluation of the 6D pose obtained through the PnP solution. This evaluation includes assessing the accuracy of pose estimation in challenging scenarios, such as images with drones in highly tilted poses or with motion-induced blur, by plotting the calculated pose. Dataset To train the Heatmap Estimation Model, we created a dataset with Drone Keypoint annotations. We captured drone footage on a building rooftop and labeled keypoints for a total of 8 classes. The dataset comprises four different videos, each shot at different times. The training dataset contains a total of 8,598 images, and the test dataset consists of 1,801 images. The sequences of the video recordings are as follows: The drone approaches the camera in a straight line. The drone approaches the camera in a straight line. The drone ascends away from the camera in a straight line. The drone ascends away from the camera in a straight line. The drone descends towards the camera while shaking. The drone descends towards the camera while shaking. The drone ascends away from the camera while shaking. The drone ascends away from the camera while shaking. The recorded videos were annotated using the Computer Vision Annotation Tool (CVAT). As shown in Fig. 4, the 8 classes are labeled starting from the right front leg of the drone in a counterclockwise direction as classes 1 through 4, and from the right front corner of the inner docking structure in a counterclockwise direction as classes 5 through 8 (Fig. 5). Labeled keypoints of a drone in our dataset: classes 1 to 4 indicate the drone’s legs in a counterclockwise direction, and classes 5 to 8 represent the inner docking structure corners in a counterclockwise sequence. Implementation details We conducted training and testing using our custom dataset. The model was trained on a single RTX 4090 GPU employing the Automatic Mixed Precision (AMP) method. Input images were resized based on the Backbone architecture. We used AdamW as the optimizer with a learning rate of 1e-3 for generalization, and experimentally found that AdamW demonstrated better generalization performance compared to Adam. The model are trained for 1500 epochs, and the best results among the measurements were reported for each epoch. To enhance the model’s generalization performance, Data augmentation techniques were applied. Specifically, ColorJitter and either a 180-degree rotation or a random rotation within ± 30 degrees were utilized to increase the training data. In Table 1, FPS measurements were conducted on a Jetson AGX Orin 64GB Development Kit using FP32 precision to verify whether real-time execution is feasible. To assess the effectiveness of the proposed method, we compare two settings: (i) a baseline with the heatmap sigma fixed at 1, and (ii) the Adaptive Sigma approach with and . We utilized the Efficient Perspective-n-Point (EPnP) algorithm provided by the OpenCV library, and the camera intrinsics were set to the values of the camera used to capture the dataset. The 3D coordinates of the UAV’s keypoints were manually measured and defined in a coordinate system centered at the UAV’s center of gravity, which also serves as the center of the docking area. Evaluation metrics This paper presents a practical approach for 6D pose estimation of automatic recovery UAVs in the absence of target pose information. Due to the difficulty in directly evaluating pose accuracy quantitatively, we present our results through visualizations, showcasing the effectiveness of our approach. Our method utilizes a correspondence-based approach, where the performance of the PnP algorithm heavily relies on the accuracy of the 2D-3D correspondences. Therefore, we quantitatively evaluate our approach by comparing the performance of various model architecture for heatmap-based keypoint extraction on the target drone. To construct the optimal architecture that meets the requirements of the task, we present three key metrics: Percentage of Correct Keypoints (PCK), frames per second (FPS) of the entire pipeline, and model parameter count. PCK, a commonly used metric in human pose estimation as introduced by17, measures the accuracy of keypoint localization. It calculates the proportion of correctly predicted keypoints, where a keypoint is considered correct if the Euclidean distance between the predicted and ground truth keypoint locations is within a predefined threshold. This can be expressed as: Where: N is the total number of keypoints. N is the total number of keypoints. represents the predicted 2D coordinates of the i-th keypoint within an image. represents the predicted 2D coordinates of the i-th keypoint within an image. represents the ground truth 2D coordinates of the i-th keypoint within an image. represents the ground truth 2D coordinates of the i-th keypoint within an image. calculates the Euclidean distance between the predicted and ground truth keypoint locations. calculates the Euclidean distance between the predicted and ground truth keypoint locations. t is the predefined distance threshold. t is the predefined distance threshold. is an indicator function that returns 1 if the condition is true, and 0 otherwise. is an indicator function that returns 1 if the condition is true, and 0 otherwise. By analyzing these metrics, we can comprehensively assess the accuracy, efficiency, and complexity of our proposed method for 6D pose estimation of automatic recovery UAVs. Experiment results and discussion Quantitative results Table 1 presents the quantitative results of our experiments, comparing various state-of-the-art backbone architectures in terms of keypoint detection accuracy, computational efficiency, and model complexity. The PCK metric at different thresholds provides a comprehensive assessment of the models’ ability to localize keypoints accurately. Our results demonstrate that the proposed heatmap-based keypoint detection method effectively identifies the UAV’s feature points across a wide range of backbone architectures. Across all backbones tested, the average PCK scores are consistently high, indicating that the majority of keypoints are localized within acceptable error margins. This high level of accuracy is essential for establishing precise 2D-3D correspondences required for reliable 6D pose estimation through the PnP algorithm. Lightweight models such as EfficientViT-M0 and PoolFormer-S12 achieve average PCK scores of 84.2% and 90.7%, respectively, demonstrating that even models with fewer parameters can effectively detect keypoints. Mid-sized models like FastViT-S12 and ViT-S/16 show higher average PCK scores of 91.4% and 94.1%, respectively. Larger models, including LiTV2-S and RegNetY-3.2G, achieve even higher average PCK scores of 92.5% and 94.1%, respectively. These results indicate that our heatmap-based keypoint detection method is effective across various model sizes and architectures, confirming its robustness and general applicability for localizing drone-specific features. To further enhance keypoint localization accuracy, we introduced the Adaptive Sigma technique in the heatmap generation process. By adjusting the sigma value based on the UAV’s distance from the camera, the model better accounts for variations in the object’s apparent size, leading to improved keypoint detection performance. As shown in Table 1, incorporating Adaptive Sigma results in consistent improvements in PCK across most models. When applying Adaptive Sigma, we observe improvements in average PCK scores across different groups of backbones. Lightweight models, such as EfficientViT-M0 and PoolFormer-S12, see improvements of 1.9% and 1.6%, respectively. Mid-sized models like FastViT-S12 and CoAtNet-0 exhibit improvements of 1.2% and 2.2%, respectively. Larger models, including LiTV2-S and RegNetY-3.2G, show significant improvements of 2.5% and 1.6%, respectively. Notably, even for metrics demanding high precision, such as PCK@0.0025, using the Adaptive Sigma approach yields substantial performance gains. These enhancements indicate that the Adaptive Sigma method effectively enhances the model’s ability to localize keypoints accurately across different architectures. Furthermore, we assessed the real-time capability of our method to ensure practical usability in UAV recovery operations. The FPS measurements in Table 1 demonstrate that our models maintain high computational efficiency, with several models achieving over 50 FPS on the Jetson AGX Orin platform. This confirms that our approach is suitable for real-time applications, balancing both accuracy and efficiency. We excluded the FPS measurement for the LiTV2-S backbone because its PyTorch implementation did not function properly in the Jetson environment. Additionally, the ViT backbone exhibits higher FPS compared to its FLOPs, which we attribute to its design being optimized for efficient computation within PyTorch. To assess the statistical significance of the observed improvements, we performed a one-tailed paired t-test on the per-image PCK scores. Figure 6 visualizes the comparison at the strictest threshold (PCK@0.0025) and summarizes the statistical significance of the improvements. Statistical significance of PCK improvements with adaptive sigma. The bar chart compares the macro-average PCK@0.0025 between the Fixed baseline and our Adaptive method for each backbone model. Error bars represent the standard error of the mean across all images in the validation set, with a typical magnitude of approximately 0.008. Model names marked with (**) indicate that the improvement from Adaptive is highly statistically significant (p < 0.005), as determined by a one-tailed paired t-test. As shown in the Fig. 6, our Adaptive Sigma method leads to a statistically significant performance increase for the vast majority of the tested architectures. Models spanning various families, from the lightweight EfficientViT-M0 to the larger LiTV2-S, demonstrate highly significant gains (marked with **, p < 0.005). This provides strong evidence that our proposed method effectively enhances the precision of keypoint localization. Interestingly, the models that did not show significant improvement, such as the pure ViT, are those with architectural properties that may not fully align with our scale-aware learning targets, as we will discuss further in the ablation study. Conversely, models that did not exhibit significant improvements, such as ViT-S/16 and MobileOne-S3, possess distinct architectural properties that may limit the benefits of the Adaptive Sigma mechanism. Unlike hybrid transformer backbones, ViT-S/16 is a pure transformer architecture that relies solely on global self-attention from its earliest layers, thereby lacking strong local inductive biases. We hypothesize that, in such architectures, the dynamically varying precision of the heatmap targets may interfere with the model’s ability to establish spatial correspondences from scratch. In the case of MobileOne-S3, which employs a highly sensitive re-parameterization scheme, the non-stationary nature of the learning targets could hinder the stable convergence required for effective branch fusion, ultimately resulting in performance degradation. To better evaluate the effectiveness of Adaptive Sigma across different distances, we conducted an ablation study by dividing the validation set into four distance bins. We selected three representative backbones-PoolFormer-S12 (15.6M), FastViT-S12 (21.5M), and LiTV2-S (40.7M)-which represent small, medium, and large parameter scales, respectively. This setup allows us to examine how adaptive target scaling behaves across architectures of different capacities. As shown in Table 2, the results show a clear and consistent trend. Adaptive Sigma brings the largest improvements at the extreme distance ranges, where using a fixed sigma becomes less effective. For example, LiTV2-S achieves gains of +10.06% in the farthest bin and an even larger +32.46% in the nearest bin. Likewise, PoolFormer-S12 improves by +20.86% in far-distance cases. Although the mid-distance bins show smaller changes, the strong improvements at both ends demonstrate the robustness and adaptability of our method. These results indicate that adjusting sigma dynamically helps maintain accurate localization across different scales-especially under the challenging conditions of UAV recovery scenarios. Table 2Ablation study comparing Fixed and Adaptive performance (PCK@0.0025) across different distance bins for representative backbones. The value in parentheses indicates the percentage point change from the baseline. The distribution of the validation dataset across these bins is detailed on the right.BackboneMethodFar BinMid-Far BinMid-Near BinNear Bin()()()()PoolFormer-S12Fixed Sigma71.1585.8358.5238.25+Adaptive85.99 (+20.86%)84.49 (-1.56%)58.13 (-0.67%)38.23 (-0.05%)FastViT-S12Fixed Sigma83.4181.4955.9733.32+Adaptive88.21 (+5.75%)83.39 (+2.33%)56.74 (+1.38%)40.29 (+20.92%)LiTV2-SFixed Sigma83.2887.5557.5729.17+Adaptive91.66 (+10.06%)91.85 (+4.91%)66.43 (+15.39%)38.64 (+32.46%)BinSamples (%)Far406 (22.5%)Mid-Far698 (38.8%)Mid-Near351 (19.5%)Near346 (19.2%)Total1801 Ablation study comparing Fixed and Adaptive performance (PCK@0.0025) across different distance bins for representative backbones. The value in parentheses indicates the percentage point change from the baseline. The distribution of the validation dataset across these bins is detailed on the right. Qualitative results While quantitative metrics such as PCK provide valuable insights into the accuracy of keypoint localization, it is essential to qualitatively evaluate the overall effectiveness of our proposed 6D pose estimation method in practical scenarios. To this end, we present qualitative results that demonstrate the capability of our method to accurately estimate the 6D pose of the UAV under various challenging conditions. We selected three backbone networks-MambaVision, FastViT-S12, and LiTV2-S-based on a balanced consideration of performance, model size, computational efficiency, and FPS in our quantitative evaluations We evaluated our method in several challenging scenarios: UAV at varying proximities to the camera: The UAV appears at different scales within the image due to changes in proximity, which affects the resolution and detail of the UAV’s features, making keypoint detection more challenging. To simulate these varying scales, the UAV was positioned at diverse altitudes. UAV at varying proximities to the camera: The UAV appears at different scales within the image due to changes in proximity, which affects the resolution and detail of the UAV’s features, making keypoint detection more challenging. To simulate these varying scales, the UAV was positioned at diverse altitudes. UAV with significant tilt: The UAV may be significantly tilted relative to the camera, resulting in foreshortening and occlusion of certain keypoints. UAV with significant tilt: The UAV may be significantly tilted relative to the camera, resulting in foreshortening and occlusion of certain keypoints. Motion blur due to UAV movement: Rapid movement of the UAV can cause motion blur in the images, potentially degrading the quality of keypoint detection. Motion blur due to UAV movement: Rapid movement of the UAV can cause motion blur in the images, potentially degrading the quality of keypoint detection. Figures 7 and 8 illustrate the qualitative results for these scenarios using the MambaVision, FastViT-S12, and LiTV2-S backbones, respectively. In each figure, we overlay the projected 3D model of the UAV onto the input image using the estimated 6D pose parameters. The alignment between the projected model and the actual UAV in the image serves as an indicator of the pose estimation accuracy. Qualitative comparison of 6D pose estimation results using the MambaVision and FastViT-S12 backbone, highlighting the improvements achieved by applying the Adaptive Sigma technique. Qualitative comparison of 6D Pose Estimation results using the LiTV2-S backbone, highlighting the improvements achieved by applying the Adaptive Sigma technique. In the varying proximities to the camera scenario, our method successfully estimates the 6D pose of the UAV across different scales within the image due to changes in distance. The Adaptive Sigma technique adjusts the sigma value in the heatmap generation process to account for the UAV’s varying apparent size resulting from its proximity to the camera. This adjustment allows for accurate keypoint localization regardless of distance, leading to precise alignment of the projected 3D model with the UAV in the image. In the case of significant tilt, the UAV presents unusual angles and potential occlusions. Our method demonstrates robustness in handling such challenging orientations. The keypoints are accurately detected even when some parts of the UAV are less visible, and the estimated 6D pose accurately reflects the UAV’s orientation. For the motion blur scenario, where the UAV’s rapid movement causes blur, our method maintains reliable keypoint detection and pose estimation. Despite the degraded image quality, the model can infer the keypoint positions effectively. The use of heatmap-based keypoint detection proves advantageous in such situations, as it allows the model to consider probabilistic distributions over keypoint locations, rather than relying on precise pixel values. In the representative scenarios and with the selected backbone networks, the qualitative results confirm that our proposed method provides accurate 6D pose estimations. By focusing on specific cases-such as the UAV at various altitude, with significant tilt, or experiencing motion blur-we demonstrate that accurate 2D-3D correspondences have been established, enabling the PnP algorithm to compute reliable pose parameters. Both methods produce reasonable pose estimates without large deviations, which can be attributed to the robustness of the EPnP solver that yields a valid pose as long as the keypoints are reasonably accurate. However, by applying the Adaptive Sigma technique, the accuracy of keypoint localization is further improved, resulting in a more precise and stable 6D pose. This improvement is also quantitatively verified by the reduction in the mean difference Diff(mean) values within the images, indicating more accurate 2D-3D correspondences and, consequently, higher overall pose estimation precision. Future works Our proposed method has demonstrated effectiveness in real-time 6D pose estimation for UAV recovery operations, successfully utilizing heatmap-based keypoint detection and the PnP algorithm to obtain accurate UAV localization. This approach provides a practical solution that balances accuracy and computational efficiency, making it suitable for real-world applications where timely and precise pose estimation is essential. However, despite these strengths, there are inherent limitations associated with the reliance on keypoint detection and the PnP solver. The PnP algorithm requires precise 2D-3D correspondences to compute an accurate 6D pose, necessitating the extraction of highly accurate keypoints. The creation of training datasets and the manual labeling of keypoints is a time-consuming and labor-intensive process, which can limit the scalability and applicability of the method to different UAV models or scenarios. Furthermore, in situations involving occlusion or extreme viewing angles, some keypoints may not be detectable, leading to less reliable 2D-3D correspondences and potentially inaccurate pose estimations. These challenges suggest that moving towards methods that can directly predict the 6D pose without relying solely on keypoint detection could enhance robustness and accuracy. Future work will explore end-to-end approaches for 6D pose estimation that can directly regress pose parameters from image data. Leveraging deep learning techniques that integrate feature extraction and pose estimation into a unified framework may alleviate the dependency on manual keypoint labeling and improve performance under challenging conditions such as occlusions. Additionally, to address the dataset creation burden, we plan to this challenge by leveraging synthetic data augmentation and domain adaptation techniques. Recent advances in Human Pose Estimation (HPE) have shown that training with synthetic data and employing domain adaptation can improve model generalization to real-world data30. Utilizing such synthetic datasets would also overcome a key limitation of the current study: the absence of direct pose ground truth. Simulation environments can provide precise 6D pose labels, enabling a more direct and rigorous evaluation of pose accuracy using standard metrics like ADD(-S). In future work, we plan to construct a large-scale synthetic dataset within a high-fidelity simulation environment to enable cross-model validation across different UAV types. By leveraging these simulation-based trainings, we aim to achieve robust sim-to-real transfer and ensure the proposed framework’s applicability to diverse real-world scenarios. Conclusion We proposed a practical approach for real-time 6-DOF pose estimation of UAVs using monocular RGB imagery, specifically designed for the near-range docking phase in autonomous UAV recovery systems. By introducing a novel Adaptive Sigma technique for heatmap-based keypoint detection, we enhanced localization accuracy according to the UAV’s distance from the camera. By establishing accurate 2D–3D correspondences and solving for the pose through the PnP algorithm, the proposed method achieved high Percentage of Correct Keypoints (PCK) scores and demonstrated robustness under challenging conditions such as varying distances, significant UAV tilts, and motion blur. This study serves as a foundational module for high-precision, near-range localization in UAV recovery scenarios. Extensive experiments with multiple state-of-the-art backbone architectures verified both the accuracy and real-time capability of the proposed approach. Nevertheless, realizing a fully autonomous recovery pipeline will require integrating this fine-grained localization with long-range detection and mid-range tracking modules. While acknowledging limitations related to the reliance on keypoint detection and manual dataset creation, our work provides useful insights for future research. Future efforts should focus on developing a unified framework that seamlessly integrates precise pose estimation with hierarchical tracking and control components. In addition, exploring direct 6-DOF pose regression and leveraging synthetic data for training could further enhance robustness and generalization across diverse UAV platforms and real-world environments. Author contributions A.J.C. conceived the original idea and supervised the project. M.J. designed and conducted the experiments, developed the methodology, and analyzed the results. M.J. wrote the initial draft of the manuscript. Both authors discussed the results and contributed to the final manuscript. Funding This work was supported by the Gachon University research fund of 2024(GCU-202500670001) Data availability The datasets generated and/or analysed during the current study are available in the ph-10 huggingface repository at https://huggingface.co/datasets/jaezic/ph-10 Declarations Competing interests The authors declare no competing interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References Associated Data Data Availability Statement The datasets generated and/or analysed during the current study are available in the ph-10 huggingface repository at https://huggingface.co/datasets/jaezic/ph-10