Robust Vision-Based Runway Detection through Conformal Prediction and Conformal mAP
Alya Zouzou, Léo Andéol, Mélanie Ducoffe, Ryma Boumazouza · 2025
Read Robust Vision-Based Runway Detection through Conformal Prediction and Conformal mAP by Alya Zouzou, Léo Andéol, Mélanie Ducoffe, Ryma Boumazouza (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16740.txt.
Introduction
Alya Zouzou and co-authors add firm error bounds to runway spotting for landing aircraft, in this 2025 paper. Vision systems must find the runway from the air, and a wrong guess can be unsafe. The team fine-tuned two common object-spotting models on aerial photos.
They then applied a method that turns model scores into bounds with a chosen risk level. They also propose a new score that blends spotting quality with those bounds. Results show the bounds make the reliability of each guess clearer in a sound way. The authors argue this kind of proof will help when learning systems seek approval for use in flight.
Catalog record
| CREATOR | Alya Zouzou, Léo Andéol, Mélanie Ducoffe, Ryma Boumazouza |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-16740
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Robust Vision-Based Runway Detection through Conformal Prediction and Conformal mAP Alya Zouzou, Léo Andéol, Mélanie Ducoffe, Ryma Boumazouza Abstract We explore the use of conformal prediction to provide statistical uncertainty guarantees for runway detection in vision-based landing systems (VLS). Using fine-tuned YOLOv5 and YOLOv6 models on aerial imagery, we apply conformal prediction to quantify localization reliability under user-defined risk levels. We also introduce Conformal mean Average Precision (C-mAP), a novel metric aligning object detection performance with conformal guarantees. Our results show that conformal prediction can improve the reliability of runway detection by quantifying uncertainty in a statistically sound way, increasing safety on-board and paving the way for certification of ML system in the aerospace domain. † † volume: 266 † † year: 2025 † † workshop: Conformal and Probabilistic Prediction with Applications † † proceedings: PMLR: Proceedings of Machine Learning Research keywords Conformal Prediction, Average Precision, Object Detection, Aeronautics 1 Introduction The emergence of Machine Learning (ML), particularly deep learning and neural network (NN) models, has enabled transformative progress across a wide range of domains including transportation, healthcare, and finance. However, despite their impressive performance, neural networks exhibit inherent vulnerabilities, most notably their sensitivity to adversarial inputs and data distribution shifts, as highlighted in foundational work such as Szegedy et al. (2014) . These weaknesses raise critical concerns when deploying ML systems in safety-critical contexts, where guarantees on reliability, robustness, and uncertainty quantification are paramount. One of the most stringent and high-stakes domains for ML deployment is aerospace. Within this field, vision-based landing systems (VLS) have emerged as a promising application of ML to enhance or potentially replace traditional landing aids. Aircraft landing is among the most complex and risk-prone phases of flight, currently supported by redundant human and technical systems—two-pilot operations, standardized communication protocols (e.g., ATIS), and ground-based navigation aids such as the Instrument Landing System (ILS) and GPS. As the industry moves toward single-pilot operations and increasing automation, ML-based systems must be able to provide reliable, interpretable, and certifiable support for tasks like visual runway detection and localization. We explore recent developments in conformal prediction as a tool for uncertainty quantification (UQ) in object detection models applied to aerial imagery. These tools offer statistical guarantees on localization coverage and can help satisfy regulatory demands regarding AI in aerospace. Specifically, we focus on the core task of runway detection, which serves as a critical anchor for downstream aircraft pose estimation. Accurate detection of the runway is foundational—errors at this stage propagate through the pipeline and could compromise the entire landing guidance system. Our contributions are as follows: • We apply and evaluate conformal prediction with different margins over boxes (e.g., additive or multiplicative penalties) on multiple object detection architectures for the task of runway detection in aerial images. • We introduce a novel evaluation metric, the conformal mean Average Precision (C-mAP), which quantifies the trade-off between accurate bounding box predictions and their coverage of ground-truth boxes at user-defined risk levels—thus aligning conformal prediction with established object detection benchmarks. • We release our codebase 1 1 1 https://github.com/alyasltd/conformal_runway_detection , trained models, and experimental protocols as an open-source repository to encourage reproducibility and further research in this critical area. 2 Object Detection Overview Object detection refers to the task of identifying and localizing objects within an image, along with classifying them into predefined categories. Recent works in object detection show a great variety of models, mostly performed by neural networks, which can be categorized into two main types: two-stage detectors and single-stage detectors . • Two-stage detectors first identify Regions of Interest (RoIs) in the image, and then classify those regions as containing or not objects (R-CNN, Mask R-CNN, Bharati and Pramanik (2020) ). • Single-stage detectors , such as the YOLO (You Only Look Once, Terven et al. (2023) ) family, are known for their efficiency and trade-off between speed and accuracy. They directly predict bounding boxes and class probabilities for each object in a single pass through the network. Other examples of single-stage detectors include DETR and FCOS Carion et al. (2020) ; Tian et al. (2020) . The goal of object detection can be formally defined as follows: given an image containing potential objects from a target set of classes, the object detector must localize each object and predict its class label. Localization is achieved through the use of bounding boxes. Each predicted bounding box is associated with the following information: • Localization vector : a vector denoted as \mathbb{\hat{b}}=[x_{min},y_{min},x_{max},y_{max}] , which are typically represented by their top-left and bottom-right coordinates. • Objectness score o : A probability value between 0 and 1, representing the model’s confidence that an object exists within the bounding box, regardless of its class label. • Class probability vector \mathbf{p} : A vector containing class probabilities, where the highest value indicates the predicted class of the object, provided that the objectness score exceeds a certain threshold (indicating that the object exists). From these two last sources of information, we compute the confidence score , defined as the product of the objectness score and the class probability. Although this score is sometimes used as a proxy for uncertainty estimation Wenkel et al. (2021) , it lacks reliability—often over- or underestimating the true uncertainty—and provides no statistical guarantees. In practice, object detectors can produce thousands of bounding boxes for a single image, many of which may significantly overlap. This overlap is typically quantified using the Intersection over Union (IoU) metric, defined as the area of the intersection divided by the area of the union of the predicted and ground truth bounding boxes: \text{IoU}=\frac{\text{Area of Intersection}}{\text{Area of Union}} To handle such redundancy, a post-processing step is applied, most commonly the Non-Maximum Suppression (NMS, Neubeck and Van Gool (2006) ) algorithm, which removes overlapping bounding boxes. A second step consists in filtering out insufficiently confident boxes of the final sequence of prediction boxes. We define an object detection model f as the composition of the object detector (after NMS) and confidence threshold. Formally, we start with a model f^{\mathrm{NMS}} taking as input an image x and returning a sequence of fixed length N . These predictions are then filtered for (when two boxes shared an IoU over t , the least confident is disregarded). This results in a sequence of predictions of variable length N^{\mathrm{NMS}}_{x} formally denoted by f^{\mathrm{NMS}}:x\rightarrow\left(\mathbf{\hat{b}}_{i},o_{i},\mathbf{p}_{i}\right)_{i=1}^{N^{\mathrm{NMS}}_{x}}, where \mathbf{\hat{b}}_{i} is the bounding box for the i -th prediction, o_{i} is the corresponding objectness score, \mathbf{p}_{i} is the vector of class probabilities. Finally, all predictions with objectness scores above a threshold \tau are preserved, I(x)=\left\{i\in\{1,\dots,N^{\mathrm{NMS}}_{x}\}:\ o_{[i]}\geq\tau\right\}, where o_{[i]} is the i -th largest objectness score in f(x) . We then obtain the final predictions f:x\rightarrow\left\{f^{\mathrm{NMS}}(x)_{[k]},\ k\in I(x)\right\}. The threshold \tau is a predefined value from the filtering stage used to discard predictions with low objectness scores, ensuring that only detections with sufficiently high objectness score are considered. 2.1 Model Evaluation: Mean Average Precision (mAP) The performance of object detection models is commonly evaluated using the mean Average Precision (mAP) metric, which is the mean of the Average Precision (AP) values computed across all object classes. Consider the set of predicted bounding boxes \{\mathbf{\hat{b}}^{x}_{i}\} output by the object detector, and the ground truth annotations be \{(\mathbf{b}^{x}_{j},c^{x}_{j})\}_{j} ,