Aerial Image Classification in Scarce and Unconstrained Environments via Conformal Prediction
Farhad Pourkamali-Anaraki Department of Mathematical and Statistical Sciences, University of Colorado Denver, CO, USA Email: [email protected] · 2025
Read Aerial Image Classification in Scarce and Unconstrained Environments via Conformal Prediction by Farhad Pourkamali-Anaraki Department of Mathematical and Statistical Sciences, University of Colorado Denver, CO, USA Email: [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-17655.txt.
Introduction
Farhad Pourkamali-Anaraki, writing alone, studies how to label aerial images when data are scarce, in a 2025 paper. The images show varied events in open, uncontrolled settings, which makes the task hard. The method used is conformal prediction, which turns a classifier's output into a set of likely labels with a stated guarantee.
Three pretrained image models were tuned on small labeled samples, and two pipelines were compared, with and without a calibration step called temperature scaling. Even with few labels and simple scores, the method gave useful estimates of uncertainty. Scaling did not always shrink the label sets. The paper also notes that smaller models may suit field use where computing power is limited.
Catalog record
| CREATOR | Farhad Pourkamali-Anaraki Department of Mathematical and Statistical Sciences, University of Colorado Denver, CO, USA Email: [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-17655
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Aerial Image Classification in Scarce and Unconstrained Environments via Conformal Prediction Farhad Pourkamali-Anaraki Department of Mathematical and Statistical Sciences, University of Colorado Denver, CO, USA Email: [email protected] Abstract This paper presents a comprehensive empirical analysis of conformal prediction methods on a challenging aerial image dataset featuring diverse events in unconstrained environments. Conformal prediction is a powerful post-hoc technique that takes the output of any classifier and transforms it into a set of likely labels, providing a statistical guarantee on the coverage of the true label. Unlike evaluations on standard benchmarks, our study addresses the complexities of data-scarce and highly variable real-world settings. We investigate the effectiveness of leveraging pretrained models (MobileNet, DenseNet, and ResNet), fine-tuned with limited labeled data, to generate informative prediction sets. To further evaluate the impact of calibration, we consider two parallel pipelines (with and without temperature scaling) and assess performance using two key metrics: empirical coverage and average prediction set size. This setup allows us to systematically examine how calibration choices influence the trade-off between reliability and efficiency. Our findings demonstrate that even with relatively small labeled samples and simple nonconformity scores, conformal prediction can yield valuable uncertainty estimates for complex tasks. Moreover, our analysis reveals that while temperature scaling is often employed for calibration, it does not consistently lead to smaller prediction sets, underscoring the importance of careful consideration in its application. Furthermore, our results highlight the significant potential of model compression techniques within the conformal prediction pipeline for deployment in resource-constrained environments. Based on our observations, we advocate for future research to delve into the impact of noisy or ambiguous labels on conformal prediction performance and to explore effective model reduction strategies. 1 Introduction Although deep classifiers output softmax scores for training with cross-entropy loss, their primary goal at test time is to predict the single most likely class [ 37 , 28 , 30 ] . Consequently, their single-label predictions lack direct uncertainty quantification. Yet, in many real-world applications, understanding a model’s confidence is as important as the prediction itself [ 10 , 1 ] . Conformal prediction offers a simple yet effective post-hoc solution to generate prediction sets from any trained classifier using calibration data [ 35 , 2 , 7 , 18 , 23 ] . These prediction sets—collections of labels guaranteed to contain the true label with a user-specified coverage level (e.g., 90%)—provide critical information about model confidence, as smaller sets indicate higher certainty, assuming the coverage guarantee is maintained. Thus, conformal prediction transforms point predictions into confidence-informed decisions, facilitating safer deployment of machine learning models in high-stakes scenarios [ 50 , 36 , 29 , 5 ] . A key concept in conformal prediction is defining the nonconformity score function, which quantifies how “atypical” a given relationship between an input and its corresponding output is [ 48 , 47 , 19 , 39 ] . Lower scores suggest higher confidence (typical examples), while higher scores indicate unusual examples. Thus, to apply conformal prediction, a calibration step follows training, where nonconformity scores are computed on a separate held-out calibration dataset. For a user-specified coverage level of (1-\alpha) , where \alpha is a chosen level of tolerance to error (e.g., \alpha=0.1 ), a threshold is determined by calculating the (1-\alpha) -quantile of these calibration scores. At test time, for a new input, a prediction set is formed by considering labels that are typical for the input based on the nonconformity measure and including all classes whose nonconformity scores are below or equal to this threshold. This calibration process relies on the assumption of exchangeability [ 25 , 8 ] , meaning the calibration and test examples are assumed to be drawn from the same underlying distribution and are permutable without changing their joint distribution. Therefore, the effectiveness of conformal prediction heavily depends on the choice of the nonconformity score function, which captures crucial information about the underlying model and the data. If the score function does not meaningfully measure atypicality or uncertainty, the resulting prediction sets will be uninformative. For instance, if the scores are random noise unrelated to the model’s confidence, the prediction sets will simply contain random subsets of possible labels, large enough to achieve the desired marginal coverage but offering no useful insight. Thus, selecting a representative and meaningful score function is essential. Since deep classifiers naturally convert the raw output scores (logits) into a probability distribution over the classes (softmax scores), a common choice of the nonconformity score function for an input-output pair is one minus the softmax score assigned to the true class (known as the Least Ambiguous set-valued Classifier or LAC [ 32 ] ). When the model assigns a softmax score of exactly one to the true class, the nonconformity score becomes zero, indicating perfect conformity and high confidence. Conversely, if the softmax probability assigned to the true class is near zero, the nonconformity score becomes large, signaling that this input-output pair is atypical and does not conform to the learned patterns of the underlying model and data. To achieve a desirable balance between coverage guarantees and the practical usability of prediction sets, algorithmic efforts have been directed towards designing improved score functions. For example, alternative approaches, including Adaptive Prediction Sets (APS) [ 31 ] and Regularized Adaptive Prediction Sets (RAPS) [ 3 ] , consider cumulative sorted softmax probabilities rather than focusing exclusively on the probability of the true class. APS constructs prediction sets by accumulating the most probable classes until a cumulative probability threshold is reached. This adaptive method results in dynamically sized prediction sets that reflect the model’s overall confidence across multiple classes. RAPS further enhances APS by introducing a regularization penalty that discourages excessively large prediction sets. Beyond algorithmic improvements, the urgent need for reliable prediction sets in high-stakes applications highlights the importance of thorough and realistic empirical evaluations of conformal prediction methods. Although numerous studies have empirically assessed these methods, most evaluations rely heavily on standard benchmark datasets such as MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100 (e.g., see [ 12 , 46 , 49 ] ). These datasets are large-scale and extensively preprocessed, and thus fail to fully capture the complexity and variability encountered in practical scenarios. Real-world image classification problems typically involve fewer data points—often far fewer than the tens of thousands available in standard benchmarks—and include a broader, more diverse range of classes, environmental conditions, and data-quality issues [ 41 , 20 ] . Consequently, there is a need for new empirical studies utilizing datasets that better represent real-world conditions. This work makes a significant contribution through a two-pronged approach. Empirically, we leverage the recently introduced ERA (Event Recognition in Aerial Videos) dataset [ 24 ] , a meticulously human-annotated dataset of 2,864 unconstrained aerial videos made available by the remote sensing community. ERA features labels for 25 distinct dynamic event classes, each spanning a 5-second duration, and was designed with substantial intra-class variation and inter-class similarity to reflect the complexity of real-world aerial video data captured across diverse environments and scales. Recognizing the real-world difficulties of acquiring labeled aerial data in diverse and unconstrained settings (e.g., as acquired by drones), this dataset provides an excellent testbed for evaluating conformal prediction methods. Building upon the availability of the