Ensuring Calibration Robustness in Split Conformal Prediction Under Adversarial Attacks
Xunlei Qian Yue Xing Michigan State University · 2025
Read Ensuring Calibration Robustness in Split Conformal Prediction Under Adversarial Attacks by Xunlei Qian Yue Xing Michigan State University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-18562.txt.
Introduction
Xunlei Qian and Yue Xing test how split conformal prediction holds up when inputs are attacked. Conformal methods give coverage pledges with no distribution claims, but they rest on data being exchangeable — a state attacks break. The team study both pledge validity and set size when test inputs face adversarial tweaks.
Theory ties pledge shifts to the attack strength used at calibration time. Coverage moves in a steady, one way fashion with that strength, so a set nonzero attack at calibration can tune coverage under attack at test. Model training with attacks gets the same check. Tests back the theory, and target coverage can hold across a range of test attack strengths. The work gives knobs for sound sets under attack.
Catalog record
| CREATOR | Xunlei Qian Yue Xing Michigan State University |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 107 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-18562
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Ensuring Calibration Robustness in Split Conformal Prediction Under Adversarial Attacks Xunlei Qian Yue Xing Michigan State University Abstract Conformal prediction (CP) provides distribution-free, finite-sample coverage guarantees but critically relies on exchangeability, a condition often violated under distribution shift. We study the robustness of split conformal prediction under adversarial perturbations at test time, focusing on both coverage validity and the resulting prediction set size. Our theoretical analysis characterizes how the strength of adversarial perturbations during calibration affects coverage guarantees under adversarial test conditions. We further examine the impact of adversarial training at the model-training stage. Extensive experiments support our theory: (i) Prediction coverage varies monotonically with the calibration-time attack strength, enabling the use of nonzero calibration-time attack to predictably control coverage under adversarial tests; (ii) target coverage can hold over a range of test-time attacks: with a suitable calibration attack, coverage stays within any chosen tolerance band across a contiguous set of perturbation levels; and (iii) adversarial training at the training stage produces tighter prediction sets that retain high informativeness. 1 Introduction With the rapid adoption of deep learning in high-stakes domains, such as survival analysis ( Candes et al., 2023 ) and chest X-ray report generation ( Gui et al., 2024 ) , uncertainty quantification has become an increasingly intricate and essential task for ensuring reliable predictive performance. Conformal prediction (CP) ( Vovk et al., 2005 ) offers a powerful, distribution-free framework that provides finite-sample validity by constructing prediction sets calibrated to a user-specified coverage level. For example, CP can support clinical decision workflows in which physicians must deploy complex models for diagnosis ( Banerji et al., 2023 ; Kiyani et al., 2025 ; Hullman et al., 2025 ) . Despite its flexibility, CP typically relies on independent and identically distributed (i.i.d.) or, more generally, at least exchangeable data. These assumptions are often violated in practice ( Barber et al., 2023 ) . When exchangeability is broken, the conformity-score distribution in the calibration set need not match that of the test data due to distribution shift commonly observed in real-world settings ( Moreno-Torres et al., 2012 ; Sugiyama et al., 2007 ; Koh et al., 2021 ) A large body of work proposes mitigation strategies within the classical CP paradigm: some methods address random perturbations at test time ( Gendler et al., 2022 ; Yan et al., 2024 ) , while others demonstrate that adversarial perturbations can be particularly harmful for deep models ( Kos and Song, 2017 ) . These observations motivate the study of CP under non-exchangeable conditions. On the other hand, Adversarial attack is a prominent form of distribution shift in which inputs are deliberately perturbed in worst-case directions to degrade performance ( Biggio et al., 2013 ; Goodfellow et al., 2014 ) . Although such perturbations are often imperceptible to humans ( Wang et al., 2019 ) , the deep neural network or even transformer-based models like ViT(vanilla vision transformer) can suffer substantial drops in performance compared with unperturbed inputs ( Shao et al., 2021 ; Aldahdooh et al., 2021 ; Narodytska and Kasiviswanathan, 2017 ; Miller et al., 2020 ) . Prior work spans attack generation ( Xiao et al., 2018 ; Assion et al., 2019 ) , adversarial training ( Shafahi et al., 2019b ; Andriushchenko and Flammarion, 2020 ; Shafahi et al., 2020 ) , and adversarial robustness Bai et al., 2021 ; Silva and Najafirad, 2020 . In this paper, we examine test-time adversarial perturbations through the lens of conformal prediction. We analyze how such attacks affect validity (coverage) and efficiency (prediction set size), how calibration under adversarial perturbations can improve robustness at the test time, and how adversarial training will improve the efficiency of split conformal prediction. Our contributions are as follows: • We provide a theoretical analysis of how the strength of adversarial perturbations applied to the calibration set affects prediction validity on both clean and adversarially perturbed test data. In particular, we show that test-time prediction accuracy is a monotone (non-decreasing) function of the calibration attack strength (Theorem 1 ). • We show that introducing a proper adversarial perturbation during calibration yields more robust predictions under potential test-time attacks. Specifically, for a 90% target accuracy, CP attains a “reasonable” accuracy (e.g., 87%–93%) across a wider range of test-time attack strengths (Theorem 2 ). • We show that adversarial training can reduce prediction set size for Split conformal prediction, thereby improving both robustness and informativeness for deep neural networks (Theorem 3 ). • Experiments are conducted to verify the observations from the above theorems. 2 Related Work 2.1 Adversarial training Adversarial examples have become central to the study of machine learning safety because of their implications for model robustness ( Goodfellow et al., 2014 ) . A growing body of research has explored the use of adversarial training as a primary defense. Despite substantial progress, modern models remain vulnerable to small, often imperceptible, perturbations in both regression and classification tasks. Recent theoretical and empirical results clarify a fundamental robustness–accuracy tradeoff in adversarially trained models. In linear settings, where analysis is more tractable, adversarial training improves robustness at the expense of standard accuracy. ( Javanmard et al., 2020 ) precisely characterize this tradeoff and establish limits on the accuracy achievable by any algorithm. More recently, Xing et al. ( Xing, 2023 ) showed that two-stage training with additional unlabeled data and pseudolabeling can enhance robustness even in two-layer neural networks. Taken together, these findings highlight the particular effectiveness of adversarial training for simple models, while also offering insights for extending such defenses to richer architectures. Other related literature can be found in ( Goodfellow et al., 2015 ; Madry et al., 2018 ; Zhang et al., 2019 ; Shafahi et al., 2019a ; Carmon et al., 2019 ; Xie et al., 2020 ) . 2.2 Adversarially robust conformal prediction One critical high-stakes task in machine learning is constructing reliable prediction sets for classification tasks in the presence of adversarially perturbed test data. Recent work proposes several approaches toward this goal. ( Gendler et al., 2022 ) demonstrated that standard CP fails to provide valid coverage under adversarial perturbations bounded in \ell_{2} norm and introduce Randomized Smoothed Conformal Prediction (RSCP), which combines random smoothing with a modified thresholding rule to improve robustness. Extending this line, ( Yan et al., 2024 ) proposed two approaches, Post-Training Transformation (PTT) and Robust Conformal Training (RCT), that significantly reduce the size of prediction sets while maintaining valid coverage under adversarial conditions. More recently, ( Scholten and Günnemann, 2025 ) aggregate predictions from multiple models trained on partitioned datasets, yielding provable robustness to both test-time adversarial attacks and training-time data poisoning. Complementary developments include probabilistic robustness via quantile-of-quantiles, adversarially valid guarantees for sequential settings, certifiable reasoning with probabilistic circuits, Lipschitz-bounded models for scalable certification, and conformal training/attack frameworks that directly optimize set size under threat ( Ghosh et al., 2023 ; Bastani et al., 2022 ; Areces et al., 2025 ; Kang et al., 2024 ; Massena et al., 2025 ; Bao et al., 2025 ) . Collectively, these contributions advance provably robust conformal prediction in adversarial scenarios by addressing both validity and efficiency. 3 Preliminaries The following section provides the general framework of split conformal prediction