Get Global Guarantees: On the Probabilistic Nature of Perturbation Robustness Conference: Proceedings of the 34th ACM International Conference on Information and Knowledge Management; November 10–14, 2025; Seoul, Republic of Korea Proceedings of the 34th ACM International Conference on Information a
Wenchuan Mu, Kwan Hui Lim · 2025
Read Get Global Guarantees: On the Probabilistic Nature of Perturbation Robustness Conference: Proceedings of the 34th ACM International Conference on Information and Knowledge Management; November 10–14, 2025; Seoul, Republic of Korea Proceedings of the 34th ACM International Conference on Information a by Wenchuan Mu, Kwan Hui Lim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-19183.txt.
Introduction
Wenchuan Mu and Kwan Hui Lim propose a testing-based yardstick for how robust a neural model is. Their 2025 paper works on perturbation robustness, the ability to shrug off tiny input tweaks. In safety-critical uses, a fooled model can do real harm.
Older checks trade cost against precision and rarely satisfy both needs. The new measure, tower robustness, uses hypothesis testing to give a probability-style score. A broad comparison against older definitions shows where each applies. The authors claim tighter, cheaper pre-deployment checks. The work aims to make robustness a routine gate for risky models rather than a research luxury.
Catalog record
| CREATOR | Wenchuan Mu, Kwan Hui Lim |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-19183
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Get Global Guarantees: On the Probabilistic Nature of Perturbation Robustness Conference: Proceedings of the 34th ACM International Conference on Information and Knowledge Management; November 10–14, 2025; Seoul, Republic of Korea Proceedings of the 34th ACM International Conference on Information and Knowledge Management (CIKM ’25), November 10–14, 2025, Seoul, Republic of Korea DOI: 10.1145/3746252.3761039 ISBN: 979-8-4007-2040-6/2025/11 Wenchuan Mu, Kwan Hui Lim Abstract . In safety-critical deep learning applications, robustness measures the ability of neural models that handle imperceptible perturbations in input data, which may lead to potential safety hazards. Existing pre-deployment robustness assessment methods typically suffer from significant trade-offs between computational cost and measurement precision, limiting their practical utility. To address these limitations, this paper conducts a comprehensive comparative analysis of existing robustness definitions and associated assessment methodologies. We propose tower robustness to evaluate robustness, which is a novel, practical metric based on hypothesis testing to quantitatively evaluate probabilistic robustness, enabling more rigorous and efficient pre-deployment assessments. Our extensive comparative evaluation illustrates the advantages and applicability of our proposed approach, thereby advancing the systematic understanding and enhancement of model robustness in safety-critical deep learning applications. 1. Introduction Deep learning has transformed numerous fields, enabling breakthroughs in image recognition ( He et al. 2016 ; Dosovitskiy et al. 2021 ) , natural language processing ( Zhang et al. 2020 ; Floridi and Chiriatti 2020 ) , and autonomous systems ( Fu et al. 2016 ) . In safety-critical domains, such as self-driving vehicles ( Kurakin et al. 2017a ) , medical diagnosis ( Ragoza et al. 2017 ; Wen et al. 2020 ) , and industrial control systems ( Al-Abassi et al. 2020 ) , the reliability and robustness of deep neural networks (DNNs) are paramount. In these contexts, even minor perturbations in input data can lead to severe consequences, highlighting the need for rigorous robustness evaluation prior to deployment. Model robustness evaluation is well-studied but remains challenging. Formal verification is one established approach, but it often suffers from incomplete problem formulations ( Katz et al. 2017 ) , failing to capture the full range of real-world perturbations. In addition, verification techniques are typically computationally expensive, rendering them impractical for large-scale models ( Zhang et al. 2023 ) . These limitations call for evaluation methods that are both more comprehensive and computationally feasible. To address these gaps, we explore probabilistic robustness assessment, which aims to estimate a model’s failure probability under adversarial perturbations. While existing probabilistic methods frequently rely on approximations, such approximations risk overlooking critical adversarial instances ( Agresti and Coull 1998 ) , potentially overestimating model robustness. We propose a new framework that employs Tower robustness with hypothesis testing to yield statistical guarantees on robustness estimates. This approach enables precise quantification of the probability of model failure, offering a more informative and actionable robustness measure. By grounding our evaluation in exact statistical methods, we aim to better capture model behaviour in real-world deployments. We make our method available at https://github.com/confidential-submission/adversarial-attacks-pytorch , offering researchers and practitioners a user-friendly and statistically rigorous tool for robustness assessment. We validate our approach through extensive experiments on several large-scale DNNs, comparing it against state-of-the-art baselines. Results demonstrate that our method consistently delivers more accurate and reliable robustness estimates, underscoring its value for both theoretical analysis and practical deployment in high-stakes applications. The rest of the paper is organized as follows: Section 2 reviews related work on robustness evaluation; Section 3 outlines the theoretical basis of our metric; Section 4 presents experiments and results, and; Section 5 concludes with key findings. (a) (b) (c) Figure 1 . Comparison of space coverage: (a) Adversarial attacks explore a small portion in the vicinity around input sample x to find adversarial example x^{\prime} . There is no guaranteed prediction for the untouched space. (b) Existing probabilistic methods predict over the entire vicinity but lack precision. The proportion of adversarial examples is possibly higher than the predicted "guarantee". (c) The proposed method achieves comprehensive coverage and maintains precision for robustness evaluation. 2. Preliminaries This section discusses the prerequisite knowledge. We first introduce the notations of each quantity. After that, we review the essentials of statistical learning, e.g., data sampling, objective functions, and neural networks. 2.1. Notations Table 1 . Mathematical object letter styles (e.g., using letter A) A variable or variable function a A random variable or random variable function \mathbf{a} A set \mathbb{A} A distribution A A function whose value is 1 if a is true, 0 otherwise \mathbf{1}_{a} We use the simple letter styles as given in Table 1 to distinguish different types of mathematical objects. Note that the letter P is reserved for probability, and \operatorname{E} for expectation. The notation (e.g., for variables) is generally insensitive to whether an object is a scalar or a vector. Subscripts or superscripts in variable names are simply part of the name, not as indices. In fact, we will never index into vector variables or sets in this manuscript. Also, lowercase Greek letters are always used for scalars. 2.2. Data, Distribution, and Samples In the following, we briefly review the probability and statistics background of this work. Let D denote a joint probability distribution for random variables \mathbf{x} and \mathbf{y} , such that for some positive integer n , a sample from D could look like (1) \left\{(x_{1},y_{1}),(x_{2},y_{2}),\ldots,(x_{n},y_{n})\right\} where n is the sample size. In practice, we are usually given the sample. For example, MNIST ( LeCun et al. 1998 ) dataset could be seen as a sample drawn from the distribution of handwritten digits 1 1 1 To view MNIST as an unbiased sample, we might as well say it is from a distribution of American employees’ and high school students’ handwritten digits. . If we further look into the set of MNIST training images and testing images, they can be considered two samples drawn from this distribution where n=60,000 for the training set and n=10,000 for the testing set. Let \mathbb{X} be the outcome space of \mathbf{x} , and \mathbb{Y} be the outcome space of \mathbf{y} . Intuitively, in the MNIST context, \mathbb{X} would be the set containing all 28 \times 28 pixel greyscale images, and \mathbb{Y} would be a set of integers from 0 to 9. Often, we are interested in the expected value of some function f(\mathbf{x},\mathbf{y}) , denoted by \operatorname{E}(f(\mathbf{x},\mathbf{y})) and computed as follows. (2) \operatorname{E}\left[f(\mathbf{x},\mathbf{y})\right]=\int_{\mathbb{Y}}\int_{\mathbb{X}}f(x,y)P(\mathbf{x}=x,\mathbf{y}=y)\penalty\ dx\penalty\ dy In practice, if a sample is large enough, the law of large numbers ensures that the sample average converges to the true expected value, i.e., as n\to\infty (3) \operatorname{E}\left[f(\mathbf{x},\mathbf{y})\right]\approx\frac{1}{n}\sum_{i=1}^{n}f(x_{i},y_{i}). 2.3. Learning and Neural Models Let us denote a function h:\mathbb{X}\to\mathbb{Y} . Essentially, \mathbb{X} is considered an input feature space, and \mathbb{Y} is considered a label space, while a model function h predicts a label from a given feature. A good model makes correct predictions, i.e., we want to maximise (4) \operatorname{E}\left[1_{h(\mathbf{x})=\mathbf{y}}\right]. Its practical measurement, however, often relies on a finite set. According to approximation ( 3 ), we can rewrite expression ( 4 ) as expression ( 5 ). Indeed, expression ( 5 ) denotes the commonly used test accuracy metric when the set is the testing set. (5) \frac{1}{n}\sum_{i=1}^{n}1_{h(x_{i})=y_{i}} The process of actively optimising the model is referred to as learning (or training). Learning is complicated and