Theoretical Analysis of Relative Errors in Gradient Computations for Adversarial Attacks with CE Loss
Yunrui Yu, Hang Su, Cheng-zhong Xu, Zhizhong Su, Jun Zhu · 2025
Read Theoretical Analysis of Relative Errors in Gradient Computations for Adversarial Attacks with CE Loss by Yunrui Yu, Hang Su, Cheng-zhong Xu, Zhizhong Su, Jun Zhu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-22428.txt.
Introduction
Yunrui Yu and co-authors study a quiet source of error in attacks that fool image classifiers. Such attacks follow gradients computed from a loss function, but floating-point arithmetic distorts those gradients, and the distortion can inflate how strong an attack looks. The team gives the first full theoretical account of these relative errors, covering four cases formed by targeted or untargeted attacks that succeed or fail. They trace the trouble mainly to underflow and rounding.
Guided by the theory, the authors design a new loss function with a scaling factor tuned to shrink the numerical error. Across the MNIST, CIFAR-10, and CIFAR-100 image sets, their loss beat four established rivals in attack strength and in the accuracy of robustness checks. The work suggests that some past results in this area may have been skewed by arithmetic noise.
Catalog record
| CREATOR | Yunrui Yu, Hang Su, Cheng-zhong Xu, Zhizhong Su, Jun Zhu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-22428
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Theoretical Analysis of Relative Errors in Gradient Computations for Adversarial Attacks with CE Loss Yunrui Yu, Hang Su, Cheng-zhong Xu, Zhizhong Su, Jun Zhu Abstract Gradient-based adversarial attacks using the Cross-Entropy (CE) loss often suffer from overestimation due to relative errors in gradient computation induced by floating-point arithmetic. This paper provides a rigorous theoretical analysis of these errors, conducting the first comprehensive study of floating-point computation errors in gradient-based attacks across four distinct scenarios: (i) unsuccessful untargeted attacks, (ii) successful untargeted attacks, (iii) unsuccessful targeted attacks, and (iv) successful targeted attacks. We establish theoretical foundations characterizing the behavior of relative numerical errors under different attack conditions, revealing previously unknown patterns in gradient computation instability, and identify floating-point underflow and rounding as key contributors. Building on this insight, we propose the Theoretical MIFPE ( T-MIFPE ) loss function, which incorporates an optimal scaling factor T = t ∗ T=t^{*} to minimize the impact of floating-point errors, thereby enhancing the accuracy of gradient computation in adversarial attacks. Extensive experiments on the MNIST, CIFAR-10, and CIFAR-100 datasets demonstrate that T-MIFPE outperforms existing loss functions, including CE, C&W, DLR, and MIFPE, in terms of attack potency and robustness evaluation accuracy. 1 Introduction Deep learning has fundamentally reshaped the landscape of artificial intelligence (AI), achieving remarkable success across diverse domains. Its applications span safety-critical fields such as aviation [ 1 ] , where neural networks enable object classification in X-ray imagery, medical diagnosis [ 2 ] , facilitating precise segmentation of medical images, and autonomous driving [ 3 ] , where models support real-time object detection for navigation. Moreover, the advent of large-scale language models, exemplified by ChatGPT, has further expanded the reach of deep learning, revolutionizing natural language processing and enabling applications in areas such as automated content generation, education, and human-computer interaction [ 4 ] . As deep learning becomes increasingly integrated into human life, its applications continue to grow, permeating sectors like healthcare, education, and public policy, thereby amplifying its societal impact. However, these advancements are accompanied by a critical vulnerability: deep neural networks (DNNs) exhibit limited robustness to adversarial attacks. Even imperceptible perturbations to input data can mislead models into producing erroneous outputs [ 5 , 6 ] , posing significant safety risks to DNN-based systems. For instance, in autonomous driving, a misclassification induced by a small perturbation could result in catastrophic failures [ 7 ] . This vulnerability highlights the pressing need to enhance the robustness of DNNs and to develop reliable evaluation strategies capable of accurately assessing their resilience against adversarial threats, thereby ensuring the safe and ethical deployment of deep learning technologies in an increasingly interconnected society. To tackle these challenges, the research community has proposed numerous defense strategies [ 6 , 8 , 9 , 10 , 11 , 12 , 13 ] aimed at improving model robustness, alongside various attack techniques [ 10 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 ] designed to evaluate defense effectiveness. Among attack methods, white-box attacks represent the most rigorous robustness test, as attackers possess complete knowledge of the target model’s architecture, parameters, training algorithms, and datasets. A canonical example is the Projected Gradient Descent (PGD) attack [ 10 ] , which exploits gradient information to generate adversarial examples for resilience assessment. However, studies demonstrate that PGD paired with conventional cross-entropy (CE) loss frequently overestimates model robustness [ 13 , 24 , 21 ] . This occurs because CE-computed gradients inadequately guide adversarial example generation—a phenomenon associated with gradient masking [ 25 ] . Advanced evaluation methods have thus employed attack algorithm ensembles [ 13 , 24 ] to combine multiple strategies for improved accuracy. Yet these methods lack fundamental analysis of root causes behind gradient-based attack overestimation. Research by [ 22 ] reveals persistent overestimation issues even when evaluating advanced defenses like ensemble methods [ 26 , 27 , 28 ] . The failure of gradient-based attacks like PGD with CE loss has prompted the development of alternative loss functions to mitigate the overestimation problem. Notable examples include the Carlini and Wagner (C&W) loss [ 9 ] , the Difference-of-Logits Ratio (DLR) loss [ 13 ] , and the Minimize the Impact of Floating-point Errors (MIFPE) loss [ 29 ] . The C&W loss, also known as the hinge loss, avoids exponential operations present in CE, reducing the risk of floating-point underflow errors, but it discards some logit elements, potentially weakening the attack’s effectiveness. Similarly, the DLR loss scales the difference between the largest logits to improve gradient computation but suffers from similar limitations due to partial logit utilization. In contrast, MIFPE [ 29 ] provides a deeper insight into the root cause of overestimation. The authors reveal that the relative error in gradients, induced by floating-point errors—such as underflow and rounding errors—is a fundamental reason for the failure of CE-based gradient attacks. This relative error is particularly influenced by the difference between the first and second largest logits ( \Delta=\mathbf{z}_{\pi_{1}}-\mathbf{z}_{\pi_{2}} ). To mitigate this, MIFPE scales the logits by a factor c=T/\Delta_{detach} where T=1 , significantly reducing the impact of floating-point errors and improving the accuracy of robustness evaluation. Experimental results demonstrate that MIFPE outperforms CE, C&W, and DLR in terms of efficiency and reliability across various defense mechanisms. However, a critical limitation of MIFPE lies in its empirical determination of T=1 , which lacks a theoretical foundation, raising questions about its optimality. In this paper, we extend the foundational work of MIFPE [ 29 ] by pioneering a comprehensive theoretical framework that dissects relative errors in gradients caused by floating-point arithmetic in the context of the cross-entropy (CE) loss function. Our groundbreaking analysis is the first to systematically explore floating-point computation errors across four distinct attack scenarios: (i) unsuccessful untargeted attacks, (ii) successful untargeted attacks, (iii) unsuccessful targeted attacks, and (iv) successful targeted attacks. By establishing robust theoretical foundations, we uncover novel insights into the patterns of numerical error behavior under varying attack conditions, shedding light on previously unrecognized instabilities in gradient computations. Building on this theoretical insight, we propose the Theory Version of Minimize the Impact of Floating Point Error ({\text{T-MIFPE}}) loss function, which adopts the optimal scaling factor T=t^{*} . This adjustment ensures that the relative error due to floating-point errors is minimized, thereby enhancing the accuracy of gradient computation in adversarial attacks. Experimental results on the MNIST, CIFAR-10, and CIFAR-100 datasets validate that the T-MIFPE loss function outperforms existing loss functions, including CE, C&W, DLR, and MIFPE, in terms of attack potency and robustness evaluation accuracy. To summarize, our main contributions are as follows: • We conduct the first comprehensive study of floating-point computation errors in gradient-based attacks across four distinct scenarios: (i) unsuccessful untargeted attacks, (ii) successful untargeted attacks, (iii) unsuccessful targeted attacks, and (iv) successful targeted attacks. • We establish theoretical foundations characterizing the behavior of relative numerical errors under different attack conditions, revealing previously unknown patterns in gradient computation instability. • Leveraging this theoretical analysis, we propose the T-MIFPE loss function, which incorporates an optimal scaling factor T=t^{*} to minimize the impact of floating-point errors on gradient computation. • Through extensive experiments, we demonstrate that the T-MIFPE loss function