Identifying and Understanding Cross-Class Features in Adversarial Training
Zeming Wei, Yiwen Guo, Yisen Wang · 2025
Read Identifying and Understanding Cross-Class Features in Adversarial Training by Zeming Wei, Yiwen Guo, Yisen Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-05032.txt.
Introduction
Zeming Wei, Yiwen Guo, and Yisen Wang study a 2025 question on how adversarial training makes image models robust. Such training feeds models with trick inputs so they learn to resist attacks, but its inner workings stay debated. The team sorts the traits models use by whether they are shared across classes or tied to one class.
Early in training, models lean on shared traits, up to the point of best robustness. As training squeezes the loss further, models shift to class-tied traits, and robustness starts to slip. That arc, the authors argue, explains both the gain from soft labels and the late-training drop.
Catalog record
| CREATOR | Zeming Wei, Yiwen Guo, Yisen Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-05032
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Identifying and Understanding Cross-Class Features in Adversarial Training Zeming Wei, Yiwen Guo, Yisen Wang Abstract Adversarial training (AT) has been considered one of the most effective methods for making deep neural networks robust against adversarial attacks, while the training mechanisms and dynamics of AT remain open research problems. In this paper, we present a novel perspective on studying AT through the lens of class-wise feature attribution. Specifically, we identify the impact of a key family of features on AT that are shared by multiple classes, which we call cross-class features. These features are typically useful for robust classification, which we offer theoretical evidence to illustrate through a synthetic data model. Through systematic studies across multiple model architectures and settings, we find that during the initial stage of AT, the model tends to learn more cross-class features until the best robustness checkpoint. As AT further squeezes the training robust loss and causes robust overfitting, the model tends to make decisions based on more class-specific features. Based on these discoveries, we further provide a unified view of two existing properties of AT, including the advantage of soft-label training and robust overfitting. Overall, these insights refine the current understanding of AT mechanisms and provide new perspectives on studying them. Our code is available at https://github.com/PKU-ML/Cross-Class-Features-AT . 1 Introduction As the existence of adversarial examples ( Goodfellow et al. 2014 ) has led to significant safety concerns of deep neural networks (DNNs), a series of methods ( Papernot et al. 2016 ; Cohen et al. 2019 ; Chen et al. 2024 ) for defending against this threat have been proposed. Adversarial training (AT) ( Madry et al. 2018 ) , which adaptively adds adversarial perturbations to samples in the training loop, has been considered one of the most effective ways to make the DNNs more robust to adversarial attacks ( Athalye et al. 2018 ) . Given the unique success in improving adversarial robustness and the complex optimization process of AT, several studies have attempted to interpret AT through different perspectives like feature visualization ( Ilyas et al. 2019 ; Bai et al. 2021a ; Li et al. 2023 ) and coverage analysis ( Wang et al. 2019 ) . However, there are still a few mysterious properties of AT whose underlying mechanisms remain open research problems. First, AT can lead to a phenomenon known as robust overfitting ( Rice et al. 2020 ) . During AT, a model may achieve its best test robust error at a certain epoch, but the test robust error will gradually increase in the latter stage of training. By contrast, the training robust error consistently decreases, resulting in a large robust generalization gap. Furthermore, although one-hot labels are usually adequate for standard training, integrating soft-label training methods such as knowledge distillation ( Hinton et al. 2015 ) into AT can significantly improve AT whilst mitigating robust overfitting ( Chen et al. 2021 ) ( e.g. 41\%\to 48\% on CIFAR-10 dataset). However, the reasons why soft labels are typically advantageous for AT remain unclear. In this paper, we explore the mechanisms of AT and offer a unified understanding of the two properties from a new aspect of class-wise feature attribution. Specifically, we divide the features learned by the model into cross-class features and class-specific features. The cross-class features are shared among multiple classes in the classification task, e.g. the feature wheels shared by the automobile and truck classes in the CIFAR-10 dataset ( Krizhevsky et al. 2009 ) . We examine how these features are utilized across various stages of AT. Intriguingly, we observe that at the initial stage, the model gradually learns more cross-class features until reaching the most robust checkpoint. In contrast, at later checkpoints where robust overfitting occurs, the model tends to make decisions based more on class-specific features and decreases its dependence on cross-class features. Furthermore, we find that models trained with properly learned soft labels, like knowledge distillation, can preserve more cross-class features during AT whilst mitigating robust overfitting. Motivated by these observations, we propose a novel hypothesis of the AT training dynamics. During the initial stage of AT, the model learns both class-specific and cross-class features simultaneously, since these features are both helpful for reducing robust loss ( i.e. , the cross-entropy loss on adversarial examples) when this loss is large. However, as training progresses and the robust loss decreases to a certain degree, the model begins to abandon cross-class features and makes decisions based mainly on class-specific features, which is caused by cross-class features raising positive logits on other classes and yielding positive robust loss in AT under one-hot labels. Therefore, the model tends to neglect these features to further decrease the robust loss. However, these cross-class features are helpful for robust classification ( e.g. , a feature shared by classes y_{1},y_{2} helps the model distinguish samples in class y_{1} from other classes y_{3},\cdots,y_{n} ), and using only class-specific features is insufficient to achieve the best robust accuracy. We discuss this insight in detail in Section 4 . As a result, the robust test accuracy ( i.e. , the accuracy of the model on adversarial examples) gradually decreases, leading to the robust overfitting issue. In addition, this hypothesis also explains why soft-label training methods typically improve AT as well as alleviate robust overfitting, as their softened labels can preserve more cross-class features during AT than standard one-hot labels. We provide extensive empirical evidence to support the observations and the hypothesis. First, we propose a metric to measure the usage of the cross-class features for a certain model. Then, among various perturbation norms, datasets, and architectures, we show that the best robustness model consistently uses more cross-class features than the robust overfitted ones, showing a clear correlation between robust generalization and cross-class features. We further provide theoretical insights to intuitively understand this effect through a synthetic data model, where we show that cross-class features are more sensitive to robust loss, but they are indeed helpful for robust classification. Finally, we extend our study to more scenarios, including discussions on larger training perturbation \epsilon , alternative metrics, standard training, and fast adversarial training ( Wong et al. 2020 ; Andriushchenko & Flammarion 2020 ) , to further support our insights. Our contributions can be summarized as follows: 1. We propose a new hypothesis for the training mechanism in AT from the perspective of cross-class features. Specifically, the model gradually learns them at the initial stage of AT, and tends to reduce the reliance on them after a certain stage. However, these features are actually helpful for robust generalization. 2. We provide both empirical and theoretical evidence to support this understanding. Empirically, we measure the usage of cross-class features through different stages of AT. We also substantiate these assertions in a synthetic data model with decoupled cross-class and class-specific features. 3. Based on our understanding, we further provide a unified interpretation of some intriguing properties of AT, like robust overfitting and the advantage of soft-label training, substantiating a novel perspective to study AT that warrants further investigation. 2 Background and Related Work 2.1 Adversarial Training Adversarial training (AT) ( Madry et al. 2018 ) has been widely recognized as one of the most effective approaches