Relation-Guided Adversarial Learning for Data-free Knowledge Transfer
Yingping Liang, Ying Fu · 2024
Read Relation-Guided Adversarial Learning for Data-free Knowledge Transfer by Yingping Liang, Ying Fu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-11380.txt.
Introduction
Yingping Liang and Ying Fu work on teaching a new model from an old one without any of the old training data, in a 2024 paper. The method rebuilds fake data from the trained model itself. Past work chased variety across all data but missed variety within each class and overlap between classes, so the fake data came out too uniform.
Their approach works in two phases with an adversarial push and pull. First it spreads same class samples apart and draws unlike classes close; then training does the reverse for the student. A weighted sampling rule keeps the two goals from clashing. Tests across tasks showed gains in accuracy and data savings.
Catalog record
| CREATOR | Yingping Liang, Ying Fu |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-11380
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Relation-Guided Adversarial Learning for Data-free Knowledge Transfer Yingping Liang, Ying Fu Abstract Data-free knowledge distillation transfers knowledge by recovering training data from a pre-trained model. Despite the recent success of seeking global data diversity, the diversity within each class and the similarity among different classes are largely overlooked, resulting in data homogeneity and limited performance. In this paper, we introduce a novel R elation- G uided A dversarial L earning method with triplet losses, which solves the homogeneity problem from two aspects. To be specific, our method aims to promote both intra-class diversity and inter-class confusion of the generated samples. To this end, we design two phases, an image synthesis phase and a student training phase. In the image synthesis phase, we construct an optimization process to push away samples with the same labels and pull close samples with different labels, leading to intra-class diversity and inter-class confusion, respectively. Then, in the student training phase, we perform an opposite optimization, which adversarially attempts to reduce the distance of samples of the same classes and enlarge the distance of samples of different classes. To mitigate the conflict of seeking high global diversity and keeping inter-class confusing, we propose a focal weighted sampling strategy by selecting the negative in the triplets unevenly within a finite range of distance. RGAL shows significant improvement over previous state-of-the-art methods in accuracy and data efficiency. Besides, RGAL can be inserted into state-of-the-art methods on various data-free knowledge transfer applications. Experiments on various benchmarks demonstrate the effectiveness and generalizability of our proposed method on various tasks, specially data-free knowledge distillation, data-free quantization, and non-exemplar incremental learning. Our code will be publicly available to the community. ∎ Journal: IJCV 1 Introduction Deep convolutional neural networks (CNNs) have demonstrated remarkable success in various computer vision tasks, such as classification ( Krizhevsky et al, 2012 ; Simonyan and Zisserman, 2015 ; He et al, 2016 ) , object detection ( Girshick et al, 2014 ; Redmon et al, 2016 ; Zhang et al, 2018 ) , and semantic segmentation ( Long et al, 2015 ; Ronneberger et al, 2015 ) . However, these well-designed models require significant computational resources, which limits their deployment on resource-limited platforms such as mobile phones and edge devices ( Cheng et al, 2018 ) . To address this issue, knowledge distillation (KD) has emerged as a standard model compression method that aims to transfer knowledge from a well-trained teacher model to a new, smaller one ( Gou et al, 2021 ; Hinton et al, 2014 ; Romero et al, 2015 ) . Despite its success, most KD methods assume that the training dataset is still accessible. However, in many cases, data privacy and security concerns prevent the sharing of sensitive data ( Ha et al, 2020 ) . For instance, biometric and medical data are often considered private and not publicly shared ( Arora and Bhatia, 2021 ) . Therefore, there is a pressing need to transfer knowledge without accessing the original dataset or the real data. Data-free KD can effectively solve the challenge of transferring knowledge between neural networks without access to the original training data. Inspired by the generative adversarial network ( Goodfellow et al, 2014 ) , existing data-free KD methods typically generate synthetic samples that the teacher classifies with high certainty ( Mahendran and Vedaldi, 2015 ) . Subsequently, adversarial learning is introduced into the field of data-free knowledge distillation to explore more valuable samples by minimizing the maximum KL divergence between the outputs of the teacher and the student ( Micaelli and Storkey, 2019 ; Choi et al, 2020 ) . However, real-world images of different classes often contain different contents, while synthesized samples tend to be homogenized, leading to limited performance of models trained on these synthesized samples. It is essential for the synthesized samples to sufficiently match the diversity in real data. Therefore, recent data-free methods try to enhance the diversity of synthetic data. For example, Deepinversion ( Yin et al, 2020 ) introduces an additional Jensen-Shannon divergence to expand the coverage of the image distribution. IntraQ ( Zhong et al, 2022 ) proposes diversity-seeking regularization to prevent the generated samples from being too similar. CMI ( Fang et al, 2021 ) improves diversity with a large amount of pre-synthesized images and encourages the synthesizing images to distinguish from the previous ones. Although these methods enhance the diversity of synthetic data, models trained with real data still outperform those trained with synthetic data. This may be because current methods often overlook the relation among individual samples and optimize the samples independently in the same direction. This causes synthetic samples to tend to have similar results without avoiding homogenization problems. Intra-class samples cluster together, and inter-class samples become too easily distinguishable. Therefore, we emphasize that it is necessary to improve the diversity of individual samples within the same training batch, especially those of the same classes. In addition, our study has also revealed class-level heterogeneity issues in the data generation process of typical data-free methods, particularly at embedding levels. Therefore, the lack of hard samples with high inter-class confusion also prevents the student from learning more robust class-level discrimination. These gaps lead to difficulties in learning from synthetic data with high homogeneity. Figure 1: Illustration of sample optimization in different data generation methods. Red arrow indicates pushing away while green indicates pulling close. Both adversarial based and contrastive methods ignore the relation between individual samples, resulting in limited intra-class diversity and inter-class confusion. Our method aims to deal with the distances among samples, leading to both high intra-class diversity and meanwhile maintaining inter-class confusion. To address these problems, we propose a novel Relation-Guided Adversarial Learning (RGAL) framework by promoting both intra-class diversity and inter-class confusion. Unlike prior methods that optimize the samples independently, we focus on optimizing individual samples and the relation among the samples in a single batch. Our proposed method refers to two learning phases, an image synthesis phase for training the generator and a student training phase. In the image synthesis phase, we enlarge the distance between positive pairs of the same classes in the embedding space, which prevents the samples from being too similar and ensures intra-class diversity. Negative pairs of different classes are forced to be close to each other, which ensures that the samples of different classes are around the decision boundary and ”confusing” enough. On the contrary, when training the student, the positive pairs are pulled closer than the negative ones by a distance margin to learn much stronger class-level discrimination. Figure 1 illustrates the comparison between the knowledge transfer processes using different data-free knowledge distillation (KD) methods: adversarial methods Micaelli and Storkey (2019) ; Choi et al (2020) ; Yin et al (2020) , contrastive methods Zhong et al (2022) ; Fang et al (2021) ; Fang et al (2022) , and our proposed method. Adversarial methods generate synthetic samples classified with high uncertainty by adversarially learning pseudo data with larger gaps between the teacher and the student. However, the samples tend to be homogenized. Contrastive methods enhance diversity by driving the samples away from the previous ones. But the optimization of relationships among samples within a batch is also overlooked.