Intra-class Patch Swap for Self-Distillation

Hongjun Choi 1, Eun Som Jeon, Ankita Shukla, Pavan Turaga · 2025

Read Intra-class Patch Swap for Self-Distillation by Hongjun Choi 1, Eun Som Jeon, Ankita Shukla, Pavan Turaga (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14124.txt.

Introduction

Big teacher networks can train small ones, but teachers are costly to build and store. Hongjun Choi and co-authors drop the teacher entirely in a 2025 paper, using a patch-swapping trick to let one network teach itself.

The trick swaps patches between two images of the same class, making paired versions of differing confidence. The network then learns by aligning its own predictions across those pairs, a stand-in for a teacher's guidance. Nothing is added to the model: no extra parts, no new parameters, just one augmentation step. Across image classification, segmentation, and detection tests, the method beat other teacher-free approaches and even many teacher-based ones. Simplicity, the authors argue, is the point.

Catalog record

CREATORHongjun Choi 1, Eun Som Jeon, Ankita Shukla, Pavan Turaga
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE76 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-14124

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Intra-class Patch Swap for Self-Distillation Hongjun Choi 1, Eun Som Jeon, Ankita Shukla, Pavan Turaga Abstract Knowledge distillation (KD) is a valuable technique for compressing large deep learning models into smaller, edge-suitable networks. However, conventional KD frameworks rely on pre-trained high-capacity teacher networks, which introduce significant challenges such as increased memory/storage requirements, additional training costs, and ambiguity in selecting an appropriate teacher for a given student model. Although a teacher-free distillation (self-distillation) has emerged as a promising alternative, many existing approaches still rely on architectural modifications or complex training procedures, which limit their generality and efficiency. To address these limitations, we propose a novel framework based on teacher-free distillation that operates using a single student network without any auxiliary components, architectural modifications, or additional learnable parameters. Our approach is built on a simple yet highly effective augmentation, called intra-class patch swap augmentation. This augmentation simulates a teacher-student dynamic within a single model by generating pairs of intra-class samples with varying confidence levels, and then applying instance-to-instance distillation to align their predictive distributions. Our method is conceptually simple, model-agnostic, and easy to implement, requiring only a single augmentation function. Extensive experiments across image classification, semantic segmentation, and object detection show that our method consistently outperforms both existing self-distillation baselines and conventional teacher-based KD approaches. These results suggest that the success of self-distillation could hinge on the design of the augmentation itself. Our codes are available at https://github.com/hchoi71/Intra-class-Patch-Swap. Journal: Neurocomputing 1 Introduction Deep neural networks have made impressive progress on a wide range of tasks, but they still struggle with overfitting and poor generalization due to the large number of parameters they have. To overcome these issues, knowledge distillation (KD) acts as a powerful regularization method by leveraging the complex pre-trained teacher’s ability to convey learned logit and/or feature representations from the teacher network to the student network [ 1 , 2 , 3 , 4 , 5 ] . This way, additional knowledge beyond the conventional learning from hard targets (i.e., one-hot labels) guides the student network to find better solutions and accelerate convergence. Figure 1: Three different distillation mechanisms. Teacher-to-Student and Student-to-Student require multiple networks, while self-distillation (ours) needs a single network to train it. Meanwhile, Zhang et al. [ 6 ] proposed online student learning, where the ensemble of students collaboratively learn from each other. The success of this collaborative learning strategy is related to increasing each student’s posterior entropy when transferring knowledge from the Student-to-Student network. However, training a high-capacity teacher and multiple students (Teacher-to-Student and Student-to-Student in Figure 1 ) is always cumbersome due to the need for high computational resources. Moreover, there is no universal standard for determining which teacher/student networks are the most suitable for distilling student networks, which often makes it challenging to select appropriate teacher networks. That is, there is an ambiguity in selecting the “best-fit” teacher for a given student. Even poorly-trained teachers can improve the student’s performance [ 7 ] . To address this issue, teacher-free distillation (self-distillation) approach has been introduced [ 7 , 8 , 9 ] . Self-distillation approach is to train a single model based on knowledge distillation, where the model behaviors as both the teacher and the student, and uses its own knowledge in training process. The goal is to match or even surpass the performance of conventional teacher-student KD methods without the need for an external teacher. However, several prior self-distillation methods require architectural modifications such as adding auxiliary classifiers which can be difficult to generalize across diverse backbone networks [ 8 , 9 ] . In addition, some approaches introduce complex training procedures, including meta-learning or multi-stage optimization, which increase both training cost and memory footprint [ 9 ] . Figure 2: Top: Illustrations of design choices when learning the single network (ResNet-18) on CIFAR100. Instead of matching function directly (Hard Label, CutMix), our method tries to match the outputs (called relaxed knowledge) coming from two swapped inputs while we still use cross-entropy loss with the hard label. Bottom: Averaged top-1 probabilities of the target label from the test data. The blue and orange graphs indicate the averaged top-1 probability of correctly classified samples and misclassified samples, respectively. Compared to other methods, our method guarantees a high quality of predictive distribution, judging by the fact that even mispredictions still belong to the people class. This study addresses a central question: How can we enable efficient and truly teacher-free distillation without relying on auxiliary components or architectural modifications, while maintaining applicability to diverse network types and achieving substantial performance gains? To this end, intra-class samples are used, where one instance shares its knowledge with another instance in the same class. This sharing of instance-to-instance knowledge allows them to learn from each other as training progresses. However, this training approach may restrict performance improvement as it tends to prematurely converge toward local minima which are closely tailored to the training data [ 10 , 11 , 12 ] . To deal with this limitation, we further suggest an intra-class patch swap that gives and takes patches from input pairs from the same class. We present critical intuitions of why this is suited for the proposed self-distillation and what aspects of the designed augmentation differ from other popular data augmentations. Generating event difficulty: As a deep network focuses on the most discriminative part of the image, it is important to let the network see other relevant parts to broaden the understanding of the object [ 13 ] . Motivated by this, we randomly exchange patches between two images in the same class. This could result in one image containing a strong teaching signal, such as the head of an object, while the other contains a relatively weak signal, such as the body. Then, the strong teaching signals generate highly confident knowledge about the true class, similar to that provided by the high-capacity teacher. In contrast, weaker knowledge reflects a lower confidence level. Matching these two different types (strong and weak) of knowledge results in a higher loss contribution compared to matching raw inputs. In other words, more weight is assigned to examples that show a substantial difference in confidence. Throughout the training process, the random swap of patches within intra-class samples keeps generating challenges, thereby preventing the dominance of ‘easy-to-learn’ samples in the classification . Here, the ‘easy-to-learn’ sample, figuratively speaking, refers to the image, mostly covered by the strong teaching signal. Preserving class relationship: Mix-based augmentations such as MixUp [ 14 ] and CutMix [ 15 ] have attempted to enhance the generalizability of the networks by providing new training samples that blends two random images and labels proportionally. However, mixing two random images might be detrimental to the class relationship, which does not consider intra-class variations or similarities. The histogram of Figure 2 provides compelling evidence, which depicts the averaged top-1 predictive distribution of the baby class across all baby test data. CutMix generates a new example by combining the target image (baby) with a non-target image (clock). Being trained with these images and labels might result in inferior predictive distribution. One can see that CutMix assigns probabilities of non-target categories with clock, flatfish, and pear (shown in orange), which are semantically unrelated to the

Read the complete text