Competitive Distillation: A Simple Learning Strategy for Improving Visual Classification

Daqian Shi, Xiaolei Diao, Xu Chen, Cédric M. John · 2025

Read Competitive Distillation: A Simple Learning Strategy for Improving Visual Classification by Daqian Shi, Xiaolei Diao, Xu Chen, Cédric M. John (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-23285.txt.

Introduction

Daqian Shi, Xiaolei Diao, Xu Chen, and Cédric John propose a training scheme where a group of image networks learn by competing. In classic knowledge distillation, a fixed teacher network guides a student in one direction. Group methods let networks teach each other, but gains stay small because who teaches whom rarely changes with performance.

In competitive distillation, every network in the group does the same task, and the best performer at each step becomes the teacher for the others. A competitive update rule reshapes how parameters change, and random perturbations push the networks to explore better internal representations. Tests across varied vision tasks and datasets showed strong results. The authors present the method as a simple, general way to lift classification training.

Catalog record

CREATORDaqian Shi, Xiaolei Diao, Xu Chen, Cédric M. John
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-23285

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Competitive Distillation: A Simple Learning Strategy for Improving Visual Classification Daqian Shi, Xiaolei Diao, Xu Chen, Cédric M. John Abstract Deep Neural Networks (DNNs) have significantly advanced the field of computer vision. To improve DNN training process, knowledge distillation methods demonstrate their effectiveness in accelerating network training by introducing a fixed learning direction from the teacher network to student networks. In this context, several distillation-based optimization strategies are proposed, e.g., deep mutual learning and self-distillation, as an attempt to achieve generic training performance enhancement through the cooperative training of multiple networks. However, such strategies achieve limited improvements due to the poor understanding of the impact of learning directions among networks across different iterations. In this paper, we propose a novel competitive distillation strategy that allows each network in a group to potentially act as a teacher based on its performance, enhancing the overall learning performance. Competitive distillation organizes a group of networks to perform a shared task and engage in competition, where competitive optimization is proposed to improve the parameter updating process. We further introduce stochastic perturbation in competitive distillation, aiming to motivate networks to induce mutations to achieve better visual representations and global optimum. The experimental results show that competitive distillation achieves promising performance in diverse tasks and datasets. 1 Introduction Recent advancements in computer vision have been greatly facilitated by deep neural networks (DNNs), which have proven significant success in tasks such as image classification [ 21 ] , object detection [ 30 ] , and semantic segmentation [ 39 ] . Knowledge distillation exploits the deep representation capabilities of pre-trained DNNs, aiming to achieve effective network training with a lower cost [ 27 ] . By setting up a teacher-student learning paradigm, knowledge distillation allows a student network to learn from a teacher network by imitating its outputs [ 14 ] . Such a paradigm is considered a manifestation of collective intelligence [ 23 ] , which is a dynamic optimization strategy aimed at leveraging collaborative and competitive processes among multiple entities to generate complex decisions. The effectiveness of knowledge distillation methods demonstrates that learning by network collaboration is often an easier optimization task than directly learning from the target function [ 2 ] . Figure 1 : Comparison of parameter update strategies between general model distillation, mutual learning-based distillation, and proposed competitive distillation. The most common cases of the teacher-student paradigm allow a student network to learn from a fixed teacher network, named model distillation, as shown in Fig. 1 (a). The success of such methods in enhancing the efficiency of training student networks validates the rationality and effectiveness of the distillation process. Meanwhile, some distillation-based methods are specifically proposed to improve the overall task performance, e.g., deep mutual learning [ 42 ] and self-distillation [ 41 ] . Fig. 1 (b) demonstrates the structure of mutual learning-based methods, where each of the networks serves as both a teacher and a student network, enabling them to learn from each other by aligning each network’s class posterior with the class probabilities of other networks during the learning iterations. Compared with training networks independently, mutual learning-based methods indeed enhance the overall task performance, aligning with the collaborative processes emphasized by collective intelligence strategies. However, it is reasonable for networks with poor performance to learn from superior ones, but counter-intuitive for the best-performing networks to learn from others. Thus, the learning direction of each network in a given iteration should be dynamically planned. In this paper, we introduce a simple yet effective distillation-based strategy for improving visual classification, namely competitive distillation. Inspired by the competitive process from collective intelligence, we propose a competitive optimization-based parameter updating method to distill knowledge from the best-performing network to other networks at each training iteration dynamically. Specifically, the method involves a group of networks trained without a predetermined learning direction, whereby the winning network becomes the teacher for the current iteration and the others learn from the teacher as students, as shown in Fig. 1 (c). Besides, inspired by genetic algorithm [ 18 ] , we propose a stochastic perturbation process to motivate networks to enhance competition for better visual representations. We randomly introduce image perturbations into one of the networks at each iteration, the aim is to induce mutations 1 1 1 Mutation is the operator to maintain the genetic diversity from one population to the next in the genetic algorithm. [ 26 ] for providing the possibility for the network to escape local optima. The experimental results show that the proposed competitive distillation method achieves better performance than every single network when it learns individually in the traditional supervised learning scenario. In particular, the competitive distillation method outperforms state-of-the-art distillation methods on various visual tasks, including vision classification, person re-identification, and object detection. These results demonstrate that the competition optimization-based training strategy and stochastic perturbation process effectively improve the performance and efficiency of deep neural networks for various applications. The contributions of our paper are summarized as follows: • We propose a novel training strategy, i.e., competitive distillation, for improving the performance of DNNs on various computer vision tasks, which exploits a competitive optimization strategy for effective parameter updating and knowledge distillation. • We introduce perturbations to our method by randomly introducing perturbed images on each network. Benefiting from the competitive optimization strategy, the positive perturbation results will be retained to provide the network with the possibility of breaking through local optima. • We validate the proposed method on various image classification benchmarks. Compared to state-of-the-art distillation methods, our method achieves promising results in both task performance and training efficiency. 2 Related Work 2.1 Knowledge Distillation Knowledge distillation has been proposed to open up new avenues for improving the efficiency and performance of deep neural networks [ 14 ] . Generally, these distillation-based methods [ 31 , 28 , 10 , 34 ] are conducted with a teacher-student framework that distills powerful large teacher networks into small student networks to reduce training costs. [ 1 ] introduces an ensemble knowledge distillation architecture that exploits multiple teacher networks to promote heterogeneity in feature learning. [ 4 ] attempt to narrow down the teacher-student performance gap by reusing a discriminative classifier from a pre-trained teacher network for student inference. To further improve the performance of trained networks, [ 8 ] first exploited knowledge distillation as a network development method instead of a compression method, where they train student networks parameterized identically to their teachers. This observation gives a hint that knowledge distillation is able to improve the performance of target networks, where researchers then introduce a strategy that implements the student network also as a teacher in the same distillation process, which means, one specific network can learn from others and also be distilled by others. For instance, deep mutual learning (DML) allows an ensemble of students to teach each other in a mutual distillation framework, which makes each student network improve compared with training them separately [ 42 ] . Different from using multiple networks, [ 41 ] proposed a self-distillation method that makes only one network as both the teacher and student during the training process, allowing it to distill and learn from itself. Besides, Curriculum Temperature for Knowledge Distillation [

Read the complete text