Towards Undistillable Models by Minimizing Conditional Mutual Information
Linfeng Ye [email protected], Shayan Mohajer Hamidi [email protected], En-Hui Yang [email protected] · 2025
Read Towards Undistillable Models by Minimizing Conditional Mutual Information by Linfeng Ye [email protected], Shayan Mohajer Hamidi [email protected], En-Hui Yang [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-00012.txt.
Introduction
A valuable trained model can be stolen by distillation, where a student network learns to copy a teacher's answers. Linfeng Ye, Shayan Mohajer Hamidi, and En-Hui Yang build models that resist that theft in 2025, calling them undistillable.
Their insight is that a protected model's answer patterns for each class should collapse into one tight cluster, leaving a copycat nothing extra to learn. They measure cluster tightness with conditional mutual information and train the network to shrink it alongside the usual loss, across all softness settings. In wide tests, students distilled from such models did worse than students trained with plain label smoothing. The method offers model owners a built-in defense of their intellectual property.
Catalog record
| CREATOR | Linfeng Ye [email protected], Shayan Mohajer Hamidi [email protected], En-Hui Yang [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 90 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-00012
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Undistillable Models by Minimizing Conditional Mutual Information Linfeng Ye [email protected], Shayan Mohajer Hamidi [email protected], En-Hui Yang [email protected] Abstract A deep neural network (DNN) is said to be undistillable if, when used as a black-box input-output teacher, it cannot be distilled through knowledge distillation (KD). In this case, the distilled student (referred to as the knockoff student) does not outperform a student trained independently with label smoothing (LS student) in terms of prediction accuracy. To protect intellectual property of DNNs, it is desirable to build undistillable DNNs. To this end, it is first observed that an undistillable DNN may have the trait that each cluster of its output probability distributions in response to all sample instances with the same label should be highly concentrated to the extent that each cluster corresponding to each label should ideally collapse into one probability distribution. Based on this observation and by measuring the concentration of each cluster in terms of conditional mutual information (CMI), a new training method called CMI minimized (CMIM) method is proposed, which trains a DNN by jointly minimizing the conventional cross entropy (CE) loss and the CMI values of all temperature scaled clusters across the entire temperature spectrum. The resulting CMIM model is shown, by extensive experiments, to be undistillable by all tested KD methods existing in the literature. That is, the knockoff students distilled by these KD methods from the CMIM model underperform the respective LS students. In addition, the CMIM model is also shown to performs better than the model trained with the CE loss alone in terms of their own prediction accuracy. The code for the paper is publicly available at https://anonymous.4open.science/r/CMIM-605C/README.md . 1 Introduction Originally aiming for model compression, knowledge distillation ( Buciluǎ et al. 2006 ; Hinton et al. 2015 ) (KD) has received significant attention from both academia and industry in recent years due to its remarkable effectiveness. The essence of KD is to transfer the knowledge of a pre-trained large model (teacher) to a smaller model (student). Building on the work of Hinton et al. 2015 , numerous follow-up works have endeavored to enhance the performance of KD ( Romero et al. 2014 ; Anil et al. 2018 ; Park et al. 2019 ) and to gain deeper insights into why distillation is effective ( Phuong & Lampert 2019 ; Mobahi et al. 2020 ; Ye et al. 2024 ; Allen-Zhu & Li 2020 ; Menon et al. 2021 ; Borup & Andersen 2021 ) . In the scenario where the teacher does not want its knowledge to be transferred, however, KD is undesirable and indeed poses a threat to intellectual property (IP) of the teacher ( Shokri & Shmatikov 2015 ) . Developing and training a high-quality large DNN requires significant investments of time, effort, finances, and resources, including extensive data annotation and computational infrastructure. Developers of the large DNN may want to prevent the knowledge of the large DNN from being transferred by their competitors. However, once the large DNN is released as a “black box”, anyone can apply a logit-based KD method (or equivalently a distribution-based KD method ( Zheng & YANG 2024 ) ) to distill the DNN as a teacher. The goal is to train a student, referred to as a knockoff student in the context of DNN IP protection, that mimics the teacher’s behavior to gain competitive advantages. As such, in this case, it would be desirable for the developers to build the large DNN so that it is undistillable. The question, of course, is how. Before delving deeper into the above question, let us first clarify what we mean by saying that a DNN is undistillable. At this point, we invoke the concept of distillable DNN introduced recently in Yang & Ye 2024 : Definition 1 . [Distillability of a DNN ( Yang & Ye 2024 ) ] When used as a black-box input-output teacher, a DNN is said to be distillable 1 1 1 There are two reasons for us to adopt this definition of distillability. First, as shown theoretically in Zheng & YANG 2024 , knowledge distillation reduces to label smoothing in the limit as the temperature approaches infinity. Therefore, if the teacher is distillable, the distilled student should perform no worse than the LS student. Second, if a knockoff student cannot outperform the LS student, then there is no incentive for the teacher to be leveraged since the LS student can be trained on its own. with respect to a student if there exists a KD method which, when applied to the teacher and student, yields a knockoff student outperforming the student trained alone with label smoothing (LS student) in terms of prediction accuracy. Therefore, a DNN is undistillable if no knockoff student can outperform the respective LS student regardless of which logit-based KD method is used. Since there are so many logit-based KD methods and so many students, what type of a DNN is undistillable? In comparison with the training of LS student, Definition 1 provides some insight into a trait that an undistillable DNN may possess. For each label, consider the cluster of the output probability distributions of the DNN in response to all input sample instances with that label. If each cluster corresponding to each label is highly concentrated to the extent that all probability distributions within the cluster more or less collapse into one probability distribution, then the student training within KD is similar to that of the respective LS student regardless of which logit-based KD method and which student are applied. In this case, one would expect that no knockoff student would perform significantly better than the respective LS student. Therefore, a DNN possessing this trait will likely be undistillable. From a theoretical perspective, KD relies on the diversity within the teacher’s predicted distributions to convey richer contextual information to the student. The greater the diversity in the cluster of predictions for a given class, the more nuanced guidance the student can receive (see Ye et al. 2024 ). Conversely, when the teacher’s outputs become highly concentrated—meaning predictions for inputs with the same label converge to a single probability vector—this contextual richness is lost. The student, in this case, receives largely redundant information, rendering the distillation process as ineffective as direct training with label smoothing. To quantify this notion of concentration, we turn to conditional mutual information (CMI) ( Yang et al. 2025 ) . Specifically, let X denote the random input sample to the DNN, and Y be the ground truth label of X . Let \hat{Y} denote the random label predicted by the DNN in response to input X . It was shown in Yang et al. 2025 that for each label y , the label specific CMI \mathrm{I}(X;\hat{Y}|\penalty\ Y=y) measures the concentration of the cluster corresponding to label y , and the CMI \mathrm{I}(X;\hat{Y}|\penalty\ Y) measures the average concentration across all clusters. To build an undistillable DNN, one then is motivated to minimize jointly the conventional cross entropy (CE) loss and the CMI \mathrm{I}(X;\hat{Y}|\penalty\ Y) . In this paper, we will go one step further. In KD ( Hinton et al. 2015 ) , temperature scaling of logits is often applied. It was shown in