Why Do Unlearnable Examples Work: A Novel Perspective of Mutual Information
Yifan Zhu, Yibo Miao, Yinpeng Dong, Xiao-Shan Gao · 2026
Read Why Do Unlearnable Examples Work: A Novel Perspective of Mutual Information by Yifan Zhu, Yibo Miao, Yinpeng Dong, Xiao-Shan Gao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-03725.txt.
Introduction
Yifan Zhu and three others explain why poisoned training data can make a data set unlearnable, using information theory. Unlearnable cases carry invisible tweaks that stop unauthorized models from generalizing, protecting data owners. Current recipes were built on hunches. Their 2026 paper shows the common thread: effective poisons always shrink the mutual information between clean features and poisoned features, and deeper networks squeeze it further as protection improves.
A proof from covariance shape follows: shrinking the conditional covariance within each class lowers that mutual information. Their method, MI-UE, therefore maximizes the cosine similarity among same-class poisoned features, collapsing the covariance directly. Extensive tests show it beating earlier unlearnable example methods sharply, and holding up even when the learner deploys known defense mechanisms against the poisoning.
Catalog record
| CREATOR | Yifan Zhu, Yibo Miao, Yinpeng Dong, Xiao-Shan Gao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 88050 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.03725
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Why Do Unlearnable Examples Work: A Novel Perspective of Mutual Information Yifan Zhu, Yibo Miao, Yinpeng Dong, Xiao-Shan Gao Abstract The volume of freely scraped data on the Internet has driven the tremendous success of deep learning. Along with this comes the growing concern about data privacy and security. Numerous methods for generating unlearnable examples have been proposed to prevent data from being illicitly learned by unauthorized deep models by impeding generalization. However, the existing approaches primarily rely on empirical heuristics, making it challenging to enhance unlearnable examples with solid explanations. In this paper, we analyze and improve unlearnable examples from a novel perspective: mutual information reduction . We demonstrate that effective unlearnable examples always decrease mutual information between clean features and poisoned features, and when the network gets deeper, the unlearnability goes better together with lower mutual information. Further, we prove from a covariance reduction perspective that minimizing the conditional covariance of intra-class poisoned features reduces the mutual information between distributions. Based on the theoretical results, we propose a novel unlearnable method called Mutual Information Unlearnable Examples (MI-UE) that reduces covariance by maximizing the cosine similarity among intra-class features, thus impeding the generalization effectively. Extensive experiments demonstrate that our approach significantly outperforms the previous methods, even under defense mechanisms. 1 Introduction Deep neural networks (DNNs) have achieved unprecedented performance across various fields in the past decade ( 47 ) . This progress is largely driven by the availability of large-scale datasets freely scraped from the Internet, such as ImageNet ( 24 ) and LAION-5B ( 77 ) , which continue to advance state-of-the-art deep models ( 2 ; 57 ) . However, a concerning fact is that some of this data collection occurs without authorization ( 9 ) . Users may be reluctant to contribute their privacy-sensitive data, such as face images and medical reports, to training large-scale commercial models ( 2 ; 86 ; 57 ) . Indeed, according to a report ( 39 ) , a tech company illicitly acquired over three billion facial images to develop a commercial facial recognition model. Other investigations revealed an increasing number of lawsuits between data owners and machine learning companies ( 89 ; 11 ; 21 ; 29 ) . Consequently, there is a growing emphasis on safeguarding data from unauthorized use for training. Tremendous efforts have been made to craft unlearnable examples (UEs) in order to prevent data from being illicitly learned by unauthorized deep models ( 30 ; 41 ; 31 ; 75 ; 109 ) . These methods add elaborate and imperceptible perturbations into the training data so that the model cannot learn meaningful information from the data and thereby significantly degrading the test accuracy. A representative method in this domain is the error-minimization poisoning approach ( 41 ) , which employs iterative optimization of a bi-level min-min problem to create poisoning noise. The underlying intuition is that smaller losses trick the model into believing there is nothing to learn from the dataset. Recently, several methods ( 31 ; 100 ; 75 ; 54 ) have been developed to further enhance the effectiveness of UEs. However, existing methods primarily rely on empirical heuristics, lacking of convincing explanations. Some studies ( 96 ; 110 ) interpret UEs as attempts to create linear shortcuts that yield linear separability, leading models to overwhelmingly depend on spurious features. However, we find that linear classifiers trained on UEs achieve fair generalization (over 30% test accuracy on CIFAR-10), suggesting that interpreting UEs merely as linear shortcuts does not fully explain their much worse generalization in deep neural networks (10% on CIFAR-10, equivalent to random guessing levels). Besides, not all UEs are linearly separable, such as autoregressive poisons ( 75 ; 76 ) . Therefore, the prevailing explanation regarding the linear separability of UEs is incomplete in its applicability. There are still unclear of why UEs are effective, posing significant challenges to further enhancing UEs with better principled approaches. Unlearnable examples contain elaborately injected poisons and are therefore out-of-distribution. Recently, mutual information (MI) in representation learning, which quantifies the degree of correlation between random variables from two distributions, has gained widespread attention ( 68 ; 17 ) . This inspires us to use MI as a surrogate metric to evaluate the unlearnability of UE poisoned datasets. Specifically, we introduce a novel perspective: the reduction of mutual information, to elucidate the underlying mechanism of UEs. We evaluate both test accuracy drop and MI reduction between clean and poisoned features across many UEs, showcasing that effective UEs consistently decrease MI with clean features. Beyond exploring different UEs, we also test the decrease of MI across networks of varying depths. As networks become deeper, MI between features becomes smaller, resulting in test accuracy becomes lower. This consistent relationship between MI reduction and accuracy drop demonstrate our findings. Based on these analyses, we further enhance the efficacy of UEs by directly decreasing MI between the poisoned and clean distributions in the feature space. However, the complexity of estimating MI poses significant challenges for optimization ( 69 ; 60 ) . To tackle this issue, we prove from the perspective of covariance reduction that reducing MI can be achieved by minimizing the conditional covariance of the poisoned data’s intra-class features, and we then introduce a novel poisoning method called Mutual Information Unlearnable Examples (MI-UE) . Specifically, MI-UE optimizes a mutual information reduction loss that maximizes the cosine similarity among intra-class features for covariance reduction, while minimizing the cosine similarity between inter-class features to prevent class collapse. We conduct extensive experiments to validate that our MI-UE significantly outperforms the previous state-of-the-art UEs in reducing the model’s generalization ability. Remarkably, even under defenses such as adversarial training, MI-UE still achieves superior poisoning effects. 2 Related Work Unlearnable examples. Privacy issues have received extensive attention in privacy-preserving machine learning ( 79 ; 1 ; 80 ) , including studies on UEs ( 41 ) . Unlearnable examples are a type of data poisoning that allows an attacker to perturb the training dataset under a small norm restriction. These attacks aim to induce errors during test-time while maintaining the semantic integrity and ensuring the usage by legitimate users. A representative method in this field is the error-minimization poisoning strategy ( 41 ) , which employs iterative optimization of a bi-level minimization problem to generate poisoning noise by minimizing the loss. Following the initial work ( 41 ) , additional UE strategies ( 100 ; 96 ; 75 ; 33 ; 72 ; 56 ; 54 ; 109 ) have also been proposed. However, existing methods predominantly rely on empirical heuristics and lack a convincing framework to explain the efficacy of UEs, posing significant challenges for advancing UEs in a principled manner. Mutual information in machine learning. Mutual Information (MI) ( 78 ) serves as a metric for quantifying the dependency between two random variables and has been widely applied in machine learning ( 7 ; 12 ; 3 ; 34 ) . Recently, MI maximization in representation learning has gained widespread attention ( 68 ; 17 ) . People maximize MI to improve domain adaption ( 106 ) , and have achieved domain generalization by conducting MI regularization with pre-trained models ( 15