Distill on a Diet: Efficient Knowledge Distillation via Learnable Data Pruning ∗ * Equal contribution. ✉ Corresponding authors.
Yifan Wu *, Yiqi Wang *, Xichen Ye *, Wenjing Yan, Xiaoqiang Li (✉), Cheng Jin, Xiangyu Yue, Weizhong Zhang (✉) · 2026
Read Distill on a Diet: Efficient Knowledge Distillation via Learnable Data Pruning ∗ * Equal contribution. ✉ Corresponding authors. by Yifan Wu *, Yiqi Wang *, Xichen Ye *, Wenjing Yan, Xiaoqiang Li (✉), Cheng Jin, Xiangyu Yue, Weizhong Zhang (✉) (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-25488.txt.
Description
Distilling a big teacher model into a small student usually trains on the full dataset, so the savings arrive only after an expensive process. Yifan Wu and colleagues ask in 2026 whether a pruned dataset can distil a better student with less data and less compute. Their framework, IF-Beta, first scores each training example's influence on distillation outcomes, then learns a sampling policy shaped like a Beta distribution that decides how likely each example is to be kept.
A two level objective trains the policy: the inner loop practises quickly inside the teacher's feature space, while the outer loop judges real distillation performance. On standard image benchmarks the method beat other pruning baselines across removal rates. Students trained on the pruned sets sometimes surpassed students distilled on everything, suggesting much full dataset training is redundant for knowledge transfer.
Catalog record
| CREATOR | Yifan Wu *, Yiqi Wang *, Xichen Ye *, Wenjing Yan, Xiaoqiang Li (✉), Cheng Jin, Xiangyu Yue, Weizhong Zhang (✉) |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 88 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.25488