Distill on a Diet: Efficient Knowledge Distillation via Learnable Data Pruning ∗ * Equal contribution. ✉ Corresponding authors.

Yifan Wu *, Yiqi Wang *, Xichen Ye *, Wenjing Yan, Xiaoqiang Li (✉), Cheng Jin, Xiangyu Yue, Weizhong Zhang (✉) · 2026

Read Distill on a Diet: Efficient Knowledge Distillation via Learnable Data Pruning ∗ * Equal contribution. ✉ Corresponding authors. by Yifan Wu *, Yiqi Wang *, Xichen Ye *, Wenjing Yan, Xiaoqiang Li (✉), Cheng Jin, Xiangyu Yue, Weizhong Zhang (✉) (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-25488.txt.

Description

Distilling a big teacher model into a small student usually trains on the full dataset, so the savings arrive only after an expensive process. Yifan Wu and colleagues ask in 2026 whether a pruned dataset can distil a better student with less data and less compute. Their framework, IF-Beta, first scores each training example's influence on distillation outcomes, then learns a sampling policy shaped like a Beta distribution that decides how likely each example is to be kept.

A two level objective trains the policy: the inner loop practises quickly inside the teacher's feature space, while the outer loop judges real distillation performance. On standard image benchmarks the method beat other pruning baselines across removal rates. Students trained on the pruned sets sometimes surpassed students distilled on everything, suggesting much full dataset training is redundant for knowledge transfer.

Catalog record

CREATORYifan Wu *, Yiqi Wang *, Xichen Ye *, Wenjing Yan, Xiaoqiang Li (✉), Cheng Jin, Xiangyu Yue, Weizhong Zhang (✉)
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE88 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.25488

Browse the full library · About the Tower of Babel