Data Pruning by Information Maximization
Haoru Tan, Sitong Wu, Wei Huang, Shizhen Zhao, Xiaojuan Qi · 2025
Read Data Pruning by Information Maximization by Haoru Tan, Sitong Wu, Wei Huang, Shizhen Zhao, Xiaojuan Qi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01701.txt.
Description
Haoru Tan and co-authors prune training data by maximizing information in a paper from two thousand twenty-five. Big data sets carry many near-duplicate samples that add little. Picking a small, rich subset, called a coreset, can cut training cost without hurting results.
Their method, InfoMax, scores each sample by its influence or difficulty, then penalizes pairs that look alike. The pick is framed as an optimization task and solved with a fast gradient method plus tricks that scale to millions of samples. Tests covered image sorting, vision-language pre-training, and tuning large language models. InfoMax beat rival pruning methods across these tasks, the authors report.
Catalog record
| CREATOR | Haoru Tan, Sitong Wu, Wei Huang, Shizhen Zhao, Xiaojuan Qi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01701