Data Pruning by Information Maximization

Haoru Tan, Sitong Wu, Wei Huang, Shizhen Zhao, Xiaojuan Qi · 2025

Read Data Pruning by Information Maximization by Haoru Tan, Sitong Wu, Wei Huang, Shizhen Zhao, Xiaojuan Qi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01701.txt.

Description

Haoru Tan and co-authors prune training data by maximizing information in a paper from two thousand twenty-five. Big data sets carry many near-duplicate samples that add little. Picking a small, rich subset, called a coreset, can cut training cost without hurting results.

Their method, InfoMax, scores each sample by its influence or difficulty, then penalizes pairs that look alike. The pick is framed as an optimization task and solved with a fast gradient method plus tricks that scale to millions of samples. Tests covered image sorting, vision-language pre-training, and tuning large language models. InfoMax beat rival pruning methods across these tasks, the authors report.

Catalog record

CREATORHaoru Tan, Sitong Wu, Wei Huang, Shizhen Zhao, Xiaojuan Qi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-01701

Browse the full library · About the Tower of Babel