MIG: Automatic Data Selection for Instruction Tuning by Maximizing Information Gain in Semantic Space
Yicheng Chen, Yining Li, Kai Hu, Zerun Ma, Haochen Ye, Kai Chen · 2025
Read MIG: Automatic Data Selection for Instruction Tuning by Maximizing Information Gain in Semantic Space by Yicheng Chen, Yining Li, Kai Hu, Zerun Ma, Haochen Ye, Kai Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-13835.txt.
Description
Yicheng Chen and five co-authors choose instruction-tuning data by measuring what the whole set teaches. Their paper appeared in two thousand twenty-five. Open instruction datasets are vast and uneven. Picking a good subset beats training on all of it. Older pickers score single examples and use rough diversity rules. Embedding distances miss the intent of complex instructions. A global view is missing.
Their method, MIG, builds a label graph over the semantic space. Diversity is scored from how labels spread across that graph. Selection then maximizes total information gain. Quality and coverage are judged together, not in turns. The subsets train models that punch above their size. Data curation gains a principled yardstick.
Catalog record
| CREATOR | Yicheng Chen, Yining Li, Kai Hu, Zerun Ma, Haochen Ye, Kai Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-13835