MIG: Automatic Data Selection for Instruction Tuning by Maximizing Information Gain in Semantic Space

Yicheng Chen, Yining Li, Kai Hu, Zerun Ma, Haochen Ye, Kai Chen · 2025

Read MIG: Automatic Data Selection for Instruction Tuning by Maximizing Information Gain in Semantic Space by Yicheng Chen, Yining Li, Kai Hu, Zerun Ma, Haochen Ye, Kai Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-13835.txt.

Description

Yicheng Chen and five co-authors choose instruction-tuning data by measuring what the whole set teaches. Their paper appeared in two thousand twenty-five. Open instruction datasets are vast and uneven. Picking a good subset beats training on all of it. Older pickers score single examples and use rough diversity rules. Embedding distances miss the intent of complex instructions. A global view is missing.

Their method, MIG, builds a label graph over the semantic space. Diversity is scored from how labels spread across that graph. Selection then maximizes total information gain. Quality and coverage are judged together, not in turns. The subsets train models that punch above their size. Data curation gains a principled yardstick.

Catalog record

CREATORYicheng Chen, Yining Li, Kai Hu, Zerun Ma, Haochen Ye, Kai Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-13835

Browse the full library · About the Tower of Babel