LEAD: Iterative Data Selection for Efficient LLM Instruction Tuning
Xiaotian Lin, Yanlin Qi, Yizhang Zhu, Themis Palpanas, Chengliang Chai, Nan Tang, Yuyu Luo ∗ · 2025
Read LEAD: Iterative Data Selection for Efficient LLM Instruction Tuning by Xiaotian Lin, Yanlin Qi, Yizhang Zhu, Themis Palpanas, Chengliang Chai, Nan Tang, Yuyu Luo ∗ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-07437.txt.
Description
Xiaotian Lin, Yuyu Luo, and five co-authors pick better training data for language models at low cost. Their paper, from twenty twenty-five, is called LEAD. Tuning a model on instructions works best with the right examples. Older methods score every example with the full model again and again, which is slow and dear.
LEAD scores examples inside the normal training loop. Its score mixes current loss, a guess at loss change, and a smoothed record of past loss. A coarse-to-fine search first picks rich groups, then fine examples. Tests on four data sets show strong gains with far less compute. The method makes repeat tuning cycles practical.
Catalog record
| CREATOR | Xiaotian Lin, Yanlin Qi, Yizhang Zhu, Themis Palpanas, Chengliang Chai, Nan Tang, Yuyu Luo ∗ |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 71 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-07437