LEAD: Iterative Data Selection for Efficient LLM Instruction Tuning

Xiaotian Lin, Yanlin Qi, Yizhang Zhu, Themis Palpanas, Chengliang Chai, Nan Tang, Yuyu Luo ∗ · 2025

Read LEAD: Iterative Data Selection for Efficient LLM Instruction Tuning by Xiaotian Lin, Yanlin Qi, Yizhang Zhu, Themis Palpanas, Chengliang Chai, Nan Tang, Yuyu Luo ∗ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-07437.txt.

Description

Xiaotian Lin, Yuyu Luo, and five co-authors pick better training data for language models at low cost. Their paper, from twenty twenty-five, is called LEAD. Tuning a model on instructions works best with the right examples. Older methods score every example with the full model again and again, which is slow and dear.

LEAD scores examples inside the normal training loop. Its score mixes current loss, a guess at loss change, and a smoothed record of past loss. A coarse-to-fine search first picks rich groups, then fine examples. Tests on four data sets show strong gains with far less compute. The method makes repeat tuning cycles practical.

Catalog record

CREATORXiaotian Lin, Yanlin Qi, Yizhang Zhu, Themis Palpanas, Chengliang Chai, Nan Tang, Yuyu Luo ∗
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-07437

Browse the full library · About the Tower of Babel