One Algorithm, Two Goals: Dual Scoring for Parameter and Data Selection in LLM Fine-Tuning
Xinrui Chen, Liu Yang · 2026
Read One Algorithm, Two Goals: Dual Scoring for Parameter and Data Selection in LLM Fine-Tuning by Xinrui Chen, Liu Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06166.txt.
Description
Xinrui Chen, Liu Yang, and Ou Wu unify parameter selection and data selection in language model fine tuning. Both choices cut fine tuning cost, yet separate scoring rules drive them, which wastes effort and blocks coordinated selection. When a parameter mask and a data subset jointly shape a restricted run, the split becomes a real obstacle.
The authors cast both choices as bilevel selection problems under one validation objective. A shared scoring rule then yields parameter importance and data utility as two views of a single gradient interaction matrix. Their one shot algorithm, DualSFT, extracts a mask and a subset from the same gradient statistics. On models of three to nine billion parameters, it gave a better joint trade off than sequential hybrid baselines at matched budgets.
Catalog record
| CREATOR | Xinrui Chen, Liu Yang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 125 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.06166