One Algorithm, Two Goals: Dual Scoring for Parameter and Data Selection in LLM Fine-Tuning

Xinrui Chen, Liu Yang · 2026

Read One Algorithm, Two Goals: Dual Scoring for Parameter and Data Selection in LLM Fine-Tuning by Xinrui Chen, Liu Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06166.txt.

Description

Xinrui Chen, Liu Yang, and Ou Wu unify parameter selection and data selection in language model fine tuning. Both choices cut fine tuning cost, yet separate scoring rules drive them, which wastes effort and blocks coordinated selection. When a parameter mask and a data subset jointly shape a restricted run, the split becomes a real obstacle.

The authors cast both choices as bilevel selection problems under one validation objective. A shared scoring rule then yields parameter importance and data utility as two views of a single gradient interaction matrix. Their one shot algorithm, DualSFT, extracts a mask and a subset from the same gradient statistics. On models of three to nine billion parameters, it gave a better joint trade off than sequential hybrid baselines at matched budgets.

Catalog record

CREATORXinrui Chen, Liu Yang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE125 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.06166

Browse the full library · About the Tower of Babel