Influence-Preserving Proxies for Gradient-Based Data Selection in LLM Fine-tuning

Sirui Chen, Yunzhe Qi, Mengting Ai, Yifan Sun, Ruizhong Qiu, Jiaru Zou, Jingrui HeUniversity of Illinois Urbana-Champaign {sirui6, yunzheq2, mai10, yifan50, rq5, jiaruz2, jingrui}@illinois.edu · 2026

Read Influence-Preserving Proxies for Gradient-Based Data Selection in LLM Fine-tuning by Sirui Chen, Yunzhe Qi, Mengting Ai, Yifan Sun, Ruizhong Qiu, Jiaru Zou, Jingrui HeUniversity of Illinois Urbana-Champaign {sirui6, yunzheq2, mai10, yifan50, rq5, jiaruz2, jingrui}@illinois.edu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-17835.txt.

Description

Sirui Chen and team introduce Iprox, a two-stage frame for influence-preserving proxies in gradient-based data selection for language model fine-tuning. Supervised fine-tuning hinges on picking training data that most helps downstream scores. Gradient-based methods such as TracIn and Influence Functions spot useful samples, yet scale poorly for multi-billion-parameter models.

Off-the-shelf smaller models serve as common proxies, but they stay suboptimal. Their learning motion is unclear, their sizes cannot flex, and they cannot align further with the target model on influence. Iprox derives proxies from the target model. A low-rank compression stage preserves influence information. An aligning stage aligns gradients and logits, controlling cost while retaining target influence. Tests show Iprox beating off-the-shelf proxies and baselines. On Qwen3-4B, a smaller Iprox proxy beats a larger off-the-shelf one. On Llama3.2, Iprox beats baselines while halving compute.

Catalog record

CREATORSirui Chen, Yunzhe Qi, Mengting Ai, Yifan Sun, Ruizhong Qiu, Jiaru Zou, Jingrui HeUniversity of Illinois Urbana-Champaign {sirui6, yunzheq2, mai10, yifan50, rq5, jiaruz2, jingrui}@illinois.edu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE83 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-17835

Browse the full library · About the Tower of Babel