The Long-Term Effects of Data Selection in LLM Fine-Tuning

Yuxin Yang, Aoxiong Zeng, Xiangquan Yang · 2026

Read The Long-Term Effects of Data Selection in LLM Fine-Tuning by Yuxin Yang, Aoxiong Zeng, Xiangquan Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30537.txt.

Introduction

Yang, Zeng, and Yang ask whether picking training data for today's gains can hurt a language model tomorrow. Comparing six families of selection methods under one multi-stage protocol, they track not just immediate scores but later learning speed, forgetting, uneven skills, and robustness to unfamiliar data. Short-sighted picks often reverse rank: they lift the current stage while slowing later learning and deepening forgetting, a pattern the authors call myopic selection. They propose a long-horizon objective that weighs coverage and future transfer alongside immediate usefulness.

Catalog record

CREATORYuxin Yang, Aoxiong Zeng, Xiangquan Yang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35523
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.30537

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Long-Term Effects of Data Selection in LLM Fine-Tuning Yuxin Yang, Aoxiong Zeng, Xiangquan Yang Abstract Data selection is increasingly used to reduce the cost of large language model (LLM) fine-tuning, with recent methods prioritizing samples by current utility, diversity, quality, or influence. This paper studies a different question: when fine-tuning occurs over multiple stages, can selection strategies that look optimal now make the model less adaptable later? We introduce a long-horizon view of LLM data selection in which a selector is evaluated not only by immediate task performance, but also by future adaptation speed, forgetting, capability imbalance, and out-of-distribution robustness. We compare representative random, loss-based, gradient-based, diversity-based, quality-based, and utility-diversity selection families under a unified multi-stage protocol. Through controlled experiments designed to instantiate this protocol, we show how short-term selectors can exhibit rank reversal: they improve the current stage while slowing subsequent learning and increasing forgetting. We formalize this behavior as myopic selection , provide a simple local analysis of why it can occur, and propose a diagnostic Long-Horizon Aware Selection (LHAS) objective that augments immediate utility with coverage, future-proxy transfer, and anti-concentration terms. The study argues that data selection should be evaluated as a training intervention that shapes the model’s learning trajectory, rather than only as a local data-efficiency mechanism. 1 Introduction Supervised fine-tuning (SFT) is one of the standard ways to adapt large language models to downstream tasks ( 5 ; 1 ; 41 ; 4 ) . As instruction datasets grow in size and heterogeneity, training on every available example is often inefficient and sometimes harmful: low-quality examples can amplify bias, redundant examples waste compute, and overly narrow mixtures can overfit the model to a transient capability profile. These concerns have motivated data selection methods based on quality filtering, influence estimation, active learning, coreset coverage, online batch scoring, and deduplication or diversification ( 2 ; 35 ; 3 ; 10 ; 7 ; 51 ; 44 ; 20 ; 39 ; 52 ) . Most of this literature evaluates selection within a single fine-tuning stage. This is natural when the goal is to solve one task with a fixed budget, but it misses an important property of modern model adaptation: fine-tuning is often sequential. A deployed assistant may first be tuned on general instructions, then mathematical reasoning, then code, then safety data, and then domain-specific corpora. In this setting, the selected subset at stage t does not merely save compute. It changes the parameter state from which all future stages start. This paper asks whether efficient data selection can make models increasingly specialized. By specialization we do not mean only that a selector changes the label distribution of the current batch. We mean that selection may push representations, gradients, and parameter-efficient adapters toward a narrow set of capabilities, reducing future learnability or robustness. A sample can be highly useful for the current stage while still being a poor long-term training intervention. We call this phenomenon myopic selection : a selection policy is myopic when it maximizes immediate utility at the expense of future adaptation, retention, or out-of-distribution (OOD) robustness. This framing leads to five research questions. RQ1: Are short-term winning selectors also long-term winning selectors? RQ2: How do selectors affect the learning speed of later tasks? RQ3: Do they increase forgetting or capability imbalance? RQ4: Is diversity sufficient to prevent long-horizon bias? RQ5: Can a lightweight long-horizon objective improve the trade-off? The distinction matters because online selection methods are often evaluated with curves that stop at the end of the current stage. Such curves answer whether a method uses fewer tokens to solve today’s objective, but they do not answer whether it leaves the model in a state from which tomorrow’s objective is easier or harder. In particular, two selectors can reach the same current validation score while inducing very different update covariance, adapter subspaces, and capability coverage. A long-horizon evaluation therefore needs to measure both where the model arrives and how it arrived there. We make four contributions. First, we formulate long-horizon data selection for LLM fine-tuning and define metrics that capture future adaptation speed, forgetting, capability imbalance, OOD robustness, and a myopia gap . Second, we specify a unified protocol for comparing representative selection families under equal token budgets. Third, we give a simple theoretical analysis showing why two selectors with equal current-stage gain can differ in future adaptation cost. Fourth, we use controlled experiments to stress-test the protocol and introduce Long-Horizon Aware Selection (LHAS), a diagnostic baseline showing how coverage and anti-concentration terms can reduce the long-term side effects of myopic selection. 2 Related work Online data and batch selection for LLM fine-tuning. Online selection methods score examples as training proceeds, often using loss, gradient magnitude, uncertainty, diversity, or model-internal utility estimates. Classic online batch selection prioritizes high-loss examples ( 28 ; 17 ) , importance-sampling approaches use gradient information ( 18 ) , RHO-Loss emphasizes examples that are learnable and not yet learned ( 30 ) , and GREATS selects high-quality data in each training iteration ( 42 ) . Recent LLM-oriented work also studies influence-based instruction tuning and utility-diversity scoring ( 44 ; 52 ) . UDS is the closest point of departure for this work: it combines a utility term based on forward-pass logits with an inter-sample diversity estimate using a historical memory buffer ( 52 ) . Our goal is complementary. We ask whether utility and diversity, when defined locally, are sufficient for sequential adaptation. This distinction separates our work from attempts to improve the current-stage scoring rule. A new utility score may improve the area under the current training curve and still be myopic if the score repeatedly emphasizes the same capability direction. Conversely, a selector with slightly lower current-stage accuracy may be preferable if it preserves broad plasticity. Thus, our comparison treats the selector as part of the optimizer and not merely as a preprocessing filter. Data valuation, pruning, and curriculum learning. Data selection has a long history in active learning ( 36 ; 35 ; 3 ) , dataset cartography ( 38 ) , proxy-based selection ( 10 ) , data pruning ( 37 ) , and gradient- or loss-based example scoring ( 40 ; 32 ; 31 ) . In language-model training, importance resampling, deduplication, and diversification further show that the data mixture can change both efficiency and generalization ( 45 ; 20 ; 39 ) . Instruction tuning also highlights that small, carefully curated datasets can match or exceed much larger mixtures ( 51 ; 7 ) . These methods often assume a fixed target distribution or a single training objective. In multi-stage LLM fine-tuning, the target distribution itself evolves. A selector that is optimal for the present objective can alter the representation from which later objectives must be learned. Curriculum learning provides another useful analogy ( 43 ; 46 ) . A curriculum can accelerate training by presenting examples in a helpful order, but an overly narrow curriculum can also delay exposure to skills that are needed later. Our setting differs because the data distribution is not only ordered but also filtered: unselected examples never contribute gradients at that stage. The long-term effect is therefore stronger than reordering alone. Continual learning and stability-plasticity.

Read the complete text