The Finetuner’s Fallacy When to Pretrain with Your Finetuning Data
DatologyAI Team · 2026
Read The Finetuner’s Fallacy When to Pretrain with Your Finetuning Data by DatologyAI Team (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16177.txt.
Introduction
Baek and co-authors test a simple training strategy called specialized pretraining. The idea is to repeat a small domain dataset during pretraining, not save it for finetuning. They test it on ChemPile, MusicPile, and ProofPile. The method improved domain skill and kept general skill after finetuning. It also cut the tokens needed for a target score by up to 1.75 times. The authors derive scaling laws to pick the best repeat count.
Catalog record
| CREATOR | DatologyAI Team |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62657 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.16177
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Finetuner’s Fallacy When to Pretrain with Your Finetuning Data DatologyAI Team Abstract Real-world model deployments demand strong performance on narrow domains where data is often scarce. Typically, practitioners finetune models to specialize them, but this risks overfitting to the domain and forgetting general knowledge. We study a simple strategy, specialized pretraining (SPT), where a small domain dataset, typically reserved for finetuning, is repeated starting from pretraining as a fraction of the total tokens. Across three specialized domains (ChemPile, MusicPile, and ProofPile), SPT improves domain performance and preserves general capabilities after finetuning compared to standard pretraining. In our experiments, SPT reduces the pretraining tokens needed to reach a given domain performance by up to 1.75×. These gains grow when the target domain is underrepresented in the pretraining corpus: on domains far from web text, a 1B SPT model outperforms a 3B standard pretrained model. Beyond these empirical gains, we derive overfitting scaling laws to guide practitioners in selecting the optimal domain-data repetition for a given pretraining compute budget. Our observations reveal the finetuner’s fallacy : while finetuning may appear to be the cheapest path to domain adaptation, introducing specialized domain data during pretraining stretches its utility. SPT yields better specialized domain performance (via reduced overfitting across repeated exposures) and better general domain performance (via reduced forgetting during finetuning), ultimately achieving stronger results with fewer parameters and less total compute when amortized over inference. To get the most out of domain data, incorporate it as early in training as possible. Figure 1: Specialized pretraining (SPT) mixes the finetuning dataset into pretraining as a small fraction of tokens, repeating it 10–50× over the course of training. Compared to general pretraining (dashed), SPT (solid) achieves lower domain test loss (blue) and less forgetting of general knowledge (gold) throughout finetuning. For narrow domains, these gains can overcome differences in model scale. 1 Introduction Consider an organization with proprietary data such as support conversations, legal filings, or clinical notes, that wants to train a domain-specialized model. The conventional recipe is straightforward: start from a strong open-weights model pretrained on web-scale data, then finetune it on the proprietary dataset. Because this data is private and absent from public corpora, finetuning is treated as the natural mechanism for injecting missing domain knowledge. More broadly, modern training pipelines often treat pretraining and finetuning as disjoint phases: first learn general knowledge at scale, then specialize using a small curated dataset. The success of instruction tuning, RLHF, and parameter-efficient finetuning has further reinforced this view ( 41 ; 51 ; 22 ) . Growing evidence across several settings suggests that data encountered during pretraining shapes model behavior more durably than that introduced later: incorporating reasoning data into pretraining outperforms introducing it only during fine-tuning ( 1 ; 20 ) , unsafe behaviors learned during pretraining are harder to remove via post-training ( 31 ; 46 ) , and cross-language transfer during pretraining improves performance for low-resource languages ( 28 ) . Yet the question of when to introduce domain-specific data, and whether it should be mixed into pretraining rather than reserved for finetuning, remains largely unexplored. In this work, we question whether reserving all domain-specific data for the final stage of training is optimal. When the target domain is poorly represented in the pretraining corpus, introducing domain data only during finetuning may require large representational updates, leading to weaker generalization and greater forgetting of general knowledge. We study a simple alternative: interleave the domain dataset throughout pretraining as a small fraction of the training tokens (often repeating it up to 50 times), and then finetune on the same data as usual (Figure 1 ). We refer to this strategy as specialized pretraining (SPT). We observe that interleaving domain tokens with general data allows the model to tolerate far more repetitions before overfitting than it would during finetuning. Figure 2: Specialized pretraining ( SPT ) outperforms finetuning-only across domains. We pretrain models with a small fraction ( \delta ) of domain-specific tokens mixed into general web data, then finetune on the domain dataset. We plot the best post-finetuning domain loss across pretraining budgets for MusicPile, ChemPile, and ProofPile (300M tokens each). Even small domain mixtures ( \delta=1 – 5\% , blue curves) consistently outperform pretraining on general data alone ( \delta=0\% , gray) at all token scales. Across chemistry, symbolic music, and mathematical proofs, SPT consistently improves post-finetuning performance (§ 2 ). Relative to standard pretraining followed by finetuning, models trained with SPT achieve lower domain test loss, retain general pretraining knowledge more effectively during finetuning, and perform better on downstream tasks (Figure 2 ). Comparing across pretraining scales, SPT also requires substantially less pretraining compute to reach the same post-finetuning domain loss. Even replay ( 42 ) , a common strategy that reintroduces general data during continued pretraining, is not a substitute for early domain exposure: SPT’s advantage persists across all replay settings (§ 4 ). On domains far from web text, a 1B model trained with SPT outperforms a 3B model trained without domain data during pretraining. SPT also reduces the pretraining tokens needed to reach a given domain loss by up to 1.75 \times (Figure 2 ), and these loss improvements translate to downstream task accuracy: at 200B pretraining tokens, SPT improves MATH accuracy by up to 6 percentage points and MusicTheoryBench by up to 4 percentage points over the finetuning-only baseline (Figure 5 ). We further characterize when SPT is most effective. First, SPT helps most when the target domain is underrepresented in the pretraining corpus, as shown both in a controlled multilingual overlap study and across naturally occurring domain shifts (§ 3.1 ). Second, its benefit depends on the size of the domain dataset : for sufficiently large domain corpora, even repeating domain data from the beginning of pretraining is beneficial, while in more data-constrained settings, introducing domain data later in pretraining is preferable (§ 3.2 ). Third, the pretraining budget determines the optimal mixture fraction: larger domain fractions help at shorter training horizons, while smaller fractions become preferable as training lengthens to avoid overfitting from excessive repetition (§ 3.4 ). Finally, we derive overfitting scaling laws to model the total overfitting incurred by specialized pretraining and finetuning for different mixture fractions (§ 5 ), which include new expressions for modeling the effects of repeated data. Specifically, the test loss decomposes as the sum of two parts: the training loss w.r.t. pretraining tokens is modeled as a power law with a negative exponent, while the growing train-test gap is modeled as a power law with a positive exponent. This allows us to predict the optimal domain-data fraction for a given compute budget and forecast when aggressive mixing begins to hurt test loss, without running the full training sweep. In practice, this means practitioners can select the right SPT configuration from a small number of pilot runs rather than exhaustive search. Figure 3: The finetuner’s tax. Training a 1B model with specialized pretraining ( SPT ) costs more upfront than finetuning a 3B model on domain data alone, but the 3\times smaller model is cheaper to serve. The break-even point arrives after approximately 1 trillion inference tokens, after which SPT saves both compute and money while often