When Data Is Scarce: Scaling Sparse Language Models with Repeated Training

Boqian Wu, Qiao Xiao, Patrik Okanovic, Tomasz Sternal, Maurice van Keulen, Mykola Pechenizkiy, Elena Mocanu, Torsten Hoefler, Decebal Constantin Mocanu · 2026

Read When Data Is Scarce: Scaling Sparse Language Models with Repeated Training by Boqian Wu, Qiao Xiao, Patrik Okanovic, Tomasz Sternal, Maurice van Keulen, Mykola Pechenizkiy, Elena Mocanu, Torsten Hoefler, Decebal Constantin Mocanu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01155.txt.

Introduction

Wu and co-authors ask how sparse language models scale when data runs out in this 2026 paper. Dense scaling laws assume endless fresh text. Real labs face finite corpora and must retrain on the same tokens for many epochs. How sparsity interacts with that repetition was unmapped. Their study spans models near two billion parameters and sparsity past ninety percent.

Data budgets run to billions of unique tokens, with up to 16 epochs of repeats. They fit a scaling law with four inputs: active parameters, unique tokens, repetition, and sparsity. It predicts loss across compute and data budgets with care. Sparse training delays the point where repeated data stops paying, they find. With fixed data, the loss best sparsity is moderate, near half. The compute best sparsity runs higher and grows with data scale. Sparsity, they argue, is a scaling lever, not just an efficiency trick.

Catalog record

CREATORBoqian Wu, Qiao Xiao, Patrik Okanovic, Tomasz Sternal, Maurice van Keulen, Mykola Pechenizkiy, Elena Mocanu, Torsten Hoefler, Decebal Constantin Mocanu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.01155

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When Data Is Scarce: Scaling Sparse Language Models with Repeated Training Boqian Wu, Qiao Xiao, Patrik Okanovic, Tomasz Sternal, Maurice van Keulen, Mykola Pechenizkiy, Elena Mocanu, Torsten Hoefler, Decebal Constantin Mocanu Abstract Scaling laws for dense LLMs under infinite data are well explored, but how sparsity interacts with limited data is not. In this work, we study sparse training in data-constrained regimes where limited unique tokens require multi-epoch training. Our experiments span models up to 1.92B parameters in the fitting set, sparsity up to 93.75%, unique data budgets up to 2.6B tokens, and total training tokens up to 41.6B over 16 epochs; we further validate extrapolation on held-out dense-equivalent models up to 7.68B parameters. We find that: ① Sparse scaling in data-limited settings: We introduce a scaling law that models loss as a function of active parameters, unique tokens, data repetition, and sparsity, accurately predicting performance across compute and data budgets. ② Delayed data saturation: sparse training postpones diminishing returns from repeated data, making multi-epoch training more effective. ③ Resource trade-offs: With fixed data, loss-optimal sparsity is moderate ( ∼ 50 % \sim 50\% ), while compute-optimal sparsity is higher and grows with data scale. Overall, sparsity is not just a tool for efficiency, but a mechanism for improving scaling trade-offs under data scarcity. Our code is available at: https://github.com/boqian333/sparse-dc-scaling . 1 Introduction Scaling has been central to the recent success of large language models (LLMs) ( 17 ; 7 ; 14 ; 31 ; 32 ; 12 ; 15 ; 1 ; 6 ; 3 ; 39 ) . While claims about the “death” of AI scaling laws remain actively debated 1 1 1 See, e.g., public discussions on the potential limits and future evolution of scaling laws in widely read blogs ( 29 ; 20 ) and in coverage by major news outlets ( 25 ; 33 ) . , growing evidence suggests that scaling behavior continues to evolve rather than simply saturate ( 8 ; 27 ) . A key driver of this evolution is the growing mismatch between data availability and computation. While the supply of high-quality human data increases at a modest rate of approximately 1.03 \times per year, the data used for pre-training has grown much faster, at roughly 4 \times per year ( 28 ; 34 ) . As a result, the exhaustion of high-quality data has ushered in a data-constrained pre-training regime ( 35 ; 23 ) , fundamentally reshaping the landscape of LLM scaling. More broadly, this shift highlights a central research challenge: how to enhance model capabilities using a finite amount of high-quality data, rather than relying solely on increasing data volume. Under data-constrained regimes, commonly adopted approaches for improving model performance, such as allocating additional compute to increase model size or repeating training data, often exhibit diminishing returns and incur substantial computational costs ( 23 ; 18 ) . Although stronger regularization, model scaling, and ensembling can improve data efficiency in data-constrained pre-training ( 18 ) , they largely operate within the conventional dense pre-training paradigm and do not fundamentally alter the underlying architecture or introduce a qualitatively different scaling mechanism. This distinction motivates a broader question: under finite data, is scaling constrained only by the amount of model capacity, or also by how that capacity is distributed and optimized? In dense training, all parameters are updated at every step on the same limited data, which can accelerate saturation under repeated exposure. This raises the possibility that scaling efficiency may be improved by expanding the total parameter space while keeping per-token compute, determined by the number of non-zero parameters, fixed or even reduced. Such a perspective challenges the assumption of fully dense pre-training and motivates the use of sparsity ( 21 ; 10 ; 9 ; 13 ; 36 ) . Sparsity, however, can be introduced in different ways. Architectural sparsity, such as in Mixture-of-Experts (MoE) models ( 30 ) , increases model capacity through conditional activation, but largely preserves dense optimization within each expert. In contrast, sparse training methods, such as Dynamic Sparse Training (DST) ( 21 ; 5 ; 22 ; 9 ; 41 ; 19 ; 38 ; 37 ; 36 ) , impose sparsity directly during optimization by updating only a subset of parameters and dynamically evolving the network structure. Architectural sparsity alone does not change how often the same parameters are optimized, whereas parameter-level sparse optimization redistributes updates across a larger pool of weights. Nevertheless, how sparse optimization reshapes the scaling behavior of LLM pre-training under data-constrained regimes remains insufficiently understood. In this study, we seek to develop a systematic understanding of how DST scales in data-constrained LLM pre-training by formulating and analyzing its scaling behavior. To make this investigation tractable, we structure our investigation around three key questions: Sparse scaling law under data-constrained regimes. How does pre-training performance scale with model size and data repetition across different sparsity levels under data constraints? This question is addressed in Section 3 . Allocation. How does sparsity influence the optimal trade-off between model size and data repetition under fixed data constraints and compute budgets? This trade-off is analyzed in Section 4 . Optimal sparsity. Given a fixed data budget, how can we determine the optimal sparsity level for efficient pre-training? The optimal sparsity strategy is studied in Section 5 . To answer these questions, we conduct experiments across sparsity, model size, data budget, and training duration, covering sparsity up to 93.75%, unique data budgets up to 2.6B tokens, and total training tokens up to 41.6B over 16 epochs. We fit scaling laws on models up to 1.92B dense-equivalent parameters and validate extrapolation on held-out models up to 7.68B parameters, leading to the following key findings: ① Sparse training improves data utilization efficiency. We find that sparsity increases the data saturation scale R_{d}^{*}(S) , which governs the decay rate and asymptotic headroom of effective data under scaling. R_{d}^{*}(S) rises from 4.4 at S=0 to about 6.90 at S=0.5 , implying slower saturation and a larger effective-data ceiling for sparse models. ② Sparse training improves parameter efficiency. Compared to dense training, sparse training benefits relatively more from allocating additional compute to longer training rather than to increasing parameter count. This leads to up to a 3\times smaller parameter-to-token ratio at a 1.3B-token data budget and compute on the order of 10^{20} FLOPs, improving inference efficiency. ③ Compute-optimal sparsity enables cheaper large-scale training. At compute-optimal sparsity, sparse models match dense-model accuracy while using approximately 8\times to 10\times fewer training FLOPs. To our knowledge, we provide the first systematic study of DST under data-constrained scaling, showing that sparsity can improve scaling efficiency when data is limited. Figure 1 : Sparsity–capacity trade-off in the data-constrained regime. Validation loss as a function of sparsity ( S ) for models with different numbers of non-zero parameters ( N=15 M, 30M, 60M, 120M) and different unique-token budgets U_{d} . 2 Background and Motivation Scaling laws ( 17 ; 14 ; 42 ; 23 ; 11 ) have played a central role in guiding the training of large language models by prescribing how compute should be allocated between model size and training data. 2.1 Chinchilla scaling law A widely adopted model of dense scaling behavior

Read the complete text