Compute Optimal Scaling of Skills: Knowledge vs Reasoning

Nicholas Roberts Niladri Chatterji Sharan Narang Mike Lewis Dieuwke Hupkes · 2025

Read Compute Optimal Scaling of Skills: Knowledge vs Reasoning by Nicholas Roberts Niladri Chatterji Sharan Narang Mike Lewis Dieuwke Hupkes (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-10061.txt.

Introduction

Nicholas Roberts, Niladri Chatterji, Sharan Narang, Mike Lewis, and Dieuwke Hupkes published this paper in 2025. It asks whether the famous scaling laws of language models, the rules trading model size against training data, apply equally to every skill. Their answer is no: knowledge and reasoning scale differently.

Testing knowledge-based question answering against code generation, the team found distinct compute-optimal recipes for each. Extensive data-mix experiments showed the difference is not an artifact of what the models were fed; it persists after correcting for the mix. The practical bite comes in validation: choosing a test set heavy in one skill can shift the computed optimal model size by nearly half. Teams that validate on the wrong skill mix will build the wrong model. The authors urge skill-aware scaling studies and validation sets that mirror the deployment purpose, rather than one blended score for everything.

Catalog record

CREATORNicholas Roberts Niladri Chatterji Sharan Narang Mike Lewis Dieuwke Hupkes
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-10061

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Compute Optimal Scaling of Skills: Knowledge vs Reasoning Nicholas Roberts Niladri Chatterji Sharan Narang Mike Lewis Dieuwke Hupkes Abstract Scaling laws are a critical component of the LLM development pipeline, most famously as a way to forecast training decisions such as ‘compute-optimally’ trading-off parameter count and dataset size, alongside a more recent growing list of other crucial decisions. In this work, we ask whether compute-optimal scaling behaviour can be skill-dependent. In particular, we examine knowledge and reasoning-based skills such as knowledge-based QA and code generation, and we answer this question in the affirmative: scaling laws are skill-dependent. Next, to understand whether skill-dependent scaling is an artefact of the pretraining datamix, we conduct an extensive ablation of different datamixes and find that, also when correcting for datamix differences, knowledge and code exhibit fundamental differences in scaling behaviour. We conclude with an analysis of how our findings relate to standard compute-optimal scaling using a validation set, and find that a misspecified validation set can impact compute-optimal parameter count by nearly 50%, depending on its skill composition. (a) Knowledge QA. (b) Code. (c) Data- vs capacity-hunger. Figure 1: Isoflop curves and COs for code and knowledge QA, along with the APE COs. In (a), we see that on average across held-out datasets, knowledge QA tends to be capacity-hungry compared to the APE CO. On the other hand, in (b), we see that code tends to be data-hungry relative to the APE CO. We show the distribution of these relationships in (c), where we plot distributions of the log-scale differences in parameter count between skill-dependent COs and APE COs and find their means lie on opposite sides of the APE CO. The black curves in (a) and (b) represent the predicted APE COs from Dubey et al. (2024) , mapped to their respective IsoFLOP groups. 1 Introduction Used both to forecast performance for early pretraining decisions as well as decide on the optimal trade-off between parameters and pretraining dataset size given a particular compute budget, scaling laws ( Kaplan et al., 2020 , i.a.) have played an important role in the development of large language models (LLMs). Famously, with a series of experiments with models with different data/parameter trade-offs, Hoffmann et al. (2022) showed that previous LLMs were all erring on the side of too many parameters, causing a shift in the amount of training tokens to train LLMs. More recently, Dubey et al. (2024) used scaling laws not only to determine the optimal parameter count given their available compute budget, but also to forecast the impact of data selection decisions on evaluation scores. In these works, the compute optima (COs), describing the optimal parameter count and number of training tokens, are selected based on aggregate performance estimators (APEs), in the form of negative log-likelihood (NLL) on a validation set not part of the pretraining corpus. Little is known, however, about whether the COs of individual skills such as mathematical reasoning, question answering (QA), or coding, align with these APE COs. While some studies use scaling laws to predict how downstream task performance improves with scale ( Ye et al., 2025 ; Held et al., 2025 , e.g.) , none of these studies cover whether COs themselves may be skill dependent. Is it possible that some skills are more data-hungry , whereas others benefit more from extra parameters ? If so, how should that impact model training and training data selection? In this paper, with an extensive set of experiments across 9 different compute scales and 2 skills as measured with 19 datasets across two different splits, we study exactly that. Specifically, we focus on the three research questions: R1. Are COs skill dependent? First, we consider how the IsoFLOP curves and corresponding COs for code 1 1 1 We use code as a proxy for reasoning skills to avoid ambiguity and the challenge of concretely defining reasoning. - and knowledge -based skills compare to the APE COs, given a canonical datamix. Across the board, we find pronounced differences between the COs for these different skills: where knowledge QA tasks are capacity-hungry, code tasks instead prefer data. R2. Is this an artefact of the pretraining datamix, or are code and knowledge skills fundamentally different? Using only the smallest compute scale, we next investigate whether these difference are a consequence of the proportion of skill-relevant data in the pretraining datamix of our experiments or that there is in fact a difference in how data- or capacity-hungry the different skills are. We find that both are true: changing the proportion of skill-relevant data shifts the CO for that skill, but COs for skills differ even with comparable proportions of skill specific data. R3. How does the existence of skill-dependent COs impact LLM training? Lastly, we focus on the more practical question of how these findings should impact model pretraining. First, again using only the smallest compute scale, we investigate if it is possible to align the COs for the two skills under consideration. We find that it is, but that whether this should be considered advantageous for overall model training depends highly on the choice of validation set. We then investigate how much the estimated optimal parameter count depends on the choice of validation set. We find that, on our smallest scale experiments ( 6\times 10^{18} ), the optimal parameter count for the same skill varies by almost 50% across datamixes, and by more than 30% across validation sets. At larger scale, the difference is smaller, but it still exceeds 10% for the three largest compute scales. These experiments show that choosing a validation set that adequately represents what the final model should capture is critical to finding the right parameter counts. Outline With our work, we contribute to an empirical understanding of how different skills behave across model scales, and explore the concrete implications for CO training of LLMs. In the remainder of this paper, we first provide the necessary background and definitions to understand our work ( § 2 ), and we describe the data we use for our experiments ( § 3 ). Next, in § 4 , we describe our experiments and their results. Lastly, we discuss related works about scaling and data selection ( § 5 ), and we conclude in § 6 . 2 Background When computing scaling laws, prior work typically focused on modelling the loss on a validation set, which is usually a general text corpus drawn from the same distribution as the pretraining dataset. In this setup, the loss is effectively a weighted combination of all what the model should learn during training. In contrast, we consider skill-dependent scaling laws. In this section, we discuss the relevant background and notions of scaling laws, COs, and skills underpinning our analysis. In § 5 , we describe related efforts in more detail, and in Appendix C , we provide formal definitions of the concepts that this section introduces. 2.1 Scaling laws Neural scaling laws aim to model and predict loss values as a function of the compute scale in FLOPs. FLOPs are often estimated as B\approx 6pt , which involves contributions from the size of the training set in tokens, t , and the parameter count of the model,

Read the complete text