Scaling Laws for Task-Specific LLM Distillation
Lavinia Ghita, Dhruv Desai, Ioana Boier · 2026
Read Scaling Laws for Task-Specific LLM Distillation by Lavinia Ghita, Dhruv Desai, Ioana Boier (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24747.txt.
Introduction
Lavinia Ghita, Dhruv Desai, and Ioana Boier derive scaling laws for shrinking large language models for one task area. Big models perform well but cost too much where speed and price matter. Distillation trains a small model to copy a large one. The authors ask how quality scales with pruning schedule, supervision format, compression ratio, and dataset size. Their test domain is quantitative finance, with headline classification as the concrete task.
Two distillation styles are compared under repeated structural pruning: logit based and LoRA based. A blended chain of thought loss steadies training over reasoning traces. In domain quality falls predictably as compression deepens. General knowledge benchmarks, by contrast, collapse much earlier. Supervision format drives the trade off, Ghita and colleagues find. Chain of thought supervision actively recovers general knowledge that pruning erases. The team releases its FinHeadlineMix dataset, the scaling results, and practical guidance for compression choices.
Catalog record
| CREATOR | Lavinia Ghita, Dhruv Desai, Ioana Boier |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 78 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.24747
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Scaling Laws for Task-Specific LLM Distillation Lavinia Ghita, Dhruv Desai, Ioana Boier Abstract Large Language Models (LLMs) achieve strong performance across a growing range of domains, yet their scale poses deployment challenges in applications where latency and cost constraints are critical. This paper derives empirical scaling laws for domain-specific LLM compression, quantifying how in-domain and general-knowledge performance scale with dataset size, compression ratio, supervision format, and iterative pruning schedule. Using quantitative finance as our application domain, we compare logit-based and LoRA-based distillation under iterative structural pruning, introducing a blended chain-of-thought supervision loss that stabilizes KL-divergence distillation over reasoning traces. In-domain task quality degrades predictably under compression while general-knowledge benchmarks collapse well before the same point; supervision format is the key driver of this tradeoff, with chain-of-thought supervision actively recovering general knowledge that pruning erases. We release the headline dataset FinHeadlineMix , scaling-law results, and practical recommendations to provide a reusable framework for domain-specific compression decisions. 1 Introduction We study knowledge distillation for compressing LLMs in a domain- and task-specific setting, quantifying the tradeoff between preserving in-domain task performance and retaining broad general-knowledge capabilities. This tradeoff is of direct interest to practitioners who need smaller, deployable models under latency and cost constraints. Concretely, the domain-specific task we use is a 35-class event classification problem over financial news headlines: a narrow, fixed-taxonomy mapping representative of production workloads. We compare multiple distillation methods and supervision formats (label-only versus chain-of-thought), apply iterative structural pruning and distillation, and derive empirical scaling laws across dataset size, compression ratio, supervision format, and pruning schedule. We provide practical guidance for choosing methods and compression levels. We find that supervision format governs this tradeoff: blended chain-of-thought supervision not only improves data efficiency but actively recovers general knowledge lost to pruning, while label-only methods either preserve the pruned level (LoRA) or degrade it further (direct-label KD). Iterative pruning and distillation compress the teacher to 16% of its parameters while retaining meaningful task quality; self-distillation baselines show that the training process itself accounts for roughly twice the performance gap introduced by compression. We ground our domain-specific experiments in quantitative finance, where these tradeoffs are acute. Synthetic annotation with LLMs scales the creation of labeled data [ 1 ] , letting practitioners iterate quickly on data sources and signal hypotheses. Production systems in finance are latency- and cost-sensitive, making compression essential and providing a natural testbed for studying it under realistic deployment constraints. Our methodology is domain-agnostic. LLM-based annotation shifts the bottleneck from labeling to deployment, motivating the compression study that follows. We focus on compression via fewer parameters: pruning [ 2 , 3 ] and knowledge distillation [ 4 ] . Recent work [ 5 , 6 , 3 ] shows these techniques can significantly reduce size while largely preserving accuracy. Quantization, sparsity, and neural architecture search (NAS) are outside our scope. We compare LoRA-based [ 7 , 8 ] and logit-based [ 3 ] distillation, each with single-label and chain-of-thought (CoT) [ 9 ] outputs. To stabilize KL-divergence distillation over reasoning traces, we introduce a blended supervision loss that independently weights label and CoT tokens (Section 4.2 ). To our knowledge, this is the first systematic comparison of these methods on decoder-only LLM students at multi-billion parameter scale. The rest of the paper is structured as follows: Sections 2 – 4 present related work, data, and methodology; Section 5 presents the experimental results and scaling laws; Section 6 covers the conclusions and a broader discussion aimed at practitioners. The Appendix gives implementation details, dataset description, and supplementary results. We release the dataset used in our study. 2 Related Work Knowledge distillation was introduced as a compression method in [ 10 ] , and later formalized in its modern form by [ 4 ] . This foundational approach distinguishes between hard distillation, where the student learns from the teacher’s argmax labels, and soft distillation, where the student matches the teacher’s full output distribution via softened logits. An extension using chain-of-thought (CoT) [ 9 ] as supervision is discussed in [ 11 ] , showing that richer teacher supervision can match label-only distillation performance with significantly less training data. At the sequence level, [ 12 ] propose aligning the student’s output distribution with the teacher’s across entire sequences rather than individual tokens. We combine both ideas: sequence-level KL divergence with CoT supervision, weighted to stabilize label learning. In a line of work related to ours, [ 13 ] compare distilled student performance under different architectures and find that larger or more accurate teachers do not necessarily produce better students. They conclude that student capacity is the limiting factor, more so than teacher size. Relatedly, [ 14 ] show that even poor-quality teachers can produce effective students, further suggesting that teacher size is not the binding constraint. However, these observations are made on models much smaller than those studied in our paper and only cover computer vision applications. The topic of optimal compute allocation for distillation and the resulting scaling laws are analyzed in [ 15 ] , showing that distillation outperforms supervised pretraining up to a compute level that grows with student size. A statistical perspective on why distillation helps is presented in [ 16 ] , establishing a bias-variance tradeoff that explains how the teacher’s class-probability estimates aid student learning. Practical considerations for distillation at scale include online co-distillation for faster training via parallelism [ 17 ] and identification of key design choices for distillation quality [ 18 ] , both at order-of-magnitude smaller models than ours. A complementary line of work studies on-policy KL distillation for language models, where the student samples its own outputs and the teacher scores them, addressing the train-inference exposure gap inherent in offline KL training [ 19 ] . More recently, distillation has become widely used in creating LLM families of different sizes, where the largest model is trained from scratch and acts as teacher to smaller variants [ 3 , 20 , 21 , 6 , 5 ] . While distillation can be applied across different model families and tokenizers [ 6 ] , a common approach is to prune a larger model and then distill knowledge back into the pruned student. An extensive analysis of iterative pruning and knowledge distillation is presented in [ 3 ] , which directly inspires our methodology. From a complementary angle, [ 22 ] show that the benefit of depth and width at training time is contingent on the target being approximable by shallower networks. We study how depth and width matter when compressing already-trained models via pruning and distillation. Unlike [ 15 ] , who study scaling with compute allocation between teacher and student, we vary dataset size, compression ratio, and supervision format and report how in-domain and general-knowledge performance scale. Relative to [ 3 ] , we add systematic comparison across data regimes and distillation methods (LoRA vs. logit-based, label vs. CoT). Work on compression fragility shows that difficult downstream tasks are more sensitive to pruning and that damage from aggressive compression can be irreparable even with continued training [ 23 ] . We report analogous sensitivity but vary data size and distillation method rather than compression level and task difficulty. Recent research demonstrates that LLMs