A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone

Jitai Hao, Qiang Huang, Hao Liu, Xinyan Xiao, Zhaochun Ren, Jun Yu · 2025

Read A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone by Jitai Hao, Qiang Huang, Hao Liu, Xinyan Xiao, Zhaochun Ren, Jun Yu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-12781.txt.

Introduction

Jitai Hao and co-authors clone big language models into small ones cheaply in a 2025 paper. Small models still cost much to train, even when distilled or pruned from large teachers. Hard pruning loses facts, alignment steps waste effort, and rich inner signals go unused.

Their low-rank clone trains small projection matrices that do two jobs at once: they soften-prune teacher weights and copy teacher activations, including feed-forward signals, into the student. No extra alignment parts are needed. The student matched top models trained on trillions of tokens while using twenty billion. That is over a thousandfold saving in training tokens.

Catalog record

CREATORJitai Hao, Qiang Huang, Hao Liu, Xinyan Xiao, Zhaochun Ren, Jun Yu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-12781

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone Jitai Hao, Qiang Huang, Hao Liu, Xinyan Xiao, Zhaochun Ren, Jun Yu Abstract Training high-performing Small Language Models (SLMs) remains costly, even with knowledge distillation and pruning from larger teacher models. Existing work often faces three key challenges: (1) information loss from hard pruning, (2) inefficient alignment of representations, and (3) underutilization of informative activations, particularly from Feed-Forward Networks (FFNs). To address these challenges, we introduce Low-Rank Clone (LRC) , an efficient pre-training method that constructs SLMs aspiring to behavioral equivalence with strong teacher models. LRC trains a set of low-rank projection matrices that jointly enable soft pruning by compressing teacher weights, and activation clone by aligning student activations, including FFN signals, with those of the teacher. This unified design maximizes knowledge transfer while removing the need for explicit alignment modules. Extensive experiments with open-source teachers (e.g., Llama-3.2-3B-Instruct, Qwen2.5-3B/7B-Instruct) show that LRC matches or surpasses state-of-the-art models trained on trillions of tokens–while using only 20B tokens, achieving over 1,000 × \times training efficiency. Our codes and model checkpoints are available at GitHub and Huggingface . 1 Introduction Figure 1: LRC results that achieve higher accuracy with 1,000 \times fewer training tokens, significantly boosting efficiency. Large Language Models (LLMs) have shown exceptional performance across a wide range of Natural Language Processing (NLP) tasks achiam2023gpt4 ; yang2024qwen2_5 ; yang2025qwen3technicalreport ; grattafiori2024llama3 ; guo2025deepseek_r1 . However, their deployment remains limited in real-world applications due to their immense computational and memory requirements, making them unsuitable for latency-sensitive, edge-based, or privacy-preserving scenarios. As a result, there is growing momentum toward developing Small Language Models (SLMs) that offer similar capabilities with significantly lower resource footprints. Yet, despite their smaller size, training high-performing SLMs remains a resource-intensive endeavor. For example, state-of-the-art SLMs such as Llama-3.2-3B llama3.2-3b-instruct and Qwen3-1.7B yang2024qwen2_5 require 9 and 36 trillion tokens, respectively, for pre-training. To reduce such substantial costs, knowledge distillation hinton2015distilling has emerged as a key strategy, enabling a compact student model to learn from a larger and more powerful teacher model li2023synthetic ; muralidharan2024compact ; gu2024minillm ; grattafiori2024llama3 . Recent efforts such as Minitron muralidharan2024compact and Sheared Llama xia2024sheared have explored the use of existing LLMs yang2024qwen2_5 ; abdin2024phi4 ; grattafiori2024llama3 ; adler2024nemotron to accelerate SLM pre-training. These methods typically combine structured pruning and distillation, first removing “unimportant” neurons and then recovering performance via distillation or continued training. Despite these advances, current distillation paradigms still fall short in fully utilizing the rich knowledge embedded in teacher models, resulting in limited efficiency and suboptimal student performance. We identify three core challenges in existing approaches: • Information Loss from Hard Pruning: Most existing methods use hard pruning , permanently removing selected neurons, channels, attention heads, or entire layers yang2024laco ; muralidharan2024compact ; xia2024sheared ; men2024shortgpt . While reducing model size, it discards valuable information from the teacher’s weights, e.g., pruning 50% of Llama-7B in LLM-Pruner ma2023llm caused a sharp performance drop from 63.25 to 48.98. • Inefficient Alignment of Representations: Feature-based distillation methods jiao2020tinybert ; wang2020minilm ; muralidharan2024compact often use additional projection matrices to align intermediate activations between teacher and student. However, as the student’s internal states evolve during training, learning effective alignment mappings becomes challenging, reducing distillation efficiency. • Underutilization of Informative Activations: Prior work has primarily focused on aligning attention scores jiao2020tinybert ; wang2020minilm , while largely overlooking the high-dimensional, information-rich activations from Feed-Forward Networks (FFNs). These FFN signals are crucial to the expressiveness of modern LLMs, as confirmed by our ablation study in Section 4.3 . To overcome these challenges, we propose Low-Rank Clone (LRC) , a highly efficient pre-training method that constructs SLMs aspiring to behavioral equivalence with strong teacher models. LRC eliminates the need to train the student’s weights, except for the RMSNorm parameters zhang2019root , which constitute less than 1% of the total, drastically reducing training overhead. Compared to prior multi-stage approaches muralidharan2024compact ; xia2024sheared , LRC introduces a unified framework that performs soft pruning and knowledge distillation simultaneously via trainable low-rank projection matrices. Each forward pass of LRC consists of two key steps: (1) Low-Rank Projection , which projects the teacher’s weights into smaller weights that directly serve as the student’s parameters. (2) Activation Clone , which aligns the student’s intermediate activations with those of the teacher to preserve behavioral fidelity. This design directly addresses the core limitations of existing methods, offering three key advantages: • Minimal Information Loss: By directly generating the student’s weights from the teacher model, LRC preserves substantially more of the teacher’s knowledge, even at aggressive compression levels, than hard pruning strategies. • Alignment-Free Distillation: The projection matrices naturally handle representational mismatches between teacher and student layers, removing the need for additional alignment modules and improving both efficiency and performance. • Full Utilization of Activations: LRC captures a broad spectrum of intermediate signals, including underutilized FFN activations, which we show to encode rich and valuable information often ignored by prior methods. We comprehensively evaluate LRC using strong, open-source teacher models such as Llama-3.2-3B-Instruct and Qwen2.5-3B/7B-Instruct, showing competitive or superior performance compared to leading SLMs trained on trillions of tokens. As depicted in Figure 1 , LRC-1.7B outperforms Qwen3-1.7B (64.98 vs. 63.17) on standard benchmarks, while requiring up to 1,000 \times fewer training tokens. These results highlight LRC’s potential to drastically improve the cost-efficiency of high-performance SLM development and advance the state of practical, high-performance language models. 2 Related Work To reduce the computational and memory overhead of LLMs, researchers have explored a range of techniques, including knowledge distillation sanh2019distilbert ; sun2019patient ; hinton2015distilling ; jiao2020tinybert ; gu2024miniplm ; wang2020minilm ; gu2024minillm , structured pruning dery2024everybody_prune ; yang2024laco ; sun2024a ; ma2023llm ; men2024shortgpt , quantization dettmers2022llm ; xiao2023smoothquant ; liu2024spinquant , KV cache compression xiao2024efficient ; hao2025omnikv ; jiang2023llmlingua ; zhang2023h2o ; liu2024kivi , and efficient training frameworks aminabadi2022deepspeed ; shoeybi2019megatron ; zhao2024galore ; hao2024meft ; jiang2019accelerating . Among these, knowledge distillation and structured pruning are most relevant to our work. Knowledge Distillation and Structured Pruning Knowledge distillation hinton2015distilling ; kim2016sequencekd aims to transfer knowledge from a large, pre-trained teacher model to a smaller student model. Early approaches either use synthetic data generated by the teacher to train the student chiang2023vicuna ; peng2023instruction ; hsieh2023distilling , or minimize the divergence between their output distributions hinton2015distilling ; gu2024miniplm ; shing2025taid . While effective, these techniques often suffer from limited transfer efficiency and scalability gu2024minillm ; sanh2019distilbert . To improve transfer quality, feature-based distillation methods like TinyBert jiao2020tinybert , MiniLM wang2020minilm and TED liang2023less_ted utilize intermediate activations from transformer layers to guide student learning. However, these methods ignore the rich information encoded in the model weights geva2021transformer , and require additional alignment matrices to bridge discrepancies in hidden states, increasing training overhead. In contrast, LRC circumvents these limitations by using trainable low-rank projection matrices that simultaneously extract weight-level knowledge and serve as implicit alignment layers, eliminating the need for student weight initialization or separate alignment training. Recent approaches such as Minitron muralidharan2024compact and Sheared Llama xia2024sheared integrate hard pruning with distillation to compress LLMs. Yet, they rely on a multi-stage pipeline–pruning followed by distillation or continued training–which increases training cost and sensitivity to pruning ratios. Moreover, hard pruning can cause substantial

Read the complete text