MoSE: Hierarchical Self-Distillation Enhances Early Layer Embeddings
Andrea Gurioli, Federico Pennino, Joao Monteiro, Maurizio Gabbrielli · 2025
Read MoSE: Hierarchical Self-Distillation Enhances Early Layer Embeddings by Andrea Gurioli, Federico Pennino, Joao Monteiro, Maurizio Gabbrielli (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-03008.txt.
Introduction
Andrea Gurioli and co-authors build a code search encoder that can stop thinking early without going dumb. Their 2025 paper presents MoSE, a one billion parameter multi exit encoder for finding and classifying code. Big models are accurate but slow; classic distillation shrinks them only by training whole separate students, at full price.
MoSE distills inside one network: during training, deep layers tutor the shallow ones, a hierarchical self distillation that fattens early embeddings. At run time the user picks an exit layer to fit the latency budget. A repository level loss stretches the training context, and a new translated dataset adds cross language search pairs. Early exits, the team reports, hold accuracy far better than usual.
Catalog record
| CREATOR | Andrea Gurioli, Federico Pennino, Joao Monteiro, Maurizio Gabbrielli |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-03008
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MoSE: Hierarchical Self-Distillation Enhances Early Layer Embeddings Andrea Gurioli, Federico Pennino, Joao Monteiro, Maurizio Gabbrielli Abstract Deploying language models often requires navigating accuracy vs. performance trade-offs to meet latency constraints while preserving utility. Traditional model distillation reduces size but incurs substantial costs through training separate models. We introduce ModularStarEncoder ( MoSE ), a 1-billion-parameter multi-exit encoder for code retrieval and classification that employs a novel Self-Distillation mechanism. This approach significantly enhances lower-layer representations, enabling flexible deployment of different model portions with favorable performance trade-offs. Our architecture improves text-to-code and code-to-code search by targeting specific encoder layers as exit heads, where higher layers guide earlier ones during training—improving intermediate representations at minimal additional cost. We further enhance MoSE with a repository-level contextual loss that maximizes training context window utilization. Additionally, we release a new dataset created through code translation that extends text-to-code benchmarks with cross-language code-to-code pairs. Evaluations demonstrate the effectiveness of Self-Distillation as a principled approach to trading inference cost for accuracy across various code understanding tasks. 1 Introduction Large language models (LLMs) have significantly impacted natural language processing ( Niu et al. 2023 ) , but their computational demands pose substantial deployment challenges. The field has responded with various efficiency strategies: quantization techniques reduce numerical precision ( Jacob et al. 2017 ; Lin et al. 2023 ; Egiazarian et al. 2024 ) ; knowledge distillation trains smaller “student” models to emulate larger “teacher” models ( Sanh et al. 2019 ; Jiao et al. 2019 ) ; and pruning methods eliminate less influential weights ( Han et al. 2015 ) . Concurrently, model families like LLaMA ( Dubey et al. 2024 ) , Qwen ( Hui et al. 2024 ) , Mistral ( Jiang et al. 2023 ) , and SmolLM ( Allal et al. 2025 ) exemplify a shift toward more efficient architectures at varying parameter scales. Dynamic inference approaches further optimize efficiency through mechanisms like multi-exit networks. Early-exit architectures such as BranchyNet ( Teerapittayanon et al. 2017 ) balance computation and accuracy by allowing predictions at intermediate layers, while Matryoshka representation learning ( Kusupati et al. 2022 ) enables adjustment of computational complexity through embedding dimensionality pruning. These approaches maintain performance while reducing computational demands in resource-constrained environments. Furthermore, insights from the SimCLR framework ( Chen et al. 2020 ) revealed that the final layer of a model does not necessarily produce the most semantically meaningful representations. Valeriani et al. 2023 analyzed the geometric evolution of hidden representations within large transformer models across layers, outlining that intermediate layers are identified as holding the most semantically rich representations for downstream tasks. Input \vdots Layer 4 \vdots Layer 9 \vdots Layer 18 \vdots Layer 27 \vdots Layer 36 \mathcal{L}_{4} \alpha=0.11 \mathcal{L}_{9} \alpha=0.25 \mathcal{L}_{18} \alpha=0.50 \mathcal{L}_{27} \alpha=0.75 \mathcal{L}_{36} \alpha=1.0 \sum_{i\in\iota}\alpha_{i}\mathcal{L}_{i} (a) (b) Figure 1 : (a) Overview of our multi-exit self-distillation encoder, shown here with exit heads at selected layers (e.g., Layers 4, 9, 18, 27, and 36). Each exit head predicts an output embedding and adds a layer loss, contribution weighted by a coefficient \alpha_{i} , summed into the overall objective \mathcal{L} . (b) Computational cost (GFLOPs) vs. performances trade-off at different exit layers over CodeSearchNet dataset (text-to-code in avg. MRR) and POJ104 (code-to-code in mAP). Despite a reduction of approximately 90% floating point operations from layer 36 to layer 4, MRR performance only drops by 6.4% in absolute terms. For POJ104, our best results are observed in the initial layers. Building on these principles, we introduce ModularStarEncoder ( MoSE ), a modular multi-exit encoder built on top of the StarCoder-2 ( Lozhkov et al. 2024 ) architecture, with 1 billion parameters that incorporate intra-model self-distillation. As illustrated in Figure 1(a) , our architecture applies training objectives at multiple exit points throughout the network (layers 4, 9, 18, 27, and 36), with each exit contributing a weighted loss component to the overall objective. This approach encourages lower layers to learn better representations by mimicking higher-layer outputs. Figure 1(b) demonstrates the resulting accuracy-computation trade-off of encoders trained under this framework: despite a 90% reduction in floating point operations from layer 36 to layer 4, performance over text-to-code retrieval drops by only 6.4% in absolute terms, while the model maintains the best performances for code-to-code retrieval in very early layers. Notably, the exit point with the best performance isn’t necessarily the final layer, as noted in previous findings due to Chen et al. 2020 ; Valeriani et al. 2023 . This offers both deployment flexibility and a natural model selection mechanism for specific tasks. Users can select which portion of the model to use during inference, starting from the first exit point at 160M parameters. By allowing users to select an earlier exit point, MoSE provides a direct mechanism to decrease computational demands. This flexibility is crucial as smaller models generally require less energy and memory for inference, leading to more sustainable deployment options ( Samsi et al. 2023 ) . We fine-tuned MoSE on SynthCoNL (cf. Sec. 5 for details) for text-to-code and code-to-code retrieval, and on BigCloneBench ( Svajlenko & Roy 2015 ) for code clone detection, achieving state-of-the-art results among open models while maintaining deployment modularity. Our contributions are as follows: 1. A self-distillation methodology that trains multiple model resolutions within a unified layer stack, reducing redundancy and improving scalability—an approach that could significantly impact LLM training pipelines dependent on multiple model distillations. 2. MoSE , pre-trained 1 1 1 https://huggingface.co/modularStarEncoder/ModularStarEncoder and fine-tuned 2 2 2 https://huggingface.co/modularStarEncoder/ModularStarEncoder-finetuned , with up to 1 billion parameters and five exit points, allowing users to select model size based on memory and computational constraints. 3. SynthCoNL 3 3 3 https://huggingface.co/datasets/modularStarEncoder/SynthCoNL , a new dataset of 1,071,367 natural language-code-code triplets constructed via code translation, expanding text-to-code benchmarks with cross-language code-to-code pairs. Figure 2 : The illustration on the left depicts the in-context loss framework, where samples from various repositories are concatenated. Positive examples originate from the same repository context, whereas negative examples are sourced from different repositories. To enable the model’s use of FlashAttention V2 , we applied left padding and positioned the CLS token at the end of the sentence. On the right side, you’ll find the pseudocode for the in-context loss framework. 2 Self-Distillation and In-Context Classification As different layers of transformer models capture varying degrees of semantic richness, often with intermediate layers proving most effective for specific tasks ( Bolya et al. 2025 ) , we design our pre-training strategy to support modularity and ease downstream task adaptability. Rather than treating the deepest layer as the sole source of meaningful representations, we aim to enhance representations throughout the network. We hypothesize that applying supervision at multiple depths , not just at the output—encourages more robust and semantically rich intermediate representations. To this end, we combine multiple training objectives, each contributing to more versatile and generalizable embeddings ( Wang et al. 2023 ) . While we retain the widely adopted Masked Language Modeling (MLM) loss Feng et al. 2020 , we discard the traditional Next Sentence Prediction (NSP) objective. NSP has been shown to offer minimal benefit after fine-tuning ( Liu et al. 2019 ) , and it imposes a rigid sentence-level structure that leads to inefficient use of context windows, especially