Flipping Knowledge Distillation: Leveraging Small Models’ Expertise to Enhance LLMs in Text Matching

Mingzhe Li, Jing Xiang, Qishen Zhang, Kaiyang Wan, Xiuying Chen · 2025

Read Flipping Knowledge Distillation: Leveraging Small Models’ Expertise to Enhance LLMs in Text Matching by Mingzhe Li, Jing Xiang, Qishen Zhang, Kaiyang Wan, Xiuying Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-05617.txt.

Introduction

Mingzhe Li and co-authors flip the teach flow: a big chat tool learns text-match craft from a small keen tool. Teach flow stock runs big to small. In match work, small tuned tools oft beat big ones, since they hone pair like marks. Their 2025 flip dresses the big tool in a squeeze-and-read split by lean add-ons. The squeeze half builds pair marks; the read half maps them out.

Train aims those marks at the small coach's like scores by a gap-aware pair law. The law holds true scores for like and unlike pairs and bends for in-pair spread. A fair small coach is all the flip asks.

Catalog record

CREATORMingzhe Li, Jing Xiang, Qishen Zhang, Kaiyang Wan, Xiuying Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-05617

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Flipping Knowledge Distillation: Leveraging Small Models’ Expertise to Enhance LLMs in Text Matching Mingzhe Li, Jing Xiang, Qishen Zhang, Kaiyang Wan, Xiuying Chen Abstract Knowledge distillation typically involves transferring knowledge from a Large Language Model (LLM) to a Smaller Language Model (SLM). However, in tasks such as text matching, fine-tuned smaller models often yield more effective domain-specific representations, as they focus on optimizing the similarity of input pairs. To leverage both the specialized strengths of small models and the rich semantic understanding of LLMs, we introduce a flipped knowledge distillation paradigm, where LLM learns from SLM. Specifically, we address the architectural gap between decoder-only LLMs and smaller encoder-based models by reinterpreting LLMs in an encoder-decoder manner using LoRA. The encoder generates compressed representations, while the decoder maps them to the output space. During training, the encoder produces representations and their similarities, which are then aligned with the similarity scores produced by the teacher, using our proposed Margin-aware Contrastive Learning (MCL) approach. The MCL ensures accurate similarity for both positive and negative pairs, and adaptively handles the internal differences within positive and negative samples. Our paradigm requires only a reasonably good-performing SLM, allowing the LLM to achieve improved performance. Experiments on financial and healthcare benchmarks, as well as real-world applications, confirm its effectiveness, and the model has been fully deployed in an online environment. 1 Introduction Large Language Model (LLMs) have demonstrated remarkable capabilities in acquiring diverse knowledge, making them highly effective across a wide range of tasks Zhao et al. 2023 ; Xi et al. 2023 ; Song et al. 2025 . Consequently, Smaller Language Models (SLM) often learn from LLMs via knowledge distillation and imitation learning Gu et al. 2023 ; Li et al. 2023a ; Gu et al. 2024 ; Xu et al. 2024b . However, despite their extensive knowledge, LLMs often underperform on domain-specific tasks compared to smaller models fine-tuned on specialized data Ma et al. 2023 ; Chen et al. 2023 ; Stewart et al. 2023 ; Hu et al. 2024 . For example, in text matching, SLMs are trained to make paired inputs more similar in representation Devlin 2018 , unlike LLMs that directly predict match or non-match Touvron et al. 2023 . This allows SLMs to better distinguish between synonyms and enhances their representation learning Hillebrand et al. 2023 , especially in specialized tasks, because they can learn in-domain terminology more effectively. Figure 1: (a) Comparison of traditional knowledge distillation from LLM to SLM, and our flipping knowledge distillation of domain-specific expertise from SLM to LLM. (b) In the representation space, when texts are highly similar but semantically unrelated, the LLM learns the correct semantic distance between them through distillation from the SLM, which provides more accurate knowledge. Therefore, in this work, we propose a novel flipping knowledge distillation approach for LLMs to learn from the expertise of SLMs in representation learning for text matching, as illustrated in Figure 1 . Previous efforts to improve the performance of large models in specific domains predominantly focused on directly utilizing domain-specific data by supervised fine-tuning Ding et al. 2023 , retrieval-augmented generation Zakka et al. 2024 , or modifying model architectures to inject domain-specific knowledge Diao et al. 2023 ; Zhang et al. 2023 . In contrast, our method takes advantage of LLM’s rich semantic understanding ability, while also enabling LLM to learn directly from the specialized knowledge of SLM. This enables LLM to acquire not only simple matching labels, but also more nuanced representation information from SLM, thereby capturing more detailed knowledge. Flipped knowledge learning presents two primary obstacles. First, state-of-the-art LLMs are predominantly decoder-only architectures, lacking dedicated modules for learning effective text representations. Second, effectively learning relationships and relative distances between and within positive and negative pairs from small model is difficult. To address these challenges, we reinterpret the decoder-only LLM structure as an encoder-decoder mechanism by leveraging Low-Rank Adaptation (LoRA) Hu et al. . In our approach, the encoder, represented by the compressed matrices in LoRA, learns compact input representations, while the decoder, defined by the expansion matrices, decodes and recombines these representations. This design allows the encoder to generate text representations and compute their similarities, which are then aligned with the similarity scores produced by the teacher model, providing guidance for capturing semantic similarity more effectively. To further enhance learning from the teacher model, we propose a Margin-aware Contrastive Loss (MCL). Unlike conventional methods that only ensure positive samples have higher similarity scores than negative ones, MCL introduces two margin zones. These zones encourage the LLM to learn greater differentiation not only between positive and negative samples but also within each category. Additionally, to address potential inaccuracies in the teacher model, we introduce a dual threshold strategy to filter noises, ensuring more reliable learning. For experiments, we evaluate our paradigm on various LLMs with different scale architectures, including Qwen-0.5b and GLM-10b, learning from three different SLMs across three benchmarks. The results demonstrate that our model not only outperforms the original LLMs but also surpasses LLMs trained using other strategies, such as supervised finetuning, parameter-efficient fine-tuning, and distillation from larger or smaller teacher models. Our main contributions are as follows: First, we introduce a paradigm that combines the strengths of LLMs and SLMs, incorporating domain-specific expertise from SLMs into LLMs, thus enhancing overall performance. Second, we develop a dual margin learning function for knowledge distillation that effectively identifies and differentiates between positive and negative samples, improving distillation precision. Finally, our approach yields a fine-tuned text matching model that outperforms existing methods, providing a robust solution for knowledge distillation. Figure 2: Comparison of SLM, traditional LLM, and our reinterpreted LLM for text matching. We reinterpret the decoder-only structure of the LLM into an encoder-decoder architecture to facilitate calculating the matching score between text representations instead of directly generating output. 2 Related Work Knowledge Distillation. Knowledge distillation is a task that involves transferring knowledge from an advanced model to a less performant model. Distillation algorithms can achieve this in various ways. For example, a teacher model can generate data for the smaller model to learn from Chiang et al. 2023 ; Xu et al. 2023 . One instance is Alpaca-7b Taori et al. 2023 , a model fine-tuned from the Llama-7b model on 52,000 instruction-following demonstration. Alternatively, the teacher model can produce features for the smaller model to learn Timiryasov and Tastet 2023 . For instance, Liang et al. 2023 propose task-aware layer-wise distillation to selectively capture relevant information from the teacher model using task-aware filters. Other approaches include reinforcement learning Chen et al. 2024b and ranking learning Yuan et al. 2023 . These works typically use a large teacher model (e.g., GPT-4, Llama-70b) and a small student model (e.g., GPT-2, Llama-7b). Knowledge Injection. Before the era of LLMs, many works focused on manipulating the transformer structure to better inject knowledge. For example, Diao et al. 2023 propose domain-specific adapters to inject domain-specific knowledge into the feedforward layer. Zhang et al. 2023 introduce a knowledge plugin that injects knowledge into frozen downstream models. However, modifying the LLM architecture is challenging, and many researchers have instead turned to parameter-efficient fine-tuning (PEFT) Hu et al. ; Yang et al. 2023 ; Chen et al. 2024a , injecting

Read the complete text