TuneShift-KD: Knowledge Distillation and Transfer for Fine-tuned Models
Yushi Guan, Jeanine Ohene-Agyei, Daniel Kwan, Jean Sebastien Dandurand, Yifei Zhang, Nandita Vijaykumar · 2026
Read TuneShift-KD: Knowledge Distillation and Transfer for Fine-tuned Models by Yushi Guan, Jeanine Ohene-Agyei, Daniel Kwan, Jean Sebastien Dandurand, Yifei Zhang, Nandita Vijaykumar (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-24518.txt.
Introduction
Guan and co-authors present TuneShift-KD, a method for moving specialised knowledge from a fine-tuned model into a newer base model. The original training data is often unavailable, for privacy or commercial reasons. Their key signal is perplexity: prompts where the fine-tuned model is confident but the base model struggles mark the specialised knowledge. TuneShift-KD uses a few such examples to build a synthetic training set. That set then distils the knowledge into the target model automatically, without access to the source data.
Catalog record
| CREATOR | Yushi Guan, Jeanine Ohene-Agyei, Daniel Kwan, Jean Sebastien Dandurand, Yifei Zhang, Nandita Vijaykumar |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62577 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.24518
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TuneShift-KD: Knowledge Distillation and Transfer for Fine-tuned Models Yushi Guan, Jeanine Ohene-Agyei, Daniel Kwan, Jean Sebastien Dandurand, Yifei Zhang, Nandita Vijaykumar Abstract To embed domain-specific or specialized knowledge into pre-trained foundation models, fine-tuning using techniques such as parameter efficient fine-tuning (e.g. LoRA) is a common practice. However, as new LLM architectures and pre-trained models emerge, transferring this specialized knowledge to newer models becomes an important task. In many scenarios, the original specialized data may be unavailable due to privacy or commercial restrictions, necessitating distillation and transfer of this specialized knowledge from the fine-tuned base model to a different pre-trained model. We present TuneShift-KD, a novel approach that automatically distills specialized knowledge from a fine-tuned model to a target model using only a few examples representative of the specialized information. Our key insight is that specialized knowledge can be identified through perplexity differences between base and fine-tuned models: prompts where the fine-tuned model responds confidently (low perplexity), but the base model struggles (high perplexity), indicate queries corresponding to the specialized knowledge learned by the fine-tuned model. TuneShift-KD leverages this insight to create a synthetic training dataset to transfer the specialized knowledge. Using an iterative process, TuneShift-KD generates more prompts similar to those that generated responses with specialized knowledge. TuneShift-KD does not require training discriminators or access to training datasets. It is an automated approach that only requires the initial fine-tuned and base models and a few representative prompts. Our experiments demonstrate that models fine-tuned using TuneShift-KD achieve higher accuracy than prior approaches, enabling ease of deployment and more effective transfer of the specialized knowledge. 1 Introduction Fine-tuning large language models (LLMs) has become the widely adopted approach to introducing specialized knowledge or capabilities into pre-trained foundation models. This technique has proven effective in diverse domains, from the integration of specialized legal and healthcare knowledge ( 30 ; 7 ) to the enhancement of coding, logical reasoning, and mathematical abilities ( 14 ; 63 ; 31 ) . Parameter Efficient Fine-Tuning (PEFT) has emerged as the dominant methodology, keeping base model weights fixed while training only a small set of additional parameters. Low-Rank Adaptation (LoRA) ( 23 ) and its variants ( 10 ; 70 ; 35 ) are widely used implementations of this concept, where trainable low-rank decomposition matrices are added to the weight matrices of the frozen base model. This approach significantly reduces memory requirements compared to full-weight fine-tuning, making it economically and computationally efficient for adapting foundation models to specialized knowledge. As LLMs continuously evolve to new architectures, there is a need to transfer the specialized knowledge from a fine-tuned source model to newer pre-trained (target) models to ensure accuracy for domain-specific usage. Conversely, this transfer may also target older model architectures for compatibility with older compute hardware. However, in many scenarios, the training data used for the initial fine-tuning process may no longer be available to directly fine-tune the target model. For example, when an LLM is hosted by a third-party cloud service provider, fine-tuning data may remain undisclosed due to privacy or commercial restrictions ( 40 ; 67 ) . Similarly, hardware vendors optimizing models for deployment typically lack access to proprietary fine-tuning datasets. In these scenarios, the ability to transfer specialized knowledge without full access to the domain-specific training dataset is a useful but challenging endeavor. Knowledge distillation (KD) ( 22 ) is an established technique for transferring knowledge between models. In the context of LLMs, KD has primarily been used to improve inference efficiency by training smaller models with larger model outputs, but it has also been used to transfer knowledge from one model to another ( 66 ) . However, there is little research on how to effectively transfer specialized knowledge (the knowledge learned by and embedded in fine-tuned models using domain-specific datasets) from a source to a target model. Such domain-specific fine-tuning can equip models with both factual knowledge and enhanced capabilities (e.g., reasoning, alignment), which we collectively refer to as “specialized knowledge” hereafter. Knowledge distillation typically requires access to the training data, and identifying fine-tuning knowledge without such access has proved difficult. Trans-LoRA ( 60 ) is the closest work that has targeted this problem by training a dedicated discriminator to distinguish fine-tuning data from other data. However, this approach requires modifications to the standard fine-tuning process (including training discriminators on the original fine-tuning data) and incurs additional storage costs since a separate discriminator must be stored for each fine-tuning dataset. LoRA-X ( 15 ) takes a different approach by directly copying LoRA weights from the source model to a target base model, but this requires that model weights are highly similar and can be aligned. Empirically, this constraint limits LoRA-X to source and target models from the same model family that are highly similar. These limitations motivate two desirable properties for specialized knowledge transfer methods: i) ability to transfer knowledge across different LLM architecture families and ii) compatibility with standard fine-tuning pipelines . Achieving both properties simultaneously remains an open challenge. During fine-tuning, a model shifts its output probability distribution to match the fine-tuning domain’s distribution, improving its probability in generating domain-relevant responses, and also assigning higher probability to in-domain examples than the original base model. Given the same in-domain prompt, the fine-tuned model is more likely to produce a relevant response than the base model’s response. When the fine-tuned model assigns high probability (low perplexity) to its own response while assigning low probability (high perplexity) to the base model’s response for the same prompt, this creates a perplexity gap. This gap reveals a response difference that stems from the fine-tuning process, which is the only systematic source of divergence between the models. We define this as the perplexity difference filtering criterion : \text{PPL}(y^{(f)};\theta_{F})<\tau\leq\text{PPL}(y^{(b)};\theta_{F}) , where \theta_{F} denotes the fine-tuned model parameters, and y^{(f)} and y^{(b)} are the fine-tuned and base model responses respectively. This criterion identifies examples that capture specialized knowledge while discarding both general knowledge (low perplexity in both responses) and overly difficult examples (high perplexity in both responses). Crucially, perplexity is readily available during standard inference and reflects model output distribution changes during fine-tuning regardless of the specific fine-tuning task. This property applies to virtually all decoder-based LLM architectures, making our approach broadly compatible with existing fine-tuning pipelines. We provide a more formal discussion in Section 3.3 . Leveraging the perplexity difference filtering criterion, we propose TuneShift-KD, an automated mechanism that distills specialized knowledge from a source fine-tuned model to a target pre-trained model without full access to the specialized training data set. TuneShift-KD implements the idea by using a few fine-tuning prompts that are representative of the specialized knowledge base to generate similar prompts through an instruction-tuned LLM. We use the phrase ”Generate 20 more samples like these 5 [ the 5 sample data ]” to create more training samples in bulk. We discard the bulk-generated responses and keep only the prompts. These prompts are then fed to both the fine-tuned and base models to collect their respective responses. We select prompt-response pairs that demonstrate perplexity differences between the fine-tuned and base model outputs. These selected examples are iteratively added to our training pool and used to generate additional prompts, creating a self-expanding collection of relevant training