MixSD : Mixed Contextual Self-Distillation for Knowledge Injection
Jiarui Liu, Lechen Zhang, Yongjin Yang, Yinghui He, Yingheng Wang, Weihao Xuan Zhijing Jin Mona T. Diab, [0.6em] Carnegie Mellon University · 2026
Read MixSD : Mixed Contextual Self-Distillation for Knowledge Injection by Jiarui Liu, Lechen Zhang, Yongjin Yang, Yinghui He, Yingheng Wang, Weihao Xuan Zhijing Jin Mona T. Diab, [0.6em] Carnegie Mellon University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-16865.txt.
Description
Liu and colleagues propose MixSD, a teacher-free approach for knowledge injection into language models without fine-tuning forgetting. Fine-tuning targets from humans or outside systems diverge from the model's distribution, forcing imitation of unlikely sequences. MixSD builds supervision from mixed tokens from two base-model conditionals: one conditional seeing the injected fact in context, and a naive conditional reflecting the prior. The mixed sequences preserve signal while staying close to the distribution. Across settings, MixSD keeps 100% of held-out capability with high accuracy, while standard fine-tuning keeps 1%.
Catalog record
| CREATOR | Jiarui Liu, Lechen Zhang, Yongjin Yang, Yinghui He, Yingheng Wang, Weihao Xuan Zhijing Jin Mona T. Diab, [0.6em] Carnegie Mellon University |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67457 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.16865