MixSD : Mixed Contextual Self-Distillation for Knowledge Injection

Jiarui Liu, Lechen Zhang, Yongjin Yang, Yinghui He, Yingheng Wang, Weihao Xuan Zhijing Jin Mona T. Diab, [0.6em] Carnegie Mellon University · 2026

Read MixSD : Mixed Contextual Self-Distillation for Knowledge Injection by Jiarui Liu, Lechen Zhang, Yongjin Yang, Yinghui He, Yingheng Wang, Weihao Xuan Zhijing Jin Mona T. Diab, [0.6em] Carnegie Mellon University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-16865.txt.

Description

Liu and colleagues propose MixSD, a teacher-free approach for knowledge injection into language models without fine-tuning forgetting. Fine-tuning targets from humans or outside systems diverge from the model's distribution, forcing imitation of unlikely sequences. MixSD builds supervision from mixed tokens from two base-model conditionals: one conditional seeing the injected fact in context, and a naive conditional reflecting the prior. The mixed sequences preserve signal while staying close to the distribution. Across settings, MixSD keeps 100% of held-out capability with high accuracy, while standard fine-tuning keeps 1%.

Catalog record

CREATORJiarui Liu, Lechen Zhang, Yongjin Yang, Yinghui He, Yingheng Wang, Weihao Xuan Zhijing Jin Mona T. Diab, [0.6em] Carnegie Mellon University
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67457
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.16865

Browse the full library · About the Tower of Babel