Stabilizing LLM Supervised Fine-Tuning via Explicit Distributional Control
Xinyu Wang, Changzhi Sun, Yuanbin Wu, Xiaoling Wang · 2026
Read Stabilizing LLM Supervised Fine-Tuning via Explicit Distributional Control by Xinyu Wang, Changzhi Sun, Yuanbin Wu, Xiaoling Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-04468.txt.
Description
Xinyu Wang and three co-authors tackle catastrophic forgetting in fine-tuning large language models, where learning a new task erodes skills the model already had. Recent evidence blames excessive drift in the model's output distribution during training.
Their anchored method controls that drift directly. Instead of pulling the model toward one fixed reference, a moving anchor blends the current model with a frozen reference. That blend forms an intermediate target. Training thus becomes a chain of small local updates. They prove a stable per-step bound on distribution change, and experiments on maths, medical calculation, and instruction tasks place the method on the best trade-off frontier.
Catalog record
| CREATOR | Xinyu Wang, Changzhi Sun, Yuanbin Wu, Xiaoling Wang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.04468