Control LLM: Controlled Evolution for Intelligence Retention in LLM
Haichao Wei, Yunxiang Ren, Zhoutong Fu, Aman Lunia, Yi-Lin Chen, Alice Leung, Ya Xu · 2025
Read Control LLM: Controlled Evolution for Intelligence Retention in LLM by Haichao Wei, Yunxiang Ren, Zhoutong Fu, Aman Lunia, Yi-Lin Chen, Alice Leung, Ya Xu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-10979.txt.
Description
Seven researchers present a way to keep teaching language models without erasing old skills in a paper from 2025. Continued training often causes catastrophic forgetting, where new lessons wipe out earlier ability. Their method, Control LLM, adds fresh transformer blocks beside the original ones. Interpolation aligns the hidden states of old and new blocks during training. The design lets new knowledge enter while old performance stays guarded.
On a popular open model, math and coding scores rose sharply after continued training. Multilingual scores improved as well, using far less data than rivals. Losses on original skills stayed under five percent, versus over thirty-five percent for some tuned models. The approach already runs in LinkedIn products.
Catalog record
| CREATOR | Haichao Wei, Yunxiang Ren, Zhoutong Fu, Aman Lunia, Yi-Lin Chen, Alice Leung, Ya Xu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-10979