Spurious Forgetting in Continual Learning of Language Models
Junhao Zheng, Xidi Cai, Shengjie Qiu, Qianli Ma · 2025
Read Spurious Forgetting in Continual Learning of Language Models by Junhao Zheng, Xidi Cai, Shengjie Qiu, Qianli Ma (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-13453.txt.
Introduction
Junhao Zheng and his three co-authors, publishing in 2025, argue that language models often seem to forget old skills for the wrong reason. When a model learns new tasks one after another, scores on old tasks plunge. The standard reading is catastrophic forgetting of knowledge. The team proposes spurious forgetting instead: the knowledge survives, but the model's alignment to the old task's format is knocked sideways.
Controlled experiments with synthetic data show the damage happens early. The first update steps on a new task shove the weights in directions that break earlier alignments, while deeper knowledge stays readable. Guided by that theory, the authors freeze the model's bottom layers during later training. The simple fix improved results across four continual learning scenarios. They urge the field to measure alignment loss and knowledge loss separately.
Catalog record
| CREATOR | Junhao Zheng, Xidi Cai, Shengjie Qiu, Qianli Ma |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 166 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-13453