Spurious Forgetting in Continual Learning of Language Models

Junhao Zheng, Xidi Cai, Shengjie Qiu, Qianli Ma · 2025

Read Spurious Forgetting in Continual Learning of Language Models by Junhao Zheng, Xidi Cai, Shengjie Qiu, Qianli Ma (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-13453.txt.

Introduction

Junhao Zheng and his three co-authors, publishing in 2025, argue that language models often seem to forget old skills for the wrong reason. When a model learns new tasks one after another, scores on old tasks plunge. The standard reading is catastrophic forgetting of knowledge. The team proposes spurious forgetting instead: the knowledge survives, but the model's alignment to the old task's format is knocked sideways.

Controlled experiments with synthetic data show the damage happens early. The first update steps on a new task shove the weights in directions that break earlier alignments, while deeper knowledge stays readable. Guided by that theory, the authors freeze the model's bottom layers during later training. The simple fix improved results across four continual learning scenarios. They urge the field to measure alignment loss and knowledge loss separately.

Catalog record

CREATORJunhao Zheng, Xidi Cai, Shengjie Qiu, Qianli Ma
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE166 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-13453

Browse the full library · About the Tower of Babel