Replay to Remember: Retaining Domain Knowledge in Streaming Language Models
Sneh Pillai · 2025
Read Replay to Remember: Retaining Domain Knowledge in Streaming Language Models by Sneh Pillai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-17780.txt.
Description
Sneh Pillai studies how language models can keep old skills while learning from a live data stream in a 2025 paper. Models tend to forget earlier facts when new data arrives, a flaw called catastrophic forgetting. Real uses, from clinics to law, need steady learning on tight budgets.
Pillai pairs a light tuning method with a tiny replay store of past items. Tests ran across three fields: medical questions, genetics, and law. Scores tracked how well the model learned, forgot, and recovered over time. Forgetting did occur on its own. Even a small replay set steadied the model and restored much lost ground. The work offers a cheap recipe for models that must adapt on the fly.
Catalog record
| CREATOR | Sneh Pillai |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 17 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-17780