Replay to Remember: Retaining Domain Knowledge in Streaming Language Models

Sneh Pillai · 2025

Read Replay to Remember: Retaining Domain Knowledge in Streaming Language Models by Sneh Pillai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-17780.txt.

Description

Sneh Pillai studies how language models can keep old skills while learning from a live data stream in a 2025 paper. Models tend to forget earlier facts when new data arrives, a flaw called catastrophic forgetting. Real uses, from clinics to law, need steady learning on tight budgets.

Pillai pairs a light tuning method with a tiny replay store of past items. Tests ran across three fields: medical questions, genetics, and law. Scores tracked how well the model learned, forgot, and recovered over time. Forgetting did occur on its own. Even a small replay set steadied the model and restored much lost ground. The work offers a cheap recipe for models that must adapt on the fly.

Catalog record

CREATORSneh Pillai
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-17780

Browse the full library · About the Tower of Babel