What Does Loss Optimization Actually Teach, If Anything? Knowledge Dynamics in Continual Pre-training of LLMs
Seyed Mahed Mousavi *, Simone Alghisi, Giuseppe Riccardi · 2026
Read What Does Loss Optimization Actually Teach, If Anything? Knowledge Dynamics in Continual Pre-training of LLMs by Seyed Mahed Mousavi *, Simone Alghisi, Giuseppe Riccardi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-03858.txt.
Description
Seyed Mahed Mousavi, Simone Alghisi, and Giuseppe Riccardi ask what continual pretraining actually teaches a language model, beyond lowering its loss score. The practice is widely used to add and update factual knowledge, yet teams usually assume that falling loss means rising knowledge. The authors built a controlled benchmark of factual documents matched in distribution and slipped diagnostic probes directly into the training loop.
Across three instruction-tuned models and several training strategies, optimization and learning systematically diverged. Loss fell steadily while factual learning wobbled and often failed to consolidate, depending heavily on prior exposure. Out-of-domain skills like math degraded from the earliest epochs, and internal knowledge circuits reshaped as training proceeded. The warning is practical: a training curve is not a knowledge curve, and updating models safely requires measuring facts directly.
Catalog record
| CREATOR | Seyed Mahed Mousavi *, Simone Alghisi, Giuseppe Riccardi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-03858