A Causal Language Modeling Detour Improves Encoder Continued Pretraining
Rian Touchent, Éric de la Clergerie · 2026
Read A Causal Language Modeling Detour Improves Encoder Continued Pretraining by Rian Touchent, Éric de la Clergerie (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12438.txt.
Description
Touchent and de la Clergerie improve domain adaptation for encoders with a short detour through causal language modelling. The usual route continues training with masked modelling alone. They switch briefly to causal modelling, then decay back to masked modelling, and test the recipe on biomedical French and English tasks with ModernBERT. Gains reach a few points in French and under a point in English, depending on model size. The benefit traces to low layers, persists through the decay phase, and grows with model size.
Catalog record
| CREATOR | Rian Touchent, Éric de la Clergerie |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25137 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.12438