A Causal Language Modeling Detour Improves Encoder Continued Pretraining

Rian Touchent, Éric de la Clergerie · 2026

Read A Causal Language Modeling Detour Improves Encoder Continued Pretraining by Rian Touchent, Éric de la Clergerie (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12438.txt.

Description

Touchent and de la Clergerie improve domain adaptation for encoders with a short detour through causal language modelling. The usual route continues training with masked modelling alone. They switch briefly to causal modelling, then decay back to masked modelling, and test the recipe on biomedical French and English tasks with ModernBERT. Gains reach a few points in French and under a point in English, depending on model size. The benefit traces to low layers, persists through the decay phase, and grows with model size.

Catalog record

CREATORRian Touchent, Éric de la Clergerie
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25137
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2605.12438

Browse the full library · About the Tower of Babel