Should We Still Pretrain Encoders with Masked Language Modeling?
Hippolyte Gisserot-Boukhlef, Nicolas Boizard, Manuel Faysse, Duarte M. Alves, Emmanuel Malherbe, André F. T. Martins, Céline Hudelot, Pierre Colombo, [1ex] Artefact Research Center Diabolocom Illuin Technology Equall Unbabel MICS, CentraleSupélec, Université Paris-Saclay Instituto Superior Técnico & · 2025
Read Should We Still Pretrain Encoders with Masked Language Modeling? by Hippolyte Gisserot-Boukhlef, Nicolas Boizard, Manuel Faysse, Duarte M. Alves, Emmanuel Malherbe, André F. T. Martins, Céline Hudelot, Pierre Colombo, [1ex] Artefact Research Center Diabolocom Illuin Technology Equall Unbabel MICS, CentraleSupélec, Université Paris-Saclay Instituto Superior Técnico & (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-00994.txt.
Introduction
Hippolyte Gisserot-Boukhlef and co-authors present a 2025 arXiv paper on how to pretrain text encoders. Encoders have long trained by guessing masked words, while decoder models trained to predict the next word can also serve as encoders. It was unclear whether the decoder edge came from the objective itself or from bigger scale. The team ran controlled comparisons to settle the question.
The group trained 38 models, from 210 million to one billion parameters, and ran over 15,000 fine-tuning checks. Masked training won on most representation tasks, while next-word training used data better and fine-tuned more stably. A two-phase plan, next-word first and masked second, did best on a fixed budget.
Catalog record
| CREATOR | Hippolyte Gisserot-Boukhlef, Nicolas Boizard, Manuel Faysse, Duarte M. Alves, Emmanuel Malherbe, André F. T. Martins, Céline Hudelot, Pierre Colombo, [1ex] Artefact Research Center Diabolocom Illuin Technology Equall Unbabel MICS, CentraleSupélec, Université Paris-Saclay Instituto Superior Técnico & |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-00994
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Should We Still Pretrain Encoders with Masked Language Modeling? Hippolyte Gisserot-Boukhlef, Nicolas Boizard, Manuel Faysse, Duarte M. Alves, Emmanuel Malherbe, André F. T. Martins, Céline Hudelot, Pierre Colombo, [1ex] Artefact Research Center Diabolocom Illuin Technology Equall Unbabel MICS, CentraleSupélec, Université Paris-Saclay Instituto Superior Técnico & Universidade de Lisboa (Lisbon ELLIS Unit) Instituto de Telecomunicações [email protected] Abstract Learning high-quality text representations is fundamental to a wide range of NLP tasks. While encoder pretraining has traditionally relied on Masked Language Modeling (MLM), recent evidence suggests that decoder models pretrained with Causal Language Modeling (CLM) can be effectively repurposed as encoders, often surpassing traditional encoders on text representation benchmarks. However, it remains unclear whether these gains reflect an inherent advantage of the CLM objective or arise from confounding factors such as model and data scale. In this paper, we address this question through a series of large-scale, carefully controlled pretraining ablations, training a total of 38 models ranging from 210 million to 1 billion parameters, and conducting over 15,000 fine-tuning and evaluation runs. We find that while training with MLM generally yields better performance across text representation tasks, CLM-trained models are more data-efficient and demonstrate improved fine-tuning stability. Building on these findings, we experimentally show that a biphasic training strategy that sequentially applies CLM and then MLM, achieves optimal performance under a fixed computational training budget. Moreover, we demonstrate that this strategy becomes more appealing when initializing from readily available pretrained CLM models, reducing the computational burden needed to train best-in-class encoder models. We release all project artifacts at https://hf.co/MLMvsCLM to foster further research. * * footnotetext: Equal contribution 1 Introduction Figure 1: Experimental setup overview and key results on sequence classification (610M model size, 40% MLM ratio). The upper plot shows downstream performance as a function of the CLM–to-MLM step ratio during pretraining (configurations outperforming MLM-only are highlighted in yellow). The lower plot illustrates the effect of applying MLM CPT to models initially trained with either MLM or CLM (Base). Learning meaningful representations is essential for a wide range of NLP tasks, such as sequence classification, named entity recognition, extractive question answering, and information retrieval. Traditionally, these representations are learned with encoders pretrained solely with Masked Language Modeling (MLM) ( Devlin et al. 2019 ) . More recently, however, decoder models pretrained with Causal Language Modeling (CLM) and later adapted with MLM have challenged the MLM-only paradigm ( BehnamGhader et al. 2024 ) , achieving state-of-the-art results on the Massive Text Embedding Benchmark (MTEB) ( Muennighoff et al. 2023 ; Enevoldsen et al. 2025 ) . Yet, these results have so far only been observed on models that are significantly larger than typical encoders and trained on substantially more data ( Lee et al. 2024 ; Meng et al. 2024 ; Kim et al. 2024 ; Muennighoff et al. 2024 ) . As a result, it remains unclear whether the success of this new training paradigm stems from the causal objective itself, or merely from increased scale. In this paper, we conduct a controlled study to investigate the impact of training with MLM and CLM objectives when learning textual representations. We compare models with equal architecture and size, trained on equal amount of data, and evaluate them on an extensive range of text representation tasks. We start by comparing pretraining exclusively on either objective, and then investigate a two-stage pretraining alternative, where training starts with the CLM objective and is followed by MLM. Additionally, given the availability of existing pretrained models, we investigate a continued pretraining scenario, where models are initialized from checkpoints trained exclusively with MLM or CLM, and then trained with MLM for a fraction of the pretraining steps. In total, we train a total of 38 models ranging from 210 million to 1 billion parameters and conduct over 15,000 fine-tuning and evaluation runs, amounting to 110k MI250X GPU hours. Backed by these large-scale experiments, our results show that: • Although CLM training delivers strong performance on certain tasks and demonstrate good data efficiency and fine-tuning stability, MLM remains essential for achieving robust performance across all tasks, underscoring that bidirectional training remains indispensable ( ). • When pretraining a model from scratch, a two-stage protocol that first applies CLM followed by MLM offers a data-efficient way to build strong text representations by combining the strengths of both objectives ( ). • In a continued pretraining setting, adapting a CLM-pretrained model with MLM proves more effective than continuing MLM training from an MLM-pretrained model. This result suggests that leveraging widely pretrained decoder models is currently the best approach to obtain a strong encoder model ( ). To facilitate reproducibility and support future research, we release all pretrained model checkpoints, 1 1 1 https://huggingface.co/MLMvsCLM along with the training 2 2 2 https://github.com/Nicolas-BZRD/EuroBERT/tree/MLM_vs_CLM and evaluation code. 3 3 3 https://github.com/hgissbkh/EncodEval/tree/MLM_vs_CLM 2 Experimental Setup In this section, we describe our controlled experimental setup, covering model architectures, pretraining and fine-tuning protocols, as summarized in . Models. The model architectures closely follow those of the EuroBERT models ( Boizard et al. 2025 ) , with sizes of 210M, 610M, and 1B parameters. All models use a maximum context length of 2,048 tokens and a RoPE \theta value of 10,000. Pretraining data. Models are trained on unique English tokens from the FineWeb-Edu dataset ( Penedo et al. 2024 ) , which is known for supporting efficient model training. To ensure fair comparison across model configurations and training setups, all models are exposed to the same sequence of samples during training. Pretraining objectives. Models are trained using one of 3 approaches: 1. CLM uses next-token prediction, where each token is predicted autoregressively using a causal attention mask. The training objective is to minimize the negative log-likelihood: \mathcal{L}_{\text{CLM}}=-\sum_{t=1}\log P(x_{t}\mid x_{<t})\ \ \ \text{(causal attention)} (1) where x=(x_{1},x_{2},\ldots,x_{T}) is the input sequence and P(x_{t}\mid x_{<t}) is the model’s predicted distribution over the vocabulary given the preceding tokens. 2. MLM , by contrast, randomly masks a subset of tokens and trains the model to reconstruct them using a bidirectional attention mask. The objective is: \mathcal{L}_{\text{MLM}}=-\sum_{i\in\mathcal{M}}\log P(x_{i}\mid x_{\setminus\mathcal{M}})\ \ \ \text{(bidirectional attention)} (2) where \mathcal{M}\subset\{1,\ldots,T\} denotes the indices of masked tokens, and x_{\setminus\mathcal{M}} is the input sequence with masked tokens replaced by special placeholders. We experiment with masking ratios of 20%, 30%, 40%, and 50%. 3. Finally, the two-stage CLM+MLM approach sequentially applies CLM pretraining followed by MLM. Pretraining hyperparameters. Pretraining is performed with a per-device batch size of 12 samples across 192 GPUs, yielding an effective batch size of 2,373,120 tokens. We employ a Warmup-Stable-Decay (WSD) learning rate schedule: a 2,000-step warmup phase, followed by 38,000 steps with a constant learning rate of 5e-4, ending with a 2,000-step linear decay phase, for a total of 42,000 training steps. Pretraining setups. To reflect two common real-world scenarios, we define two distinct pretraining setups, depending on whether the goal is to train a model from scratch or to continue training from an existing checkpoint: 1. Pretraining From Scratch (PFS) : Models are trained from random initialization for a fixed number of steps using one of 3 objectives: CLM, MLM, or sequential CLM+MLM. For CLM and MLM, a standard WSD learning rate scheduler is applied. For CLM+MLM models, training is first performed with the