ModernBERT or DeBERTaV3? Examining Architecture and Data Influence on Transformer Encoder Models Performance
Wissam Antoun, Benoît Sagot & Djamé Seddah · 2025
Read ModernBERT or DeBERTaV3? Examining Architecture and Data Influence on Transformer Encoder Models Performance by Wissam Antoun, Benoît Sagot & Djamé Seddah (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-08716.txt.
Introduction
Antoun, Sagot, and Seddah run a controlled contest between two modern transformer encoders in this 2025 paper. ModernBERT's makers report wins over DeBERTaV3, but its training data is secret. That leaves open whether design or data deserves the credit.
The authors pretrained ModernBERT on the exact dataset behind a French DeBERTaV3 model. With data held equal, the older design won on sample efficiency and overall benchmarks. ModernBERT's real edges were long context, faster training, and faster inference. Good data sped up training but did not lift final scores much, hinting benchmarks may be saturating. Both designs still clearly beat the earlier BERT family. The study argues for separating data from design in model claims.
Catalog record
| CREATOR | Wissam Antoun, Benoît Sagot & Djamé Seddah |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-08716
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ModernBERT or DeBERTaV3? Examining Architecture and Data Influence on Transformer Encoder Models Performance Wissam Antoun, Benoît Sagot & Djamé Seddah Abstract Pretrained transformer-encoder models like DeBERTaV3 and ModernBERT introduce architectural advancements aimed at improving efficiency and performance. Although the authors of ModernBERT report improved performance over DeBERTaV3 on several benchmarks, the lack of disclosed training data and the absence of comparisons using a shared dataset make it difficult to determine whether these gains are due to architectural improvements or differences in training data. In this work, we conduct a controlled study by pretraining ModernBERT on the same dataset as CamemBERTaV2, a DeBERTaV3 French model, isolating the effect of model design. Our results show that the previous model generation remains superior in sample efficiency and overall benchmark performance, with ModernBERT’s primary advantage being faster training and inference speed. However, the new proposed model still provides meaningful architectural improvements compared to earlier models such as BERT and RoBERTa. Additionally, we observe that high-quality pre-training data accelerates convergence but does not significantly improve final performance, suggesting potential benchmark saturation. These findings show the importance of disentangling pretraining data from architectural innovations when evaluating transformer models. 1 Introduction Despite the widespread adoption of decoder-only large language models (LLMs) in our post-ChatGPT era, encoder-only transformers such as BERT ( Devlin et al. 2019 ) continue to play a central role in many NLP applications. These models remain the backbone for a wide range of non-generative tasks such as classification, named entity recognition (NER), and retrieval-based systems, especially in high-throughput or latency-sensitive environments. Their relatively low compute requirements and strong performance across standard benchmarks make them a practical choice for large-scale deployment, including in Retrieval-Augmented Generation (RAG) ( Lewis et al. 2020 ) pipelines or guardrails systems ( Neill et al. 2024 ) . Continuous architectural and training objective improvements have led to more performant and efficient encoder-only transformer variants, among which DeBERTaV3 ( He et al. 2021a ) and the recently proposed ModernBERT ( Warner et al. 2024 ) stand out as major improvements. According to Warner et al. 2024 , their model reports superior performance relative to DeBERTaV3, the previous state-of-art model, across several popular NLP benchmarks. However, interpreting these performance improvements is challenging due to the lack of details regarding their training data. Without comparisons conducted on identical datasets, it remains unclear whether the reported gains reflect genuine architectural advances or simply differences arising from the choice of training data. This uncertainty motivates our study aimed to evaluate the impact of architecture versus training data by conducting a controlled comparison among ModernBERT, DeBERTaV3, and RoBERTa models. We select CamemBERTaV2 ( Antoun et al. 2024 ) , a French DeBERTaV3 model trained from scratch, as our primary reference since both its intermediate checkpoints and training dataset are publicly available. Additionally, we include CamemBERTv2 ( Antoun et al. 2024 ) , a RoBERTa ( Liu 2019 ) based model pretrained on the same dataset, to comprehensively assess how ModernBERT’s architectural advancements compare not only against the latest DeBERTaV3-based models but also against more traditional BERT/RoBERTa architectures. Leveraging these resources, we pretrained a French ModernBERT using the exact same dataset as CamemBERTaV2 and CamemBERTv2, thus ensuring that differences in performance directly reflect architectural variations rather than dataset composition or quality. Furthermore, we pretrained another ModernBERT variant on a carefully curated, high-quality French corpus to further explore the role of dataset quality in model performance. The key takeaways from our comprehensive experiments are as follows: • When controlling for dataset differences, DeBERTaV3 outperforms ModernBERT in terms of overall benchmark performance and training sample efficiency, indicating that DeBERTaV3’s architecture and training objective optimization provide superior learning capabilities compared to ModernBERT’s efficiency-oriented design. • Nonetheless, ModernBERT presents a clear practical advantage due to its significantly faster training and inference speeds, driven by an orthogonal set of optimizations. Moreover, while ModernBERT may not surpass DeBERTaV3, it offers meaningful improvements over previous transformer-based models such as BERT and RoBERTa. • We also observe that training models on high-quality, filtered datasets results in faster convergence but does not substantially increase final performance metrics. This finding highlights a potential limitation of current NLP benchmarks, suggesting possible saturation that prevents fine-grained discrimination between models of similar performance. Our findings show the importance of clearly separating respective effects of architectural changes and training datasets when evaluating NLP models. Our controlled comparison using the same pretraining dataset provides more accurate insights into the strengths and limitations of ModernBERT and DeBERTaV3 architectures To foster further research and reproducibility, we publicly release both of our pretrained French ModernBERT models, including the variant trained on the CamemBERTaV2 dataset and on our high-quality filtered corpus. These models, along with training scripts, intermediate checkpoints, and evaluation results, are available on HuggingFace 1 1 1 https://huggingface.co/collections/almanach/moderncamembert-67f7e6d85ede5f7cfc1ce012 . 2 Literature Review Transformer-based language models have become the cornerstone of modern NLP, starting with BERT ( Devlin et al. 2019 ) , which introduced masked language modeling (MLM) and next sentence prediction as self-supervised pretraining tasks used to pretrain encoder-only transformer models. RoBERTa ( Liu 2019 ) subsequently improved upon BERT by removing the next sentence prediction objective, training on larger corpora, and applying more robust optimization techniques. Despite these advances, both BERT and RoBERTa shared a fundamental architectural limitation: they used absolute positional embeddings and standard attention mechanisms that lacked efficiency and fine-grained contextual representation. To address these limitations, DeBERTa ( He et al. 2021b ) introduced a disentangled attention mechanism, decoupling content and positional information, thereby improving the model’s ability to generalize across contexts. DeBERTaV3 ( He et al. 2021a ) further extended these innovations by incorporating Replaced Token Detection (RTD) ( Clark et al. 2020 ) for more sample-efficient training, as well as Gradient-Disentangled Embedding Sharing (GDES) to prevent conflicting updates in shared embeddings between the generator and discriminator during training. In parallel, architectural and efficiency-driven improvements have become an active area of research. ModernBERT ( Warner et al. 2024 ) was recently proposed to modernize the BERT architecture by incorporating a suite of design choices aimed at improving inference speed, training throughput, and context window size. These include FlashAttention ( Dao et al. 2022 ; Dao 2024 ; Shah et al. 2024 ) , alternating global and local attention layers ( Team et al. 2024 ) , sequence packing ( Portes et al. 2023 ) , and rotary positional embeddings (RoPE) ( Su et al. 2021 ) . ModernBERT also removes architectural elements such as bias terms and introduces GeGLU ( Shazeer 2020 ) activations, making it a strong contender for production scenarios requiring high efficiency. While the authors of ModernBERT report superior benchmark performance over DeBERTaV3, the absence of transparent training data and lack of head-to-head comparisons on shared datasets introduces ambiguity. It is thus unclear whether the reported improvements are driven by architectural enhancements or the underlying training data. 3 Methodology We conduct a controlled study focusing on the performance of ModernBERT compared to DeBERTaV3-based and RoBERTa-based models. Our goal is to identify and separate architectural improvements from data-driven performance differences, addressing ambiguities in prior studies that used undisclosed datasets. 3.1 Pre-training Datasets Our experiments involve two distinct pre-training datasets: CamemBERTaV2 Original Dataset. We first make use of