T abi BERT :A Large-Scale ModernBERT Foundation Model and A Unified Benchmark for Turkish

Melikşah Türker 1,2, Asude Ebrar Kızıloğlu 1∗, Onur Güngör 1, and Susan Üsküdarlı 1 · 2025

Read T abi BERT :A Large-Scale ModernBERT Foundation Model and A Unified Benchmark for Turkish by Melikşah Türker 1,2, Asude Ebrar Kızıloğlu 1∗, Onur Güngör 1, and Susan Üsküdarlı 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-23065.txt.

Introduction

Melikşah Türker and co-authors introduce TabiBERT, a language model built for Turkish. Encoder models like BERT remain the workhorses for classification, tagging, and retrieval. Yet Turkish had no monolingual encoder trained from scratch with the modern improvements of the ModernBERT design, such as rotary embeddings and long context. TabiBERT fills that gap, pretrained on one trillion tokens drawn from a multi-domain corpus of web text, scientific writing, code, and mathematics.

The model handles contexts of more than eight thousand tokens, sixteen times longer than earlier Turkish BERT models. For fair measurement the team built TabiBench, a benchmark of twenty-seven datasets across eight task categories with standardized splits. TabiBERT led Turkish models in five categories and beat the previous best, BERTurk, in six, with the biggest gains in question answering and code retrieval. Weights, configurations, and evaluation code are released for reproducible Turkish research.

Catalog record

CREATORMelikşah Türker 1,2, Asude Ebrar Kızıloğlu 1∗, Onur Güngör 1, and Susan Üsküdarlı 1
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE92 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-23065

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

T abi BERT :A Large-Scale ModernBERT Foundation Model and A Unified Benchmark for Turkish Melikşah Türker 1,2, Asude Ebrar Kızıloğlu 1∗, Onur Güngör 1, and Susan Üsküdarlı 1 Abstract The introduction of BERT established encoder-only Transformer models as a foundational paradigm in natural language processing, motivating extensive research on architectural design and pretraining strategies. Subsequent work addressed limitations of early encoder models through improvements in computational efficiency, training stability, context length, and positional encoding. ModernBERT consolidated these improvements by integrating rotary positional embeddings, FlashAttention-based attention implementations, and refined normalization schemes to support longer contexts and more stable training. Unfortunately, the Turkish NLP landscape lacks a monolingual encoder trained from scratch that systematically incorporates these modern architectural advances. This work introduces T abi BERT , a monolingual Turkish encoder based on the ModernBERT architecture that is trained from scratch on a large, carefully curated corpus. T abi BERT is pretrained on one trillion tokens sampled from an 84.88B token multi-domain corpus comprising web text (73%), scientific publications (20%), source code (6%), and mathematical content (0.3%). The model supports a context length of 8,192 tokens, which is 16x of the original BERT models. It achieves an inference speedup of up to 2.65x while reducing GPU memory consumption, allowing for larger batch sizes during training and inference. To support rigorous and reproducible evaluation, we introduce the T abi B ench benchmark with 28 datasets across eight task categories with standardized splits and evaluation protocols. Performance is summarized using a GLUE-style macro-average score. On T abi B ench, T abi BERT attains a score of 77.58, outperforming BERT urk by 1.62 points and establishing new state-of-the-art results on five of eight task categories, with particularly strong gains on question answering (+9.55 points), code retrieval (+2.41 points), and academic understanding (+0.66 points). When compared with task-specific prior best results, including specialized models such as TurkishBERTweet, T abi BERT achieves an average improvement of 1.47 points, indicating robust cross-domain generalization. We release model weights, training configurations, and evaluation code to provide a transparent and reproducible foundation for future Turkish encoder research. \lefttitle T abi BERT :A Large-Scale ModernBERT Foundation Model and A Unified Benchmark for Turkish \righttitle Natural Language Processing \papertitle T abi BERT :A Large-Scale ModernBERT Foundation Model and A Unified Benchmark for Turkish \jnlPage T abi BERT :A Large-Scale ModernBERT Foundation Model and A Unified Benchmark for Turkish 13 \jnlDoiYr 2025 \doival 10.1017/xxxxx {authgrp} 1 1 footnotetext: The work was conducted while the author was affiliated with the Department of Computer Engineering, Boğaziçi University, Istanbul, Türkiye. Current address: Technical University of Munich (TUM), Munich, Germany. \history (Received xx xxx xxx; revised xx xxx xxx; accepted xx xxx xxx) 1 Introduction The emergence of pretrained transformer encoder models has become central to contemporary natural language processing, providing general-purpose representations that support a wide range of downstream tasks. The architectural framework introduced by Bidirectional Encoder Representations from Transformers (BERT) ( 33 ) has been a foundational component of natural language understanding, enabling the induction of deeply contextualized linguistic features through self-supervised learning on large-scale corpora. BERT was first trained on large English text corpora and achieved strong baseline performance on a wide range of natural language processing tasks, serving as the basis for many later pretrained language models. The success of BERT motivated, and thus was followed by extensive research, leading to resource-efficient variants ( 57 ; 78 ; 49 ; 52 ; 24 ) , domain-specialized models ( 13 ; 47 ; 53 ; 20 ) , and multilingual architectures designed to support cross-lingual transfer ( 69 ; 104 ; 26 ) . Initial refinements focused on pretraining procedures that improved performance without changing the core architecture. For example, RoBERTa ( 57 ) achieved improved performance via larger and more diverse pretraining corpora, dynamic masking strategies, optimized training regimes, and by eliminating the next sentence prediction objective without altering the original transformer architecture. Several works concurrently focused on efficiency through knowledge distillation ( 78 ) , parameter sharing strategies ( 52 ) , and alternative pretraining objectives ( 24 ) , enabling computational gains while maintaining or improving performance. Subsequent architectural innovations sought to address BERT’s quadratic complexity, which stems from the self-attention mechanism and limits sequence length and computational efficiency. 14 introduced sparse attention to enable longer sequences, while IO-aware implementations significantly reduced memory overhead ( 29 ) . Positional encoding improvements, such as Rotary Position Embeddings (RoPE) ( 85 ) resulted in improved length extrapolation compared to BERT’s original absolute positional embeddings. Normalization strategies were introduced to improve training stability for deeper architectures ( 113 ; 121 ) . These accumulated improvements were synthesized into ModernBERT, an updated encoder-only architecture that incorporates rotary positional embeddings, FlashAttention technology, alternating attention patterns, and refined pretraining procedures ( 108 ) . Its architecture supports context windows up to 8,192 tokens, sixteen times BERT’s original context length of 512, with improved computational efficiency that addresses limitations stemming from fine-tuning instability and restricted sequence length ( 113 ; 57 ) . While these architectural advances have been widely adopted for high-resource languages, Turkish NLP remains primarily centered on earlier architectural paradigms. BERTurk ( 80 ) laid the foundation for transformer-based Turkish NLP as the first monolingual BERT model pretrained from scratch and publicly released. Its widespread adoption across diverse downstream tasks has made it the de facto standard for Turkish language understanding. Following, subsequent models such as ConvBERTurk and various ELECTRA adaptations have emerged on community platforms, although they frequently prioritized immediate availability over technical documentation. Critical specifications regarding corpus composition, preprocessing pipelines, and systematic evaluation protocols are often underspecified, which impedes reproducibility and limits the ability of the research community to build systematically on prior work. Recent large-scale multilingual efforts have incorporated Turkish alongside other languages. Multilingual BERT (mBERT) ( 33 ) and XLM-R ( 26 ) share parameters and vocabulary across typologically diverse languages, which dilutes representational capacity for individual languages ( 26 ; 104 ) . For morphologically rich languages, multilingual tokenizers produce inefficient segmentations, leading to increased token fertility, longer effective sequences, and reduced vocabulary utilization ( 75 ; 44 ) . Table 1: Turkish BERT-based models across scale, training data, architectural features, and benchmark performance. Model # Params (M) Data Size (GB) Context Length Code & Math Flash Attention T abi B ench Score T urkish BERT weet 163 110 512 ✗ ✗ 67.48 ytu-c osmos- BERT 111 75 512 ✗ ✗ 72.26 BERT urk 110 35 512 ✗ ✗ 75.96 T abi BERT 149 426 8192 ✓ ✓ 77.58 This work identifies and addresses two distinct gaps in Turkish NLP research. First, an architectural gap exists: no monolingual Turkish encoder has been trained from scratch to incorporate the efficiency, stability, and representational improvements synthesized in ModernBERT. Second, an evaluation gap persists due to the lack of a standardized, reproducible benchmarking framework, hindering systematic comparison across model generations and architectural paradigms. We address these gaps by introducing T abi BERT , a monolingual Turkish encoder, based on the ModernBERT architecture, and T abi B ench, a unified evaluation benchmark with standardized protocols. T abi BERT is a monolingual Turkish encoder based on the ModernBERT architecture, trained from scratch on a large, carefully curated

Read the complete text