!MSA at BAREC Shared Task 2025: Ensembling Arabic Transformers for Readability Assessment
Mohamed Basem, Mohamed Younes, Seif Ahmed, Abdelrahman Moustafa · 2025
Read !MSA at BAREC Shared Task 2025: Ensembling Arabic Transformers for Readability Assessment by Mohamed Basem, Mohamed Younes, Seif Ahmed, Abdelrahman Moustafa (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-10040.txt.
Introduction
Mohamed Basem and 3 co-authors report a winning system for judging Arabic reading difficulty in a 2025 shared task paper from the BAREC 2025 competition. Fine-grained readability scoring helps schools and publishers pitch texts at the right level, yet Arabic tools lag behind. Their entry blends four transformer models, each tuned with a different loss so each hears a different signal in the text, then fuses their answers weighted by confidence.
Severe class imbalance and scarce hard examples were fought with weighted training, careful preprocessing, relabelling of a major corpus by their strongest model, and synthetic examples from a commercial generator. A final correction for skewed predictions added a solid gain. First place in all six tracks followed, at sentence and document level alike.
Catalog record
| CREATOR | Mohamed Basem, Mohamed Younes, Seif Ahmed, Abdelrahman Moustafa |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-10040
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
!MSA at BAREC Shared Task 2025: Ensembling Arabic Transformers for Readability Assessment Mohamed Basem, Mohamed Younes, Seif Ahmed, Abdelrahman Moustafa Abstract We present !MSA’s winning system for the BAREC 2025 Shared Task on fine-grained Arabic readability assessment, achieving first place in six of six tracks. Our approach is a confidence-weighted ensemble of four complementary transformer models (AraBERTv2, AraELECTRA, MARBERT, and CAMeLBERT) each fine-tuned with distinct loss functions to capture diverse readability signals. To tackle severe class imbalance and data scarcity, we applied weighted training, advanced preprocessing, SAMER corpus re-labeling with our strongest model, and synthetic data generation via Gemini 2.5 Flash, adding 10k rare-level samples. A targeted post-processing step corrected prediction distribution skew, delivering a 6.3% Quadratic Weighted Kappa (QWK) gain. Our system reached 87.5% QWK at the sentence level and 87.4% at the document level, demonstrating the power of model and loss diversity, confidence-informed fusion, and intelligent augmentation for robust Arabic readability prediction. \fontspec_if_language:nTENG\addfontfeatureLanguage=English1 \fontspec_if_language:nTENG\addfontfeatureLanguage=English1 \fontspec_if_language:nT ENG \addfontfeature Language=English1 \fontspec_if_language:nT ENG \addfontfeature Language=Englishhttps://github.com/Mohamedbasem1/BAREC-2025 \fontspec_if_language:nT ENG \addfontfeature Language=English1 Introduction The BAREC 2025 Shared Task presents a formidable challenge for Arabic readability assessment—spanning six tracks (sentence- and document-level across strict, constrained, and open conditions) with a fine-grained 1–19 readability scale. Predicting exact labels across such a wide range significantly increases difficulty, as even small deviations can dramatically impact metrics like Quadratic Weighted Kappa. The challenge is further compounded by severe label imbalance, where certain readability levels occur far more frequently than others, biasing models toward majority classes and making rare-level prediction unreliable. In strict and constrained tracks, limited training data amplified these issues, and in constrained settings, incorporating external datasets like SAMER alhafni-etal-2024-samer ; al-khalil-etal-2020-large proved non-trivial due to mismatched label distributions—simple scaling approaches often resulted in misalignment and minimal performance gains. To address these challenges, we developed an ensemble framework that combines architectural and training diversity. We fine-tuned four transformer models (AraBERTv2 antoun-etal-2020-arabert , AraELECTRA antoun-etal-2021-araelectra , MARBERT abdul-mageed-etal-2021-arbert , and CAMeLBERT inoue-etal-2021-interplay ). Each model was trained with a distinct loss function (classification, regression, or ordinal). This design captures complementary signals, with outputs merged via confidence-weighted ensembling that favors more certain predictions. To mitigate data scarcity in the open tracks, we used prompt-engineered paraphrasing with the Gemini API to generate synthetic examples, and for SAMER, we re-labeled instances with our best BAREC-trained model instead of relying on naive scaling. Our approach demonstrates robustness across all track configurations of BAREC 2025, with first-place rankings in all six tracks. This success underlines the advantages of model and loss-function diversity, confidence-informed fusion, and intelligent data augmentation for Arabic readability prediction—setting a strong precedent for future research in fine-grained, limited-data NLP tasks. \fontspec_if_language:nT ENG \addfontfeature Language=English2 Background \fontspec_if_language:nT ENG \addfontfeature Language=English2.1 Task Details The BAREC 2025 Shared Task elmadani-etal-2025-barec-shared-task focuses on fine-grained Arabic readability assessment using the Balanced Arabic Readability Evaluation Corpus (BAREC) elmadani-etal-2025-large . BAREC is a large-scale dataset containing over 1 million words across 68,000+ sentences and 1,900+ documents, each annotated into 19 readability levels, where higher numbers indicate greater difficulty. The annotation process followed the official BAREC Annotation Guidelines habash-etal-2025-guidelines , which define linguistic and pedagogical principles to ensure consistency and reliability in labeling. The shared task defines two tasks: Sentence-Level & Document-Level Readability Assessment. Each one has three tracks based on permissible resources: \fontspec_if_language:nT ENG \addfontfeature Language=English• Strict Track: Use only BAREC Corpus. \fontspec_if_language:nT ENG \addfontfeature Language=English• Constrained Track: Use BAREC along with the SAMER Corpus and SAMER Lexicon alhafni-etal-2024-samer ; al-khalil-etal-2020-large . \fontspec_if_language:nT ENG \addfontfeature Language=English• Open Track: Use any additional resources or augmentation methods. We participated in all six tracks across both subtasks, exploring resource-limited, resource-augmented, and fully open settings. \fontspec_if_language:nT ENG \addfontfeature Language=English2.2 Related Work Arabic readability assessment has been studied from multiple perspectives. el-haj-etal-2024-dares introduced the DARES dataset for evaluating the readability of Arabic educational content, demonstrating the importance of domain-specific corpora for improving prediction accuracy. liberato-etal-2024-strategies proposed a hybrid approach combining handcrafted linguistic features with transformer-based models, yielding improved robustness on small or noisy datasets. elmadani-etal-2025-large presented BAREC, the largest balanced corpus for fine-grained Arabic readability assessment, alongside baseline systems for sentence- and document-level prediction. habash-etal-2025-guidelines detailed the annotation guidelines and methodology for BAREC, ensuring consistent application of the 19 readability levels. alhafni-etal-2024-samer & al-khalil-etal-2020-large introduced the SAMER Corpus and Lexicon, designed for Arabic text simplification and multi-level difficulty annotation, which are leveraged in the Constrained Track. Additional advances in Arabic NLP include ARBERT and MARBERT abdul-mageed-etal-2021-arbert , large-scale pre-trained models that achieve state-of-the-art performance across a variety of Arabic language understanding tasks, and ensemble-based modeling for Arabic dialect identification khered-etal-2022-building , which inspired aspects of our system design. \fontspec_if_language:nT ENG \addfontfeature Language=English3 System Overview \fontspec_if_language:nT ENG \addfontfeature Language=English3.1 Addressing Data Imbalance The BAREC dataset exhibits a highly imbalanced distribution across the 19 readability levels, with certain levels (e.g., 12 and 14) being far more frequent than rare levels such as 1, 18, and 19 (see Figure in Appendix). This imbalance biases models toward predicting frequent levels, which is particularly detrimental when the target metric is Quadratic Weighted Kappa (QWK), as misclassifying rare levels incurs a high penalty. To mitigate this, we computed class weights to encourage the model to pay more attention to rare classes. The weight for each class j is calculated as: w_{j}=\frac{n_{\text{samples}}}{n_{\text{classes}}\times n_{\text{samples in class }j}} \fontspec_if_language:nT ENG \addfontfeature Language=English(1) This formulation assigns higher weights to rarer classes and lower weights to frequent ones, reducing prediction bias and improving fairness across levels. \fontspec_if_language:nT ENG \addfontfeature Language=English3.2 Model Architectures and Loss Functions Our system builds on a diverse set of Arabic transformer models: AraBERTv2 antoun-etal-2020-arabert , AraELECTRA antoun-etal-2021-araelectra , MARBERT abdul-mageed-etal-2021-arbert , and CAMeLBERT inoue-etal-2021-interplay . These models were chosen for their strong track record in Arabic NLP benchmarks, their coverage of both Modern Standard Arabic and dialectal varieties, and their complementary pretraining objectives. We trained multiple variants of each model using different loss formulations to capture complementary perspectives on the readability prediction problem: \fontspec_if_language:nT ENG \addfontfeature Language=English• Cross-Entropy Loss (CE) for standard multi-class classification. \fontspec_if_language:nT ENG \addfontfeature Language=English• Mean Squared Error (MSE) for regression over the continuous readability scale. \fontspec_if_language:nT ENG \addfontfeature Language=English• Conditional Ordinal Regression (COR) for modeling the conditional probabilities of surpassing each readability threshold. It was implemented via the CORAL framework cao2021coral . This diversity allowed the ensemble to leverage both discrete and continuous interpretations of the readability scale while incorporating ordinal constraints. \fontspec_if_language:nT ENG \addfontfeature Language=English3.3 Constrained Track: SAMER Label Transformation For the Constrained Track, we incorporated the SAMER Corpus alhafni-etal-2024-samer , originally annotated on a 3–6 scale, into our training data. To align it with BAREC’s 1–19 scale, we applied a min–max scaling transformation: \text{Scaled\_Label}=\frac{\text{Label}-3}{6-3}\times(19-1)+1 \fontspec_if_language:nT ENG \addfontfeature Language=English(2) This transformation preserves the relative difficulty ordering while ensuring compatibility with BAREC’s fine-grained labeling. Initially, we trained our model using the scaled SAMER data and evaluated it on our BAREC test set, achieving a QWK of 50%. We then tried an alternative approach: using our best-performing BAREC-trained model directly to predict labels for the SAMER dataset on the 1–19 scale. Finally, we scaled the predictions back down to the original 3-6 SAMER range, verifying that the reverse transformation maintained accuracy within a margin of \pm 0.5 . This approach significantly improved results. \fontspec_if_language:nT ENG \addfontfeature Language=English3.4