BnMMLU : Measuring Massive Multitask Language Understanding in Bengali
Saman Sarker Joy, Swakkhar Shatabda · 2025
Read BnMMLU : Measuring Massive Multitask Language Understanding in Bengali by Saman Sarker Joy, Swakkhar Shatabda (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18951.txt.
Introduction
Saman Sarker Joy and Swakkhar Shatabda build BnMMLU, a large test of language models in Bengali, a tongue left out of most such tests. Their paper is from two thousand twenty-five.
The set spans forty-one fields, from math and science to the humanities and social fields. It holds a vast count of multiple-choice question and option pairs, the most for Bengali to date. Math is kept in a form machines can read. A hard subset holds the questions top systems miss most. The team tests twenty-four model forms from eleven lines, in two styles of prompt and two counts of shown cases.
Catalog record
| CREATOR | Saman Sarker Joy, Swakkhar Shatabda |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-18951
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BnMMLU : Measuring Massive Multitask Language Understanding in Bengali Saman Sarker Joy, Swakkhar Shatabda Abstract Large-scale multitask benchmarks have driven rapid progress in language modeling, yet most emphasize high-resource languages such as English, leaving Bengali underrepresented. We present BnMMLU, a comprehensive benchmark for measuring massive multitask language understanding in Bengali. BnMMLU spans 41 domains across STEM, humanities, social sciences, and general knowledge, and contains 134,375 multiple-choice question–option pairs-the most extensive Bengali evaluation suite to date. The dataset preserves mathematical content via MathML, and includes BnMMLU-HARD, a compact subset constructed from questions most frequently missed by top systems to stress difficult cases. We benchmark 24 model variants across 11 LLM families, spanning open-weights general/multilingual, Bengali-centric open-weights, and proprietary models, covering multiple parameter scales and instruction-tuned settings. We evaluate models under standardized protocols covering two prompting styles (Direct vs. Chain-of-Thought) and two context regimes (0-shot vs. 5-shot), reporting accuracy consistently across families. Our analysis highlights persistent gaps in reasoning and application skills and indicates sublinear returns to scale across model sizes. We release the dataset and evaluation templates to support rigorous, reproducible assessment of Bengali language understanding and to catalyze progress in multilingual NLP. Dataset Format # Items # Subjects Math S:H:SS:O BanglaQuAD Rony et al. (2024) Extractive 30,808 14 ✗ 3:4:5:2 BanglaRQA Ekram et al. (2022) Extractive 14,889 20 ✗ 1:2:2:3 BEnQA Shafayat et al. (2024) MCQ 5,161 5 ✓ 1:0:0:0 BLUCK Kabir et al. (2025) MCQ 2,366 23 ✗ 0:1637:729:0 NOIRBETTIK Aurpa et al. (2025) MCQ 5,215 8 ✗ 2:8:1:4 TituLM-Bangla MMLU Nahin et al. (2025a) MCQ 87,869 11 ✗ 98:19:17:1 UDDIPOK Aurpa et al. (2023) Extractive 3,636 – ✗ – BnMMLU MCQ 134,375 41 ✓ 4:2:3:1 Table 1: Comparison of prominent Bengali QA datasets. The table lists format (extractive vs. multiple choice), size (items and subjects), preservation of mathematical content (MathML), and proportional distribution across STEM, Humanities, Social Sciences and Others. 1 Introduction The advancement of natural language processing (NLP) has been significantly driven by large-scale benchmarks that assess the capabilities of language models across various domains. Among these, the Massive Multitask Language Understanding (MMLU) Hendrycks et al. (2021) benchmark has emerged as a widely recognized evaluation framework. MMLU covers 57 diverse subjects, spanning disciplines such as mathematics, science, humanities, history, law, medicine and general knowledge. It is designed to measure a model’s ability to generalize across multiple domains. While MMLU has significantly contributed to evaluating models in high-resource languages like English, it provides little to no coverage for low-resource languages. Although Bengali 1 1 1 We use Bengali and Bangla interchangeably to denote the same language (ISO 639-1: bn ; ISO 639-3: ben ). The IANA Language Subtag Registry entry for bn lists both ( https://www.iana.org/assignments/language-subtag-registry ). is the seventh most spoken language globally Eberhard et al. (2025) , Bengali remains underrepresented in NLP research, with limited high-quality datasets, pre-trained models and benchmarks. The absence of a standardized knowledge-driven evaluation data set for Bengali language models restricts their ability to generalize across real-world tasks. While some multilingual benchmarks include Bengali Kakwani et al. (2020) , their coverage is sparse and does not adequately test subject-specific knowledge or reasoning skills in Bengali In the absence of such a benchmark, researchers lack the means to assess whether a model’s responses in Bengali reflect genuine understanding, memorization of bilingual cues or hallucination. Our study is guided by the following: (RQ1) How far do multilingual vs. Bengali-centric models transfer to native Bengali tasks across various domains? (RQ2) What are the returns to scale under standardized prompting/context regimes? (RQ3) When does elicited reasoning help (or hurt), especially on difficult items? (RQ4) Which subject areas are systematically hard vs. easy across different LLMs? To address these questions, we introduce BnMMLU, a benchmark to evaluate the multitask language understanding of Bengali in language models. Our contributions in this work are: • A 41-domain MCQ suite with 134,375 spanning STEM, humanities, social science and general knowledge. • Introduced BnMMLU-HARD , formed by ranking questions most frequently missed by models while preserving subdomain balance for stress testing. • Evaluated 24 model variants, spanning open-weights general/multilingual, Bengali-centric open-weights and proprietary models. • Comparable reporting across Direct vs. CoT and 0-shot vs. 5-shot settings and Reasoning and Non-reasoning comparisons with consistent prompts and accuracy metrics. Figure 1: An overview of the pipeline for constructing the BnMMLU benchmark. 2 Related Work The Massive Multitask Language Understanding (MMLU) benchmark Hendrycks et al. (2021) set a standard for evaluating language models on broad domain knowledge (e.g., mathematics, science, humanities, law), but is essentially English-centric and does not capture the linguistic, cultural and syntactic nuances of other languages. Language-specific MMLU-style benchmarks extend this paradigm to local exams: KMMLU for Korean Son et al. (2024) , CMMLU for Chinese Li et al. (2024) , and ArabicMMLU for Modern Standard Arabic Koto et al. (2024) , all reporting that non-English models still lag behind their English counterparts. In the multilingual setting, IndicGLUE Kakwani et al. (2020) and XGLUE Liang et al. (2020) include Bengali among many languages and cover tasks such as classification, sentiment analysis, NER and QA, but they are not broad multitask knowledge benchmarks in the MMLU sense. For Bengali specifically, existing resources such as BanglaQuAD Rony et al. (2024) , BanglaRQA Ekram et al. (2022) , BEnQA Shafayat et al. (2024) , and related datasets (e.g., NOIRBETTIK, BLUCK) provide task-specific QA and reading-comprehension style evaluations, often focusing on span extraction or short-answer questions rather than structured, curriculum-style subject coverage. TituLM-Bangla MMLU Nahin et al. (2025a) adapts MMLU-style diagnostics to Bengali multiple-choice questions across various topics, but with narrower subject breadth and less fine-grained coverage than our BnMMLU , which targets a wider set of Bengali academic and professional domains for multitask knowledge and reasoning evaluation. 3 The BnMMLU Benchmark We create BnMMLU, a multitask benchmark composed of multiple-choice question–answer pairs across 41 subjects spanning STEM, humanities, social sciences and other domains. We refer to this complete benchmark as BnMMLU-FULL throughout the remainder of the paper. The overview of the full pipeline is shown in Figure 1 . 3.1 Dataset Construction The questions were sourced from Bangladeshi educational and professional materials through two channels. Physical Resources. Scanned pages from NCTB-approved textbooks and competitive exam guides, processed using OCR tool with post-correction for script accuracy. Due to the unstructured formatting of many print materials, 20% of the data came from these sources, and they did not contain properly formatted multiple-choice questions and answers. Examples of these books are shown in Figure 5 . Digital Sources. Web-scraped questions from Bangladeshi educational portals that host structured, exam-style multiple-choice questions. The web scraping was performed using Selenium 2 2 2 https://www.selenium.dev and BeautifulSoup 3 3 3 https://pypi.org/project/beautifulsoup4 . The majority of the dataset, around 80% of the data came from these digital sources. 3.2 Optical Character Recognition (OCR) & Post-Correction We scan printed book pages, apply a standard pre-processing pipeline (grayscale conversion, adaptive binarisation, and deskewing) and then run OCR system followed by LLM-based copy-editing to clean the text while preserving math and answer keys. Full implementation details and the exact copy-editing prompt are provided in Appendix B . Post-correction reduced formatting issues and spelling errors. Additionally, approximately 10% of the question-option pairs were manually reviewed by