Evaluating LLMs’ Multilingual Capabilities for Bengali: Benchmark Creation and Performance Analysis

Shimanto Bhowmik, Tawsif Tashwar Dipto, Md Sazzad Islam, Sheryl Hsu, Tahsin Reasat · 2025

Read Evaluating LLMs’ Multilingual Capabilities for Bengali: Benchmark Creation and Performance Analysis by Shimanto Bhowmik, Tawsif Tashwar Dipto, Md Sazzad Islam, Sheryl Hsu, Tahsin Reasat (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-23248.txt.

Introduction

Shimanto Bhowmik and co-authors probe why open language models score worse in Bengali than in English, in a paper from two thousand twenty-five. The team turned eight English test sets into Bengali with one shared method and ran ten open models on both tongues. Two suspects were studied: how text cutters chop Bengali script, and how strict answer formats punish small slips.

Scores did fall in Bengali, but part of the gap came from format rules, not lack of knowledge. Format slips skewed the gap several-fold for some models and even flipped its sign for one model. Bengali also cost about five times more tokens per word, set by cutter design. Readers get released data and code to test fairly.

Catalog record

CREATORShimanto Bhowmik, Tawsif Tashwar Dipto, Md Sazzad Islam, Sheryl Hsu, Tahsin Reasat
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-23248

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluating LLMs’ Multilingual Capabilities for Bengali: Benchmark Creation and Performance Analysis Shimanto Bhowmik, Tawsif Tashwar Dipto, Md Sazzad Islam, Sheryl Hsu, Tahsin Reasat Abstract Bengali is an underrepresented language in NLP research. However, it remains a challenge due to its unique linguistic structure and computational constraints. In this work, we systematically investigate the challenges that hinder Bengali NLP performance by focusing on the absence of standardized evaluation benchmarks. We then evaluated 10 recent open source Large Language Models (LLMs) in 8 of the translated datasets and performed a comprehensive error analysis to pinpoint their primary failure modes. Our findings reveal consistent performance gaps for Bengali compared to English, particularly for smaller models and specific model families like Mistral. We also identified promising robustness in certain architectures, such as DeepSeek, that maintain more stable performance across languages. Our analysis reveals an inverse relationship between tokenization efficiency and LLM accuracy where models tend to perform worse when inputs are excessively tokenized, whereas more efficient & concise tokenization results in improved performance. These findings highlight critical areas where current models fall short and underscore the need for improved dataset quality and evaluation methodologies tailored to multilingual contexts. This work will catalyze further research on NLP for underrepresented languages, helping to democratize access to advanced language technologies worldwide. The code used in this research is publicly available at GitHub . The translated Bengali datasets can be accessed on Hugging Face . 1 Introduction Large Language Models (LLMs) have transformed text generation enabling applications in machine translation, text summarization and conversational agents. These models such as GPT-2 and GPT-3 leverage vast amounts of data and deep neural architectures to generate human-like text with fluency ( Witteveen and Andrews 2019 ) . Controlled text generation approaches have also been explored to refine outputs and guide language models toward desirable properties ( Yu et al. 2021 ) . Although these research developments have been substantial, text generation in under-resourced languages like Bengali remains a challenge. Recent efforts have sought to extend LLM capabilities to Bengali, a language spoken by over 230 million people. While general-purpose LLMs perform well in high-resource languages like English and Chinese, Bengali NLP faces limitations due to its linguistic complexity and scarcity of large-scale datasets ( Kabir et al. 2023 ) . To address this, dedicated Bengali LLMs such as BanglaBERT ( Bhattacharjee et al. 2021 ) , BanglaGPT ( Salim et al. 2023 ) have been developed. More recent Bengali-focused models like TituLLM ( Nahin et al. 2025 ) and TigerLLM ( Raihan and Zampieri 2025 ) have also emerged, demonstrating promising results in various Bengali NLP tasks. These models aim to enhance performance in Bengali NLP tasks such as text classification, sentiment analysis and machine translation. However, the development of robust Bengali LLMs is still faced by different challenges. First, the lack of large-scale, high-quality Bengali text corpora limits pretraining and fine-tuning efforts ( Shahriar and Barbosa 2024 ) . While resources like the Sangraha corpus ( Khan et al. 2024 ) developed by AI4Bharat offer numerous data across 22 Indian languages including Bengali, the quality and quantity of Bengali tokens remain limited compared to high-resource languages like English. The Sangraha corpus consists of about 251 billion tokens across all languages, but Bengali’s allocation is significantly smaller at about 30 billion tokens. In contrast, English has access to around 2 trillion tokens in large-scale multilingual corpora such as the Common Corpus ( Langlais et al. 2025 ) . This huge difference in token availability poses a major challenge in achieving comparable model performance in Bengali NLP. Second, the Bengali language’s rich morphology and complex writing system introduce significant tokenization challenges. Unlike English, which uses the Latin script with largely independent characters, Bengali employs an alphasyllabary script where base characters are frequently modified by diacritics and conjunct forms that alter pronunciation and meaning ( Alam et al. 2021 ) . These modifications can occur on either side of a base character, forming intricate multi-character grapheme clusters that do not align well with standard tokenization schemes used in LLMs. As a result, traditional subword tokenization methods such as Byte Pair Encoding (BPE) or WordPiece struggle to segment Bengali text effectively, leading to highly fragmented or inconsistent tokens ( Shahriar and Barbosa 2024 ) . This increased token complexity means that models require more training data to learn meaningful inter-token relationships in Bengali than in English. Failure to capture these linguistic nuances not only increases computational overhead but also degrades model performance on downstream tasks. Third, Bengali NLP research suffers from the absence of standardized evaluation datasets, making it difficult to benchmark model performance effectively ( Kabir et al. 2023 ) . This lack of evaluation datasets motivates the need for well-defined benchmark datasets for Bengali LLMs. Without standardized datasets, it is hard to compare models or track improvements in NLP research. While some efforts have been made to curate evaluation datasets ( Shafayat et al. 2024 ) progress is still slow due to the extensive annotation and validation required. Efforts to develop LLMs for underrepresented languages have explored various methodologies. The Khayyam Challenge ( Ghahroodi et al. 2024 ) curated a large-scale Persian dataset using original non-translated content ensuring language-specific nuances are preserved. Similarly, Cohere’s Aya model ( Üstün et al. 2024 ) employed instruction tuning across multiple low-resource languages to enhance linguistic adaptability. AI4Bharat’s Sangraha dataset tackled data scarcity by aggregating and refining multilingual corpora . In contrast, Turkish LLM research ( Acikgoz et al. 2024 ) experimented with two approaches: adapting English-trained models via transfer learning and pretraining from scratch. While these efforts have proven effective their applicability to Bengali remains uncertain due to unique linguistic characteristics and uniqueness in Bengali. Although substantial progress has been made in developing NLP resources for Bengali, there remain opportunities to accelerate advancement further. Typically, when creating initial benchmarks for lower-resourced languages, researchers bootstrap by translating existing English datasets into the target language, as demonstrated in prior works for Persian and Turkish. However, this initial step has not yet been widely adopted for Bengali, largely due to practical constraints, including the substantial manual validation effort required to correct machine translation errors, associated time investments, and overall costs. Because current machine translation systems often introduce inaccuracies and lose linguistic nuance, manual intervention becomes necessary to refine and validate the translated data. In this study, we directly address these challenges by systematically translating major English benchmark datasets into Bengali and did a performance analysis on them. Motivated by these challenges, this research aims to bridge the existing gaps in Bengali NLP by constructing high-quality evaluation datasets. To address these limitations, this work contributes in a few key areas. • We publicly release a comprehensive suite of high-quality Bengali benchmark datasets, along with the accompanying translation pipeline and codebase to facilitate reproducible research and future advancements in Bengali NLP evaluation. • We describe the methodology used to translate and curate high-quality datasets. • We conduct inference experiments and analyze results to assess model effectiveness of open source multilingual models. • We analyze tokenization behavior across Bengali and English benchmarks, revealing that Bengali inputs produce significantly larger token counts per instance and per word with dataset remaining consistent across

Read the complete text