The AI Language Proficiency Monitor – Tracking the Progress of LLMs on Multilingual Benchmarks
David Pomerenke, Jonas Nothnagel, Simon Ostermann · 2025
Read The AI Language Proficiency Monitor – Tracking the Progress of LLMs on Multilingual Benchmarks by David Pomerenke, Jonas Nothnagel, Simon Ostermann (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-08538.txt.
Introduction
A language model that shines in English may stumble in Swahili, and David Pomerenke, Jonas Nothnagel, and Simon Ostermann built a public monitor to track that gap. Their benchmark gathers translation, question answering, mathematics, reasoning, and truthfulness tasks across up to two hundred languages, leaning on established multilingual datasets, with a deliberate focus on low-resource tongues.
An open leaderboard and dashboard update automatically, ranking models and mapping their proficiency across the globe and over time. The point is not another horse race but diagnosis: where models fail whole populations, developers and policymakers can see it plainly. The authors offer the monitor as infrastructure for equitable access to language technology.
Catalog record
| CREATOR | David Pomerenke, Jonas Nothnagel, Simon Ostermann |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-08538
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The AI Language Proficiency Monitor – Tracking the Progress of LLMs on Multilingual Benchmarks David Pomerenke, Jonas Nothnagel, Simon Ostermann Abstract To ensure equitable access to the benefits of large language models (LLMs), it is essential to evaluate their capabilities across the world’s languages. We introduce the AI Language Proficiency Monitor , a comprehensive multilingual benchmark that systematically assesses LLM performance across up to 200 languages, with a particular focus on low-resource languages. Our benchmark aggregates diverse tasks including translation, question answering, math, and reasoning, using datasets such as FLORES+, MMLU, GSM8K, TruthfulQA, and ARC. We provide an open-source, auto-updating leaderboard and dashboard that supports researchers, developers, and policymakers in identifying strengths and gaps in model performance. In addition to ranking models, the platform offers descriptive insights such as a global proficiency map and trends over time. By complementing and extending prior multilingual benchmarks, our work aims to foster transparency, inclusivity, and progress in multilingual AI. The system is available at https://huggingface.co/spaces/fair-forward/evals-for-every-language . 1 Introduction To make the benefits of large language models (LLMs) available to everyone, LLMs must speak and understand everyone’s language. While a growing range of multilingual LLMs exist, mainstream LLM development still predominantly focuses on English. In contrast to this, a majority – close to 80% – of the world’s population does not speak English Eberhard et al. 2025 . In response to this disparity, a significant number of multilingual resources for evaluation and training have been developed for many languages, both for pretraining Mayer and Cysouw 2014 ; Conneau et al. 2020a ; Abadji et al. 2022 ; Weber et al. 2024 and finetuning LLMs Fu et al. 2022 ; Chiang et al. 2023 ; Asai et al. 2024 . With this work, we present the AI Language Proficiency Monitor , an attempt to compile a number of such multilingual evaluation datasets into a comprehensive benchmark for evaluating the performance of current AI models on every language that has available benchmarking data. We specifically focus on low-resource languages. Figure 1: The map view of the language proficiency monitor. The benchmark is accompanied by an open-source dashboard and is designed to support three primary use cases: • Researchers and model developers can benchmark their systems using the “AI Language Proficiency Score,” a multi-task multilingual performance metric; • Practitioners can identify the most suitable models for specific languages and tasks; • Policymakers and donor agencies can identify languages where capacity-building is most urgent. With this design, we aim to provide impact beyond academia by supporting both private- and public-sector decision-making, with a particular emphasis on underserved communities. For benchmarking we rely on FLORES+ ( Goyal et al. 2022 , translation, classification) and multilingual versions of MMLU ( Hendrycks et al. 2020 , multi-domain question answering), ARC ( Clark et al. 2018 , advanced question answering), GSM8K ( Cobbe et al. 2021b , math problems), and TruthfulQA ( Lin et al. 2022 , factuality). Depending on the task, 100-200 languages are evaluated; these cover 80-95% of the world’s speakers, and include many low-resource languages. The monitor evaluates state-of-the-art models from commercial AI providers, as well as relevant open-weights models. The presented pipeline auto-evaluates daily, always including the latest relevant models. In addition to an interactive scoreboard, we present descriptive analyses: A global map of country-level scores of AI language proficiency; progress over time for high-resource vs. low-resource languages; and the impact of a language’s number of speakers, GDP, and data availability on AI language proficiency. Our goal is to also address non-academic users, who often struggle to use and interpret academic benchmarks Ethayarajh and Jurafsky 2020 . In comparison to existing research and systems, we make the following contributions: While comprehensive multilingual benchmarks such as FLORES+ have already been used for evaluating models at singular points in time, our system continually applies them to the evolving landscape of LLMs. This is similar to the OpenLLM leaderboard 1 1 1 https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard and the underlying EleutherAI LM Evaluation Harness Gao et al. 2024 , but with a focus on systematic multilingual evaluation. Regional leaderboards for African and European languages pursue the same focus (see section 2 ) and also leverage machine translation to achieve comprehensiveness across the languages they cover; our system takes inspiration from them, while scaling to a global level, including more and up-to-date models, and providing a user-friendly interface with language-specific leaderboards and visuals. While our analysis already covers the most popular models and the largest languages, it is far from comprehensive. We thus hope that our initiative sparks the interest of regional AI developers to submit their own finetuned AI models, and to integrate datasets for further tasks and languages. 2 Background and Related Work 2.1 Multilingual Evaluation Benchmarks A number of multilingual LLM benchmark exists. Notably, all of these benchmarks either come with a very specific regional focus, or a low coverage of low-resource languages. A first example is the European LLM Leaderboard 2 2 2 https://huggingface.co/spaces/Eurolingua/european-llm-leaderboard Thellmann et al. 2024b . It is based on the OpenGPT-X LM-evaluation-harness 3 3 3 https://github.com/OpenGPTX/lm-evaluation-harness and include NLU and machine translation evaluations in 21 European languages (Irish, Maltese, Croatian are missing); on publicly available data sets such as MMLU Hendrycks et al. 2020 , HellaSwag Zellers et al. 2019 , and GSM8K Cobbe et al. 2021a . Notably, only European languages are covered. Another example is the Multilingual LLM Evaluation 4 4 4 https://huggingface.co/collections/CohereLabs/multilingual-llm-evaluation-674ebf8e49e5af2efe021c8d benchmark collection by Cohere.AI. The benchmark includes a collection of datasets like Global-MMLU or aya-redteaming . While being an exhaustive collection of multilingual evaluation datasets, the benchmark doesn’t provide a unified evaluation interface or visualizations. MEGA (Multilingual Evaluation of Generative AI) Ahuja et al. 2023 is a collection of standard NLP benchmarks, covering 16 NLP datasets across 70 typologically diverse languages. MEGAVERSE Ahuja et al. 2024 extends this to 22 datasets covering 83 languages, as well as some multimodal benchmarks. Both benchmarks are most similar to ours in terms of coverage, but still lack behind a more holistic evaluation of languages (with our benchmark comprising more than twice the amount of languages), and they come without an explicit leader board or visualizations, restricting their usefulness for non-technical practitioners. A prominent example for a more local and regional benchmark is AfroBench Ojo et al. 2025 . 5 5 5 https://mcgill-nlp.github.io/AfroBench/leaderboard.html the benchmark encompasses data from 64 African languages for nine natural language understanding datasets, six text generation datasets, six knowledge and question answering tasks, and one mathematical reasoning task. Naturally, it focuses on African languages only. 2.2 Multilingual and non-English Language Models Multilingual large language models (MLLMs) are able to process and generate text across multiple languages, enabling cross-lingual applications such as translation, retrieval, and reasoning. Early examples of multilingual language models include mBERT Devlin et al. 2019 and XLM-R Conneau et al. 2020a , encoder-based models that are small by modern standards. There also exist some embedding-based works, especially for low-resource languages Gurgurov et al. 2025 . For LLMs, the most famous examples include the LLama 3 model family Grattafiori et al. 2024 and Gemma 3 Team et al. 2025 , two types of open-weights models. Qin et al. 2025 introduce a comprehensive taxonomy of MLLMs based different techniques to bridge language gaps.