Translation as a Scalable Proxy for Multilingual Evaluation
Sheriff Issaka, Erick Rosas Gonzalez, Lieqi Liu, Evans Kofi Agyei, Lucas Bandarkar, Nanyun Peng, David Ifeoluwa Adelani, Francisco Guzmán, Saadia Gabriel · 2026
Read Translation as a Scalable Proxy for Multilingual Evaluation by Sheriff Issaka, Erick Rosas Gonzalez, Lieqi Liu, Evans Kofi Agyei, Lucas Bandarkar, Nanyun Peng, David Ifeoluwa Adelani, Francisco Guzmán, Saadia Gabriel (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-11778.txt.
Introduction
Sheriff Issaka, Erick Rosas Gonzalez, Lieqi Liu, and a broad team tackle a gap in tests for language models. Models claim skill in many tongues, yet true test sets made by humans exist for fewer than 30 tongues. That leaves most of the world's 7,000-plus tongues with no firm check.
They ask if translation can stand in as a scaled proxy for such tests. Hand made test sets face cost, few experts, and leaks of test data into training. A proxy that tracks true skill could guide work on low resource tongues. The study weighs how far that proxy can be trusted.
Catalog record
| CREATOR | Sheriff Issaka, Erick Rosas Gonzalez, Lieqi Liu, Evans Kofi Agyei, Lucas Bandarkar, Nanyun Peng, David Ifeoluwa Adelani, Francisco Guzmán, Saadia Gabriel |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-11778
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Translation as a Scalable Proxy for Multilingual Evaluation Sheriff Issaka, Erick Rosas Gonzalez, Lieqi Liu, Evans Kofi Agyei, Lucas Bandarkar, Nanyun Peng, David Ifeoluwa Adelani, Francisco Guzmán, Saadia Gabriel Abstract The rapid proliferation of LLMs has created a critical evaluation paradox: while LLMs claim multilingual proficiency, comprehensive non-machine-translated benchmarks exist for fewer than 30 languages, leaving > > 98% of the world’s 7,000 languages in an empirical void. Traditional benchmark construction faces scaling challenges such as cost, scarcity of domain experts, and data contamination. We evaluate the validity of a simpler alternative: can translation quality alone indicate a model’s broader multilingual capabilities ? Through systematic evaluation of 14 models (1B-72B parameters) across 9 diverse benchmarks and 7 translation metrics, we find that translation performance is a good indicator of downstream task success (e.g., Phi-4, median Pearson r r : MetricX = 0.89, xCOMET = 0.91, SSA-COMET = 0.87). These results suggest that the representational abilities supporting faithful translation overlap with those required for multilingual understanding. Translation quality, thus emerges as a strong, inexpensive first-pass proxy of multilingual performance, enabling a translation-first screening with targeted follow-up for specific tasks. 1 1 1 To promote accessibility, we provide translations of this abstract in 10 different languages in Appendix I , generated using Mansa and ChatGPT 5 . 2 2 2 Code & Data: https://translation-as-multilingual-proxy.github.io/ 1 Introduction The rapid advancement of large language models (LLMs) has created a major technological paradox: while these models are deployed globally and claim multilingual capabilities 47 ; 46 ; 67 ; 71 ; 75 , we lack the basic infrastructure to assess their abilities. Of the world’s 7,000 languages, comprehensive non-machine-translated benchmarks exist for fewer than 30 overlapping languages 14 ; 24 ; 3 ; 61 ; 15 ; 36 ; 78 ; 41 , leaving more than 98% of linguistic diversity unmeasured. This evaluation vacuum has profound consequences as entire language communities are excluded from AI development, not because their languages are inherently unsuitable for modeling, but because we simply cannot measure whether models work for them at all 80 ; 74 ; 12 ; 27 . For these low-resourced languages (LRLs), evaluations suffer from a dual failure: they are frequently nonexistent, and when present, systematically overestimate model capabilities 4 ; 14 . As a response, this work explores a simple but overlooked question: Can translation quality serve as a reliable, scalable, and cost-effective proxy for a model’s broader multilingual capabilities? The current paradigm of building task-specific benchmarks for each language has proven economically costly, inherently unscalable, and systematically biased 38 ; 77 ; 3 . A recent analysis of over 2,000 benchmarks revealed that despite investments exceeding tens of millions of dollars between 2021 and 2024, coverage remains overwhelmingly concentrated on English and a handful of high-resource languages 72 . Even well-intentioned efforts to automatically translate existing benchmarks fail systematically, as these machine translations cannot capture cultural and linguistic nuances, resulting in evaluations that correlate poorly with human judgments and perpetuate biases 72 ; 57 ; 54 ; 62 ; 1 . Figure 1: Pearson correlation matrix for the Phi-4 (14B) model, showing correlations between MT quality metrics and multilingual benchmark performance on FLORES-200 . Color intensity indicates the strength of correlation, with darker red denoting stronger positive correlations. We hypothesize that high-quality, human-verified parallel corpora could provide a foundation for evaluating general multilingual capabilities. This claim rests on a central insight: translation is one of the most demanding natural language tasks as it requires bridging non-isomorphic linguistic systems while preserving meaning across semantics, pragmatics, discourse structure, and cultural context 7 ; 42 . To test this hypothesis, we conduct the most comprehensive correlation analysis to date as described in §3. We examine 14 models spanning 5 distinct families (Llama 5 , DeepSeek 17 , Qwen 76 ; 53 , Gemma 66 , Phi 2 ), ranging from 1B to 72B parameters. We evaluate their performance across 9 diverse multilingual benchmarks ( TruthfulQA 34 , MLQA 36 , GlobalMMLU 61 , Belebele 9 , MGSM 60 , AfriMMLU 3 , HellaSwag 34 , AfriXNLI 3 , INCLUDE 56 ) that span the full spectrum of language understanding: knowledge retrieval, logical reasoning, reading comprehension, and natural language inference. These benchmark scores are then systematically correlated with 7 distinct machine translation (MT) quality metrics across 3 translation datasets ( Flores-200 45 , WMT24++ 19 , NTREX 21 ; 20 ; 44 ), enabling us to examine whether translation quality consistently correlates with performance across different task types, model architectures, and language families. Our findings in §4 demonstrate alignment with this hypothesis: translation quality correlates well with downstream task performance across most language-task combinations, yielding a median Pearson of r = 0.89 for MetricX , 0.91 for xCOMET , and 0.87 for SSA-COMET . This suggests that the neural representations required for high-quality translation may be intertwined with those needed for complex reasoning. Translation quality, therefore, offers a low-cost first-pass signal of multilingual performance. While it cannot replace task-specific benchmarks entirely, it enables a practical two-stage evaluation pipeline where translation metrics provide rapid assessment across thousands of languages, followed by targeted benchmark evaluation only where needed. 2 Related Work LLMs have changed natural language processing (NLP) and the space of AI 11 ; 48 ; 8 ; 46 ; 65 . Consequently, the rapid development of multilingual LLMs has created a significant challenge: how can we reliably and efficiently evaluate their capabilities across a diverse linguistic landscape? The academic community has largely pursued the path of building comprehensive, task-specific benchmarks that aim for exhaustive testing. 2.1 Multilingual Benchmark Recent research has adopted two approaches to multilingual LLM evaluation. The first involved translating existing English evaluation suites into other languages using either human translators or MT systems 59 ; 34 ; 9 ; 61 ; 26 . Works such as XNLI extended natural language inference to 15 languages, establishing a paradigm that was adopted by benchmarks for question answering ( MLQA ), commonsense reasoning ( XCOPA ), and mathematical reasoning ( MGSM ) ( 16 ; 36 ; 51 ; 60 ) . To address the limitations and potential biases of this translation-centric approach, a second line focuses on creating more culturally-centric and linguistically nuanced benchmarks curated directly in target languages. For example, studies collected human exam questions from specific regions in the target language with examples in Chinese, Indonesian, and Turkish 32 ; 37 ; 77 . Other efforts expanded multitask evaluation with a focus on low-resourced regions, as seen in Global MMLU and IrokoBench ( 61 ; 3 ) . Other studies emphasize the inclusion of culturally specific content in their evaluation, such as CulturalBench 13 , ArtELingo-28 43 , and CVQA 58 . While these benchmarks provide a thorough assessment of a model’s multilingual capabilities, their primary limitation is the difficulty in creating them. They are computationally and financially expensive, time-consuming to build and run, making frequent or large-scale evaluation prohibitive. Thus, the (left-behind) language communities that will need such evaluations the most are those least likely to have the said resources to create such benchmarks ( 28 ) . 2.2 Machine Translation Metrics The field of MT metrics has evolved significantly from its reliance on