Mind the Gap… or Not? How Translation Errors and Evaluation Details Skew Multilingual Results

Jan-Thorsten Peter David Vilar Tobias Domhan · 2025

Read Mind the Gap… or Not? How Translation Errors and Evaluation Details Skew Multilingual Results by Jan-Thorsten Peter David Vilar Tobias Domhan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-05162.txt.

Introduction

Jan-Thorsten Peter and co-authors warn that translation slips and scoring details can skew tests of language models across languages. They take math as a test domain and grade models in tongues from German and Chinese to Swahili and Telugu.

A clear, steady score gap showed up across languages. To their surprise, the gap hit both rich-resource and low-resource tongues. Then the twist: parts of the gap trace to flawed translations of test items and to scoring choices. Their point is stark: cross-language league tables need care before they guide research or product calls.

Catalog record

CREATORJan-Thorsten Peter David Vilar Tobias Domhan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-05162

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Mind the Gap… or Not? How Translation Errors and Evaluation Details Skew Multilingual Results Jan-Thorsten Peter David Vilar Tobias Domhan Abstract Most current large language models (LLMs) support a wide variety of languages in addition to English, including high-resource languages (e.g. German, Chinese, French), as well as low-resource ones (e.g. Swahili, Telugu). In addition they have also shown impressive capabilities in different domains, like coding, science and math. In this short paper, taking math as an example domain, we study the performance of different LLMs across languages. Experimental results show that there exists a non-negligible and consistent gap in the performance of the models across languages. Interestingly, and somewhat against expectations, the gap exists for both high- and low-resource languages. We hope that these results influence further research into cross-lingual capability generalization for next generation LLMs. If it weren’t for the fact that they are false! By analyzing one of the standard multilingual math benchmarks (MGSM), we determine that several translation errors are present in the data. Furthermore, the lack of standardized answer extraction from LLM outputs further influences the final results. We propose a method for automatic quality assurance to address the first issue at scale, and give recommendations to address the second one. Combining these two approaches we show that the aforementioned language gap mostly disappears, leading to completely different conclusions from our research. We additionally release the corrected dataset to the community 1 1 1 https://github.com/google-research-datasets/MGSM-Rev2 . 1 Introduction In recent years, large language models’ capabilities have expanded in two primary directions: broader language coverage and enhanced performance on complex tasks. On the language dimension, it is now usual for LLMs to support not only high-resource languages languages (e.g. Chinese, European languages, etc.), but also low-resource ones (e.g. African languages, Indian languages, etc.). This is a very important and welcome progress direction in order to improve the inclusivity of AI applications and research. In parallel, model capabilities have been steadily improving, with impressive advancements in different areas, e.g. coding, mathematics and science. For these tasks, the vast majority of data is available in English, but for other languages there is a clear imbalance. For high-resource languages there is still a sizeable amount of data in other domains, although it is only a fraction of the data available in English. For low-resource languages, the situation is more critical, with very limited data available in specialized domains. Ideally, LLMs are capable of decoupling these abilities from the language it has been presented to them in the training data. For example the math reasoning learnt in English is equally applicable to, say, Swahili. The difference lays in the verbalization of the “thought process” or the formulation of the answers. That means that language models should be able to perform capability transfer , so that the performance of the model is the same across languages. Of course this is the ideal case, in practice there are no guarantees that this is the case. In order to measure the cross-lingual capabilities of a model, a standard practice is to present the models with the same prompts translated in different languages. One example of this approach is the MGSM corpus ( 16 ) . Starting with the English GSM8K corpus 5 , the authors selected 250 grade-school math problems and translated them into ten typological diverse languages. This offers an ideal testbed for evaluating the cross-lingual math capabilities of the model, as exactly the same problem formulation can be asked to be solved by the model in different languages. When experimenting on this test bench with a variety of LLMs, we observed significant gaps in the performance for different languages when compared to English. While this did not surprise us much for low-resource languages, we also saw an important drop for high-resources languages, such as French. Furthermore, that gap seemed to be consistent across models, which made us somewhat suspicious and we turned our attention to the test data itself. We found out that several translations errors can be found in the released version of the data, and these have a big influence on the conclusions of our experiment. Furthermore, to the best of our knowledge, there is no standardized way to extract the actual numerical value of the answer from the (textual) reply of the models. Special attention has to be paid to regional conventions, e.g. some countries separate thousands using a dot or a space instead of the English comma (e.g. 10,000) or some languages use other numerical systems instead of Arabic digits (e.g. Bengali numerals). Not considering these differences can yield extremely misguided results. In this short paper we first present our “initial analysis” using the original version of the data and a simple version of result extraction taken from the widely used Kaggle page. Section 4 presents the experiments and conclusions that a research group would draw from them, with complete good faith. In Section 5 we analyze the test data in more detail and identify the problems present in them. Manually checking the dataset will probably guarantee the best quality overall, however it is a costly and slow process. We present a semi-automatic method for identifying problematic questions that can scale to bigger corpora. In addition we pay special attention as to how the numerical values should be extracted from the LLM replies, following the specific formating of each language. We repeat our previous analysis with the new data in Section 6 . Comparing the conclusions with those of Section 4 , we see that they are actually opposites. With this paper we aim to raise awareness about the importance of test data quality and nuances when considering multilingual evaluations. Additionally, as part of this publication we release our improved version of MGSM to the wider research community. 2 Related Work 16 introduced the Multilingual Grade School Math (MGSM) dataset by translating 250 English grade-school math problems to 10 languages. The data was sourced from GSM8K 5 and translated using professional translators. The authors find GPT-3 ( 2 ) and PaLM-540B 4 to perform better with English Chain-of-Thought (CoT) prompting, rather than native language CoT prompting. In addition, despite cross-lingual generalization there still is a gap between English and other languages, which is larger for low-resource than it is for high-resource languages. 20 later extended MGSM by four additional languages. MGSM is often used to evaluate the multilingual, mathematical reasoning abilities of LLMs. For example Qwen 3 ( 18 ) , Gemini ( 8 ) , DeepSeek-V3 ( 11 ) or Gemma 3 ( 9 ) report results on MGSM. It is important to note that scores have improved by a large margin since MGSM was introduced. The PaLM-540B model had an average accuracy of 55%, whereas recent models achieve average accuracies above 90% 2 2 2 https://www.kaggle.com/benchmarks/open-benchmarks/mgsm . With such high accuracy, the small remaining unsolved problems amplify the significance of any underlying issues with the test data. Our findings on the critical impact of non-standardized evaluation echo lessons learned in other NLP domains, particularly machine translation (MT). The MT community faced similar issues where differing preprocessing, tokenization, and scoring scripts led to inconsistent and incomparable results, solved by introducing standarized evaluation tools, like e.g. Sacrebleu (

Read the complete text