Found in Translation: Measuring Multilingual LLM Consistency as Simple as Translate then Evaluate

Ashim Gupta Maitrey Mehta Zhichao Xu Vivek Srikumar Kahlert School of Computing University of Utah [email protected] · 2025

Read Found in Translation: Measuring Multilingual LLM Consistency as Simple as Translate then Evaluate by Ashim Gupta Maitrey Mehta Zhichao Xu Vivek Srikumar Kahlert School of Computing University of Utah [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21999.txt.

Description

Ashim Gupta and co-authors test whether language models answer alike across languages in a 2025 paper. The models shine in English, but scored test sets in many languages are costly to build. Open ended answers add a second snag, since many replies can be right. The team offers a cheaper route: translate a model's answers, then judge how well they match across tongues.

Their frame checks steadiness along two lines, the facts an answer carries and the empathy it shows. Results across thirty languages showed wide gaps in popular models. Some language families and scripts fared far worse than others. The authors urge builders to test steadiness across languages as a routine step, and they offer their frame as a shared tool for that work.

Catalog record

CREATORAshim Gupta Maitrey Mehta Zhichao Xu Vivek Srikumar Kahlert School of Computing University of Utah [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-21999

Browse the full library · About the Tower of Babel