GlotEval: A Test Suite for Massively Multilingual Evaluation of Large Language Models

Hengyu Luo, Zihao Li, Joseph Attieh, Sawal Devkota, Ona de Gibert, Xu Huang, Shaoxiong Ji, Peiqin Lin, Bhavani Sai Praneeth Varma Mantina, Ananda Sreenidhi, Raúl Vázquez, Mengjie Wang, Samea Yusofi, Fei Yuan, Jörg Tiedemann, Correspondence: [email protected] & [email protected] Equal contribu · 2025

Read GlotEval: A Test Suite for Massively Multilingual Evaluation of Large Language Models by Hengyu Luo, Zihao Li, Joseph Attieh, Sawal Devkota, Ona de Gibert, Xu Huang, Shaoxiong Ji, Peiqin Lin, Bhavani Sai Praneeth Varma Mantina, Ananda Sreenidhi, Raúl Vázquez, Mengjie Wang, Samea Yusofi, Fei Yuan, Jörg Tiedemann, Correspondence: [email protected] & [email protected] Equal contribu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-04155.txt.

Description

Luo and colleagues wrote in twenty twenty-five about GlotEval, a test suite for language models in many tongues. Most test tools focus on English and a few large languages. That leaves model skill in smaller languages unclear. GlotEval is a light frame built for wide, even testing. It supports seven task types, from translation to reading comprehension. Coverage ranges from dozens to hundreds of languages per task.

The suite uses prompt templates shaped for each language. Its translation tests avoid an English-centered design. A translation case study shows how the tool works in practice. Teams can pinpoint where a model is strong or weak by language. The authors aim for fairer checks as models spread worldwide.

Catalog record

CREATORHengyu Luo, Zihao Li, Joseph Attieh, Sawal Devkota, Ona de Gibert, Xu Huang, Shaoxiong Ji, Peiqin Lin, Bhavani Sai Praneeth Varma Mantina, Ananda Sreenidhi, Raúl Vázquez, Mengjie Wang, Samea Yusofi, Fei Yuan, Jörg Tiedemann, Correspondence: [email protected] & [email protected] Equal contribu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-04155

Browse the full library · About the Tower of Babel