M-Prometheus: A Suite of Open Multilingual LLM Judges

José Pombal, Dongkeun Yoon, Patrick Fernandes, Ian Wu, Seungone Kim, Ricardo Rei, Graham Neubig & André F.T. Martins · 2025

Read M-Prometheus: A Suite of Open Multilingual LLM Judges by José Pombal, Dongkeun Yoon, Patrick Fernandes, Ian Wu, Seungone Kim, Ricardo Rei, Graham Neubig & André F.T. Martins (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-04953.txt.

Description

José Pombal and co-authors release a suite of open language models trained to judge text in many languages. Their 2025 paper addresses a gap in automatic evaluation: most model judges are tuned for English only. That leaves non-English generation with weaker quality checks, which slows its improvement.

Their judges come in several sizes and can score a single answer or compare two. They beat leading open judges on multilingual reward benchmarks spanning many languages. They also led on literary translation scoring for several language pairs. Used during generation, the judges improved outputs in all tested languages. Ablation tests isolate what made the judges work, from backbone choice to training data. Open judges matter because any lab can now audit multilingual quality.

Catalog record

CREATORJosé Pombal, Dongkeun Yoon, Patrick Fernandes, Ian Wu, Seungone Kim, Ricardo Rei, Graham Neubig & André F.T. Martins
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE126 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-04953

Browse the full library · About the Tower of Babel