Are Large Reasoning Models Good Translation Evaluators? Analysis and Performance Boost

Runzhe Zhan Zhihong Huang Xinyi Yang, Lidia S. Chao Min Yang Derek F. Wong · 2025

Read Are Large Reasoning Models Good Translation Evaluators? Analysis and Performance Boost by Runzhe Zhan Zhihong Huang Xinyi Yang, Lidia S. Chao Min Yang Derek F. Wong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-20780.txt.

Description

Runzhe Zhan and five co-authors test whether large reasoning models make good judges of translation quality, and how to improve them when they fall short. Reasoning models think aloud before answering, a habit that helps hard tasks but had not been checked for scoring translations.

Careful tests show the judges need tailored scoring materials, tend to overthink easy cases, and use score scales in ways that inflate grades. Training them on synthetic human like thinking traces cuts thinking length by a huge factor while raising agreement with human scores across model sizes. One mid sized model gains markedly in correlation, showing that brief guided thinking beats long free thinking for this job.

Catalog record

CREATORRunzhe Zhan Zhihong Huang Xinyi Yang, Lidia S. Chao Min Yang Derek F. Wong
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-20780

Browse the full library · About the Tower of Babel