Are Large Reasoning Models Good Translation Evaluators? Analysis and Performance Boost
Runzhe Zhan Zhihong Huang Xinyi Yang, Lidia S. Chao Min Yang Derek F. Wong · 2025
Read Are Large Reasoning Models Good Translation Evaluators? Analysis and Performance Boost by Runzhe Zhan Zhihong Huang Xinyi Yang, Lidia S. Chao Min Yang Derek F. Wong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-20780.txt.
Description
Runzhe Zhan and five co-authors test whether large reasoning models make good judges of translation quality, and how to improve them when they fall short. Reasoning models think aloud before answering, a habit that helps hard tasks but had not been checked for scoring translations.
Careful tests show the judges need tailored scoring materials, tend to overthink easy cases, and use score scales in ways that inflate grades. Training them on synthetic human like thinking traces cuts thinking length by a huge factor while raising agreement with human scores across model sizes. One mid sized model gains markedly in correlation, showing that brief guided thinking beats long free thinking for this job.
Catalog record
| CREATOR | Runzhe Zhan Zhihong Huang Xinyi Yang, Lidia S. Chao Min Yang Derek F. Wong |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-20780