When LLMs Struggle: Reference-less Translation Evaluation for Low-resource Languages
Archchana Sindhujan, Diptesh Kanojia, Constantin Orăsan , Shenbin Qian · 2025
Read When LLMs Struggle: Reference-less Translation Evaluation for Low-resource Languages by Archchana Sindhujan, Diptesh Kanojia, Constantin Orăsan , Shenbin Qian (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-04473.txt.
Description
Archchana Sindhujan, Diptesh Kanojia, Constantin Orasan, and Shenbin Qian test how well large language models grade translations in a 2025 paper. The setting is hard: language pairs with little data, and no reference translation to check against.
The models had to score each translated segment from 0 to 100. The team tried plain prompts, a few examples, and training with prompts built from human scoring guides. Even so, smaller models trained for this one task did better. The authors traced many errors to how the models split words, and to names and borrowed scripts. They released their data and models so others can build on the work.
Catalog record
| CREATOR | Archchana Sindhujan, Diptesh Kanojia, Constantin Orăsan , Shenbin Qian |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-04473