LLM-as-a-Judge: Reassessing the Performance of LLMs in Extractive QA
Xanh Ho, Jiahao Huang, Florian Boudin, Akiko Aizawa · 2025
Read LLM-as-a-Judge: Reassessing the Performance of LLMs in Extractive QA by Xanh Ho, Jiahao Huang, Florian Boudin, Akiko Aizawa (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-11972.txt.
Description
Xanh Ho and co-authors present a 2025 study on judging answers in extractive question answering. In this task a model pulls an answer span straight from a passage. The usual scores, exact match and F1, often miss whether the answer is truly right, so labs have tried using big language models as judges.
The team tested that idea across four data sets and many prompt wordings. Several model families served as both answerers and judges. Model judgments matched human ratings far better than the old scores, reaching a correlation of 0.85 with open models. Judges handled number answers well but stumbled on job titles. No self-preference bias appeared, and plain zero-shot judging often worked best.
Catalog record
| CREATOR | Xanh Ho, Jiahao Huang, Florian Boudin, Akiko Aizawa |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-11972