LLM-as-a-Judge: Reassessing the Performance of LLMs in Extractive QA

Xanh Ho, Jiahao Huang, Florian Boudin, Akiko Aizawa · 2025

Read LLM-as-a-Judge: Reassessing the Performance of LLMs in Extractive QA by Xanh Ho, Jiahao Huang, Florian Boudin, Akiko Aizawa (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-11972.txt.

Description

Xanh Ho and co-authors present a 2025 study on judging answers in extractive question answering. In this task a model pulls an answer span straight from a passage. The usual scores, exact match and F1, often miss whether the answer is truly right, so labs have tried using big language models as judges.

The team tested that idea across four data sets and many prompt wordings. Several model families served as both answerers and judges. Model judgments matched human ratings far better than the old scores, reaching a correlation of 0.85 with open models. Judges handled number answers well but stumbled on job titles. No self-preference bias appeared, and plain zero-shot judging often worked best.

Catalog record

CREATORXanh Ho, Jiahao Huang, Florian Boudin, Akiko Aizawa
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-11972

Browse the full library · About the Tower of Babel