Evaluating LLMs Without Oracle Feedback: Agentic Annotation Evaluation Through Unsupervised Consistency Signals

Cheng Chen, Haiyan Yin, Ivor W. Tsang · 2025

Read Evaluating LLMs Without Oracle Feedback: Agentic Annotation Evaluation Through Unsupervised Consistency Signals by Cheng Chen, Haiyan Yin, Ivor W. Tsang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-08809.txt.

Description

Cheng Chen and 2 co-authors evaluate language model annotations without oracle feedback in a 2025 paper. Models can label data cheaply, but judging label quality is hard when no trusted answers exist. Dynamic, unsupervised settings make standard methods fail.

A student model works with a noisy teacher model to assess and refine labels. Voting based on user preference checks how consistent teacher outputs are. A new metric, the CAI Ratio, measures that consistency. Across ten datasets and four models, the ratio tracks true accuracy closely. It can also guide model choice in real settings where trusted feedback is scarce.

Catalog record

CREATORCheng Chen, Haiyan Yin, Ivor W. Tsang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-08809

Browse the full library · About the Tower of Babel