Evaluating LLMs Without Oracle Feedback: Agentic Annotation Evaluation Through Unsupervised Consistency Signals
Cheng Chen, Haiyan Yin, Ivor W. Tsang · 2025
Read Evaluating LLMs Without Oracle Feedback: Agentic Annotation Evaluation Through Unsupervised Consistency Signals by Cheng Chen, Haiyan Yin, Ivor W. Tsang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-08809.txt.
Description
Cheng Chen and 2 co-authors evaluate language model annotations without oracle feedback in a 2025 paper. Models can label data cheaply, but judging label quality is hard when no trusted answers exist. Dynamic, unsupervised settings make standard methods fail.
A student model works with a noisy teacher model to assess and refine labels. Voting based on user preference checks how consistent teacher outputs are. A new metric, the CAI Ratio, measures that consistency. Across ten datasets and four models, the ratio tracks true accuracy closely. It can also guide model choice in real settings where trusted feedback is scarce.
Catalog record
| CREATOR | Cheng Chen, Haiyan Yin, Ivor W. Tsang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-08809