Perceived Confidence Scoring for Data Annotation with Zero-Shot LLMs
Sina Salimian, Gias Uddin, Most Husne Jahan, Shaina Raza · 2025
Read Perceived Confidence Scoring for Data Annotation with Zero-Shot LLMs by Sina Salimian, Gias Uddin, Most Husne Jahan, Shaina Raza (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-07186.txt.
Description
Salimian and co-authors grade a language model's confidence by arguing with itself. Zero-shot models now label text for sentiment and bias, but their self-assurance is unpriced. The method rewrites each input in ways that keep the meaning yet shift the wording, then checks whether the model's label survives the rewrites. Agreement across versions becomes a perceived confidence score.
Used alone, the score lifted single-model labeling accuracy by 9.3 percent across sorting tasks. Bolted onto majority voting among several models, it added 5.8 percent more. The trick needs no model internals, suiting closed commercial systems used as annotators.
Catalog record
| CREATOR | Sina Salimian, Gias Uddin, Most Husne Jahan, Shaina Raza |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-07186