The Consistency Dilemma in LLMs: Generator-Evaluator Agreement and Vulnerability to Mistakes
Marina Mancoridis, Zoë Hitzig · 2026
Read The Consistency Dilemma in LLMs: Generator-Evaluator Agreement and Vulnerability to Mistakes by Marina Mancoridis, Zoë Hitzig (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30653.txt.
Description
Marina Mancoridis and Zoë Hitzig measure whether language models judge their own work by the same standards they used to produce it. Automated pipelines increasingly let a model generate an output and then evaluate it, with no outside check. That only works if the model's grasp of a concept stays stable across both roles. The pair define generator-evaluator self-consistency to test exactly this assumption.
They applied the measure to ten frontier models across nearly five hundred concepts. Self-consistency varied substantially between models. In a clinical test built on thirty-five physician-validated mistakes, a disturbing pattern emerged: models with higher self-consistency were more vulnerable to the mistakes. Consistency, in other words, does not imply correctness, since a model can apply a wrong concept very steadily. The authors call this the consistency dilemma, and warn that self-checking pipelines inherit the risk.
Catalog record
| CREATOR | Marina Mancoridis, Zoë Hitzig |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 121 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.30653