The Consistency Dilemma in LLMs: Generator-Evaluator Agreement and Vulnerability to Mistakes

Marina Mancoridis, Zoë Hitzig · 2026

Read The Consistency Dilemma in LLMs: Generator-Evaluator Agreement and Vulnerability to Mistakes by Marina Mancoridis, Zoë Hitzig (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30653.txt.

Description

Marina Mancoridis and Zoë Hitzig measure whether language models judge their own work by the same standards they used to produce it. Automated pipelines increasingly let a model generate an output and then evaluate it, with no outside check. That only works if the model's grasp of a concept stays stable across both roles. The pair define generator-evaluator self-consistency to test exactly this assumption.

They applied the measure to ten frontier models across nearly five hundred concepts. Self-consistency varied substantially between models. In a clinical test built on thirty-five physician-validated mistakes, a disturbing pattern emerged: models with higher self-consistency were more vulnerable to the mistakes. Consistency, in other words, does not imply correctness, since a model can apply a wrong concept very steadily. The authors call this the consistency dilemma, and warn that self-checking pipelines inherit the risk.

Catalog record

CREATORMarina Mancoridis, Zoë Hitzig
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE121 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.30653

Browse the full library · About the Tower of Babel