DECK: A Consistency × Confidence Taxonomy of LLM Hallucinations

Mohit Singh Chauhan · 2026

Read DECK: A Consistency × Confidence Taxonomy of LLM Hallucinations by Mohit Singh Chauhan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-02289.txt.

Introduction

Chauhan sorts LLM hallucinations by whether a scorer could have caught them. Existing taxonomies classify errors by what is wrong. Memorised misconceptions, reasoning failures, and fabrications are typical classes. Those classes cannot say which uncertainty scorer would flag the error. The DECK taxonomy classifies by detectability signature instead.

Two axes split errors into four regimes. Inter-sample consistency is one axis of the split. Token level confidence is the second axis. The regimes are named Drift, Entrenched, Confabulation, and Knotted. A falsifiable blind spot map comes with the taxonomy. Black box consistency scorers see signal in two regimes. White box probability scorers see signal in two others. Only an independent LLM judge can detect the Entrenched regime. Three models and four QA data sets test the map.

Catalog record

CREATORMohit Singh Chauhan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.02289

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DECK: A Consistency × Confidence Taxonomy of LLM Hallucinations Mohit Singh Chauhan Abstract Existing hallucination taxonomies classify LLM errors by what is wrong with the output — memorised misconceptions, reasoning failures, fluent fabrications. These taxonomies are useful for diagnosis but cannot answer a different question: which uncertainty scorer would have caught this error? We propose a complementary taxonomy that classifies errors by their detectability signature — the signal a scorer family would read. The DECK taxonomy is a 2 × 2 2{\times}2 partition along inter-sample consistency and token-level confidence into four behavioural regimes ( D rift, E ntrenched, C onfabulation, K notted), each mapping to a specific scorer family (or families) that can detect it: black-box consistency scorers have signal in D and C, white-box token-probability scorers have signal in K and C, and only an LLM-as-a-Judge with independent pretraining can detect E. Cell membership is operationalised by a Youden’s J optimal split on each scorer axis. Across three models and four datasets we validate the taxonomy two ways: by analysing scorer-pair disagreement, and by checking that external labels (SelfAware unanswerable, HaluEval adversarial, PopQA entity popularity) land in the predicted DECK cells, with model-scale and content-specific secondary-cell refinements. We further identify a universal blind spot of output-level UQ : on knowledge-gap inputs where the generator emits confident, repeatable fabrications, every output-level family collapses by construction. A linear probe on Llama-3-8B’s hidden states also collapses to chance, giving preliminary evidence that the failure may persist at the activation level; richer internal-state methods (UQ heads, information-theoretic estimators) remain to be tested. 1 Introduction Existing hallucination taxonomies classify LLM errors by what is wrong with the output ( 10 ; 8 ; 29 ) — memorised misconceptions, reasoning failures, fluent fabrications, faithfulness violations. These taxonomies are useful for diagnosis, but they cannot answer a question that arises whenever an LLM is deployed in a high-stakes setting: which uncertainty scorer would have caught this error? The same content category can produce different detection signatures across models (a "reasoning failure" is detectable by a Judge if the answer is stable and high-probability, but only by an inter-sample consistency check if the answer varies); and the same detection signature can come from very different content categories. Without a taxonomy on the detection axis, choosing a scorer for a new domain stays a guess — even though the literature already offers three scorer families: Black-box (BB) consistency checks ( 22 ; 16 ) , White-box (WB) token-level log-probabilities ( 13 ; 20 ) , and LLM-as-a-Judge (J) factual review ( 34 ) , which have been shown to ensemble well together ( 2 ) . We propose a complementary taxonomy that classifies errors by their detectability signature — the signal a scorer family reads. This shift, from what is wrong to what would flag it , is the paper’s central contribution. (1) The DECK taxonomy (Table 1 ): a 2{\times}2 partition along inter-sample consistency and token-level confidence into four behavioural regimes (D, E, C, K). Each cell maps to a specific scorer family (or families) that can detect it: BB has signal in D and C; WB has signal in K and C; only J can detect E (and only when the judge has independent pretraining). For each scorer axis we pick the threshold that best separates correct from hallucinated responses (Youden’s J), then assign each sample to one of the four cells. The taxonomy makes testable predictions about which scorer families should detect which hallucinations. We verify these predictions in two ways — by analysing scorer-pair disagreement (§ 5.2 ), and by checking that external labels (SelfAware unanswerable, HaluEval adversarial, PopQA entity popularity) land in the predicted DECK cells, with model-scale and content-specific secondary-cell refinements (§ 5.3 ). (2) A universal blind spot of output-level UQ. We identify an empirical regime where every scorer in the three-family output-level paradigm fails simultaneously (§ 5.4 ). On SelfAware all three families collapse to or below chance because the generator emits confident, repeatable fabrications on inputs it cannot answer — eliminating the variance every output-level family needs to be informative. The right engineering response is an abstention envelope that routes such out-of-scope inputs to refusal before scoring. "Universal" here means across the three output-level families studied. As a preliminary robustness check, we further test the simplest internal-state probe ( 27 ) : a linear classifier on Llama-3-8B’s last-layer hidden states also collapses to chance on SelfAware, which is consistent with activation-level persistence of the failure mode. Richer internal-state methods — UQ heads ( 25 ) , information-theoretic estimators ( 31 ) — and multi-layer probes remain open. Table 1: The DECK taxonomy of LLM hallucinations. Each cell is a behavioral regime defined by two observable axes: inter-sample consistency and token-level confidence. The four regimes map to the letters D rift, E ntrenched, C onfabulation, and K notted. The final column gives the scorer family (or families) sensitive to each regime. Low Consistency High Consistency Detectable By High Token Confidence D — Drift Different confident wrong answer each sample. Multiple plausible-but-wrong answers are drawn confidently from the output distribution; each draw is internally coherent, but the answer drifts. E — Entrenched Same confident wrong answer every sample. Model has locked onto a memorized misconception or shared-pretraining error and reproduces it without variance. Drift: Black-box and Judge. Entrenched: Judge (independent provider required); blind to both BB and WB. Low Token Confidence C — Confabulation Different low-probability wrong answer each sample. Classic uncertainty: the model genuinely does not know, and different samples produce different wrong answers with low confidence. K — Knotted Same low-probability wrong answer every sample. Model is consistently unsure: it settles on the same hedged answer each time but assigns low token probability. Confabulation: All three families. Knotted: White-box and Judge. Note. Regimes are defined by observable output behavior, not by an internal mechanism claim. Cell membership depends on the scorer thresholds used to discretize the two axes; described in § 3 . 2 Background and Related Work 2.1 Three Families of UQ Scorers Output-level UQ methods for LLMs fall into three families distinguished by the access they require ( 26 ; 15 ) . White-box methods read confidence from token log-probabilities; representative scorers include P(True) ( 13 ) , sequence-probability measures ( 20 ) , and calibration-aware variants over token distributions ( 11 ; 4 ) . They require logit access and are cheap. A related thread elicits verbalized confidence by asking the model to state its own uncertainty ( 28 ; 30 ) . Black-box methods read confidence from inter-sample agreement: SelfCheckGPT ( 22 ) uses NLI- and embedding-based consistency, while semantic entropy ( 16 ; 6 ) clusters responses into meaning classes and computes entropy over the resulting distribution. Subsequent work explores alternative aggregations of the consistency signal: pairwise-NLI aggregation across sampled responses ( 18 ) and semantic-density estimation in embedding space ( 24 ) . These methods are model-agnostic but cost multiple generations per query. LLM-as-a-Judge methods delegate evaluation to a separate model ( 34 ; 19 ) ; they detect failures invisible to either probabilistic signal but inherit known judge biases ( 3 ) and risk shared-pretraining errors when generator and judge are related

Read the complete text