PARALLAX: Separating Genuine Progress from Benchmark Artifacts in Hallucination Detection Khizar Hussain, Murat Kantarcioglu Abstract Large language models (LLMs) hallucinate with confidence: their outputs can be fluent, authoritative, and simply wrong. In medical, legal, and scientific applications this failure causes direct harm, and detecting it from internal model states offers a path to safer deployment. A growing body of work reports that this problem is increasingly tractable, with recent methods achieving high detection performance on widely used benchmarks. We show, however, that much of this apparent progress does not survive scrutiny. Four of the six corpora embed the ground-truth answer directly in the input prompt. A naïve text-similarity baseline we call TxTemb exploits this to achieve near-perfect detection scores without any access to model internals. To measure what genuine detection capability remains once these artifacts are controlled, we conduct a large-scale evaluation spanning twenty-two detection methods, twelve open-source models spanning six architectural families, and six corpora. We further introduce DRIFT , a supervised probe over inter-layer hidden-state transitions, as a point of comparison for live-generation detection. Our findings suggest that the field’s reported progress on hallucination detection is substantially explained by benchmark construction artifacts in widely used corpora, and that the majority of established baselines perform near chance under controlled conditions; the consistent exceptions are SAPLMA and DRIFT, both supervised probes on upper-layer hidden states. 1 Introduction Reliable hallucination detection matters: errors in medical, legal, and scientific applications cause direct harm ( 60 ; 11 ; 47 ) , and a real-time detector requiring no extra inference would enable safer deployment of production LLMs. A growing body of work reports high-AUROC hallucination detectors ( 71 ; 24 ; 23 ) , but the field lacks controlled cross-method comparisons and reported scores come with two serious confounds that have not been adequately addressed. The first is benchmark construction: popular corpora such as HaluEval ( 30 ) , MedHallu ( 39 ) , and TruthfulQA ( 33 ) present the model with both a correct and a hallucinated answer in the same prompt, so the label is correlated with surface text differences rather than model uncertainty. We demonstrate this with TxTemb (Eq. 1 ): a TF-IDF cosine similarity score between the hallucinated and reference answer texts, requiring no access to model internals, which achieves AUROC =0.98 on HaluEval. Any internal-state method achieving similar performance is not detecting hallucination; it is detecting lexical phrasing patterns in the prompt. Throughout this paper, TxTemb scores serve as an artifact control: a high TxTemb AUROC on a given corpus indicates that surface text differences, not model uncertainty, drive detection performance. The second confound is evaluation scope: most papers evaluate one or two benchmarks on a single model, making it impossible to distinguish a genuine mechanistic signal from overfitting to a specific benchmark-model combination. We address both confounds with a large-scale controlled evaluation. Across six corpora (open-domain, medical, factual, legal, RAG, multi-domain QA), we evaluate twenty-two detection methods on twelve instruction-tuned models spanning six architectural families (3.8B–72B parameters). Our six probe-based approaches include DRIFT (Approach E, introduced here), self-supervised perturbation contrast (A), contrastive activation addition probe (B), semantic entropy probe (C, adaptation of ( 18 ) ), activation variance consistency (D), and answer expectation (F). These are evaluated against sixteen established and new baselines ( 8 ; 36 ; 58 ; 3 ; 10 ; 65 ; 14 ; 12 ; 18 ; 21 ; 45 ) , with TxTemb artifact controls, permutation tests, and bootstrap confidence intervals applied to every method-corpus pair. We include DRIFT among the evaluated methods; all probe hyperparameters were fixed before the live-generation test splits (RAGTruth, HaluBench) were accessed. Our findings raise serious concerns. Teacher-forced corpora embed the ground-truth answer directly in the input prompt; live-generation corpora require the model to respond freely, with post-hoc labels. On teacher-forced corpora, artifact-exploiting methods reach AUROC 0.85 – 1.00 but collapse to near-chance once the artifact is removed. On live-generation corpora, SAPLMA and DRIFT, both supervised probes, reliably exceed chance on HaluBench, reaching AUROC 0.91 (mean 0.879 across twelve architectures), while every label-free method, including MIND, HaloScope, ACT, SEPs, and HalluShift, fails near chance regardless of architecture. On RAGTruth every method falls between 0.43 and 0.57 , supervised or otherwise. Our contributions: (1) A large-scale controlled evaluation of 22 detection methods across 6 corpora and 12 models with TxTemb artifact controls, permutation tests, and bootstrap CIs. (2) The finding that teacher-forcing inflates reported AUROC by up to 0.43 points, and that SAPLMA (2023) achieves the strongest controlled HaluBench performance among all externally published prior methods, a result obscured by artifact-contaminated evaluations in the prior literature. (3) The finding that all 22 methods cluster between AUROC 0.43 and 0.57 on RAGTruth, with no method significantly exceeding chance under permutation tests, establishing RAGTruth as a concrete unsolved benchmark for activation-based detection. (4) DRIFT , a supervised inter-layer probe within 0.004 of SAPLMA on HaluBench ( 0.915 vs. 0.911 ) and statistically indistinguishable on RAGTruth ( p{=}0.82 ), confirming that the positive HaluBench result is not unique to a 2023 single-layer architecture; the DRIFT-concat ablation (same layers, concatenated) further isolates upper-layer depth selection as the primary driver of supervised probe performance. (5) Code and evaluation scripts released at https://anonymous.4open.science/r/llm-fuzzing-2A24 . L_{0} L_{n} LLM (frozen) 0.60n 0.70n 0.80n 0.85n prompt + response tokens mean-pool over T tokens \in\mathbb{R}^{d} h_{1} h_{2} h_{3} h_{4} \boldsymbol{\phi}_{34} : \mathbf{h}_{4}{-}\mathbf{h}_{3},\;\cos(\cdot),\;\|\cdot\|_{2}\;\in\mathbb{R}^{d+2} \vdots \boldsymbol{\phi}_{12} : \mathbf{h}_{2}{-}\mathbf{h}_{1},\;\cos(\cdot),\;\|\cdot\|_{2}\;\in\mathbb{R}^{d+2} concat \mathbf{z}\!\in\!\mathbb{R}^{K(d+2)} 49{,}164 -dim LogisticProbe L2-regularised 5-fold CV fit on labeled pairs P(\text{hal.}) w_{E}= contrastive dir. SAPLMA: single layer, last token \to\mathbb{R}^{d}\to probe. HalluShift: scalar 1{-}\cos(\mathbf{h}_{a},\mathbf{h}_{b}) . DRIFT-concat: 3-layer concat, no \boldsymbol{\phi} diffs. Figure 1 : DRIFT architecture. Hidden states are tapped at four upper-layer positions (60–85% depth), mean-pooled over T tokens, and differenced across all C(4,2){=}6 pairs to form \boldsymbol{\phi}_{ab}\in\mathbb{R}^{d+2} , then concatenated into \mathbf{z}\in\mathbb{R}^{49{,}164} . An L2-regularised logistic probe maps \mathbf{z} to P(\text{hallucination}) ; its weight vector w_{E} doubles as a hallucination contrastive direction. Dashed box: baseline comparisons. 2 Related Work Internal-state hallucination probes. SAPLMA ( 3 ) trains a logistic probe on the last-layer hidden state at the final token. INSIDE ( 8 ) uses eigenvalues of the hidden-state covariance matrix; MIND ( 58 ) applies an unsupervised variance detector at runtime; HaloScope ( 14 ) scores reconstruction error from a truncated SVD; HaMI ( 45 ) uses multiple-instance learning over stochastic samples; PRISM ( 21 ) unifies internal-state signals via cross-attention; HalluShift ( 12 ) computes a scalar inter-layer cosine distance; and ACT-ViT ( 5 ) applies activation tensor decomposition via vision transformer encoders. DRIFT is an alternative in this family that extends the probing approach to inter-layer directions, building on evidence that models encode incorrectness before it appears in output ( 48 ; 20 ; 53 ; 64 ; 32 ) and on probing methodology foundations ( 6 ; 61 ) . Benchmark artifacts and output-space methods. HaluEval ( 30 ) concatenates correct and hallucinated answers in the same prompt, enabling lexical shortcuts unrelated to model uncertainty; we quantify this via the TxTemb gap criterion. HADES ( 34 ) provides a reference-free token-level alternative, though surface artifacts remain pervasive across popular corpora. Analogous annotation challenges arise in faithfulness evaluation ( 38 ; 27 ; 57 ; 41 ) and knowledge-limit studies ( 52 ; 35 ) further motivate live-generation corpora. SelfCheckGPT ( 36 ) measures NLI inconsistency across N{=}10 samples; P(True) ( 25 ) prompts the model to self-assess; DoLa ( 10 ) contrasts early/late-layer logits; and semantic entropy ( 18 ; 26 ) clusters responses by entailment. All require extra inference passes and, as we show, yield near-chance AUROC on RAGTruth. Activation steering. CAA ( 50 ) , RepE ( 73 ) , ITI ( 31 ) , ACT ( 65 ) , FASB ( 9 ) , CCS ( 7 ) , and ActAdd ( 63 ) build and inject contrastive steering vectors. DRIFT’s probe weight vector doubles as a hallucination contrastive direction, offering a unified detect-then-steer path without a separate steering-vector computation. 4 show no single direction generalises across prompts, motivating multi-pair designs; dynamic and real-time variants ( 69 ; 46 ; 55 ) operationalise these at deployment. Mechanistic interpretability, cascades, and domain evaluation. Residual stream analysis ( 17 ; 40 ) and hallucination studies in RAG and reasoning ( 59 ; 13 ; 68 ) provide mechanistic grounding for internal-state methods; hallucinated tokens propagate autoregressively ( 70 ; 16 ; 28 ) . Domain evaluation in clinical and high-stakes settings ( 2 ; 47 ; 66 ; 51 ; 43 ) extends the landscape. 3 Experimental Setup Models. We evaluate 12 instruction-tuned ( 49 ) models spanning six families (3.8B–72B parameters): Llama (3.1-8B, 3.3-70B) ( 62 ; 15 ) , Qwen (3-8B, 3-32B, 2.5-72B), DeepSeek-R1 distillations (7B, 70B; reasoning models with chain-of-thought ( 67 ) ), Mistral (Mixtral-8x7B MoE, Mistral-24B), Gemma (3-4b, 3-12b), and Phi-4-mini. Llama-3.3-70B is the primary model (all 22 methods); the remaining 11 run DRIFT, SAPLMA, and SelfCheckGPT-NLI on TruthfulQA and HaluBench. Full model identifiers appear in Appendix A . Train/test split. All corpora are split 80/20 stratified by label (test set strictly held out); cross-validation methods further split the training set into 5 folds. Compute. Full evaluation takes 4–6 h per model for 8B and 18–24 h for 70B (Appendix K ). INSIDE is the slowest baseline ( \approx 6 h per corpus on 70B). Computational cost per method. All probe-based methods share one hidden-state extraction pass per prompt ( 15.6 s loading, 9.6 ms/prompt amortised). At test time: DRIFT-logp adds 2.1 ms, SAPLMA 111 ms, DRIFT 1.2 s. SelfCheckGPT-NLI requires ten additional GPU inference passes per prompt, placing it in a wholly different cost class. Full per-method benchmarks are given in Appendix B (Table 6 ) and the Pareto frontier is presented in Appendix C . 3.1 Benchmark Suite We evaluate on six publicly available corpora spanning domain, annotation style, and generation format (Table 1 ). Table 1 : Benchmark suite. TF = teacher-forced (answer text in prompt); LG = live-generated (model produces the response freely). Corpus Domain N Balance Label source Format HaluEval ( 30 ) Open-domain QA 500 250/250 ChatGPT-generated pairs ( 1 ) TF MedHallu ( 39 ) Medical QA 500 250/250 Reference pairs TF TruthfulQA ( 33 ) Factual QA 1,634 817/817 Reference pairs TF Legal ( 11 ) Legal case QA 499 159/340 Expert correctness score TF RAGTruth ( 44 ) RAG (news/QA) 500 153/347 Expert annotation LG HaluBench ( 54 ) Multi-domain QA 500 250/250 PatronusAI PASS/FAIL LG Four corpora use teacher-forced (TF) construction, where both a correct and a hallucinated response appear in the prompt; the remaining two (RAGTruth, HaluBench) are live-generated (LG), with the model responding freely. The TF artifact is strongest on HaluEval ( TxTemb =0.98 ), moderate on MedHallu ( TxTemb =0.71 ) and Legal ( TxTemb =0.89 ; despite nuanced expert correctness judgements, surface-text cues remain highly predictive), and weakest on TruthfulQA ( TxTemb =0.69 , which still requires factual knowledge to distinguish correct from incorrect answers). Figure 2 illustrates the two regimes. Teacher-Forced HaluEval \cdot MedHallu \cdot Legal \cdot TruthfulQA prompt + ‘‘[Correct: X] [Hallucination: Y]’’ \rightarrow label embedded in text TxTemb AUROC: 0.69–0.98 R 2 : 0.81 \triangle Artifact: answer text in prompt Live-Generation RAGTruth \cdot HaluBench prompt \to model generates freely \to post-hoc expert label TxTemb AUROC: \approx 0.50 R 2 : 0.08 ✓ Real signal: model uncertainty Figure 2 : Two evaluation regimes. Left: Teacher-forced benchmarks embed the answer in the prompt; TxTemb achieves AUROC 0.69–0.98 and explains 81% of variance in method AUROC ( R^{2}{=}0.81 ), a formatting artifact. Right: On live-generation benchmarks TxTemb is at chance ( {\approx}0.50 , R^{2}{=}0.08 ), isolating genuine internal-state signal. RAGTruth ( 44 ) is our primary live-generation benchmark: responses are generated by a RAG pipeline ( 29 ) under normal inference conditions, where faithfulness to source passages is essential ( 56 ) , and annotated post-hoc by human experts; no answer text is embedded in the prompt. HaluBench ( 54 ) uses PatronusAI’s automated LLM-as-judge evaluator ( 72 ) on freely generated multi-domain QA; its absence of answer text in the prompt gives it the similar live-generation character. Artifact control: text-embed baseline. On teacher-forced corpora, hallucinated and correct answers differ in surface text. We implement TxTemb : a purely text-based detection signal defined as s_{\textsc{TxTemb}}(r)=\cos\!\bigl(\mathrm{tfidf}(r_{\mathrm{hal}}),\,\mathrm{tfidf}(r_{\mathrm{ref}})\bigr), (1) where r_{\mathrm{hal}} and r_{\mathrm{ref}} are the hallucinated and reference answer texts. The key insight is that TxTemb accesses only the text of the responses, with no knowledge of the model’s internal states, probability distributions, or reasoning process. If an internal-state method achieves AUROC comparable to TxTemb on the same corpus, its detection signal is explained by the same surface-text differences that a lexical baseline can already exploit. In that case, the method is not detecting genuine model uncertainty or hallucination-prone internal states; it is, at best, re-encoding information already present in the prompt text through hidden representations. Any performance claim for such a method on that corpus is therefore attributable to the benchmark construction artifact rather than to a meaningful internal-state signal. A cell is flagged ( \dagger ) if the method’s AUROC is within 0.05 of TxTemb AUROC on the same corpus. The 0.05 threshold is chosen to exceed typical bootstrap confidence interval half-widths observed in our experiments (mean CI half-width: 0.024 ), ensuring that a flagged method’s performance advantage over the surface baseline is not statistically distinguishable from zero. This is a deliberately conservative criterion: a method that outperforms TxTemb by only 0.05 AUROC points may still be exploiting the same artifact through its hidden representations, and we do not wish to overstate the absence of artifact contamination. Evaluation metrics and verification. We use Area Under the ROC Curve (AUROC) ( 19 ) as our primary metric because it is threshold-independent, invariant to class imbalance, and measures the probability that a randomly chosen positive instance is ranked above a randomly chosen negative one, making it appropriate for comparing detectors across the six corpora in our benchmark, which vary substantially in label balance (Table 1 ). For all cells with AUROC >0.85 we additionally compute: (i) bootstrap 95% CI ( n=1000 resamples); (ii) permutation-test null mean ( n=30 label shuffles); (iii) TxTemb gap. A cell is declared Validated if the CI lower bound exceeds 0.80 and the TxTemb gap exceeds 0.05 . 4 Methods Our evaluation covers six detection approaches of our own design alongside fourteen external baselines spanning label-free, unsupervised, decoding-time, and supervised-probe paradigms. DRIFT (Approach E) is our primary design contribution and the strongest performer. Approaches A and E are self-supervised (no labels), C and D require N{=}10 stochastic samples per prompt, and B and F require teacher-forced format. All six share a common feature backbone: hidden states from a single forward pass pooled per-layer across four upper-layer checkpoints (Section 4 ), enabling any subset to be evaluated on the same stored activations. Compute and quantisation. All experiments run on a server with 8\times NVIDIA L40S GPUs (46 GB each). Hidden-state extraction runs on a single GPU in 4-bit NF4 quantisation (bitsandbytes double-quantisation) to fit 70B models in memory; batch generation uses 4-GPU tensor parallelism ( tp=4 ). One model–corpus pair (Qwen3-32B \times TruthfulQA) required re-extraction with 8-bit quantisation after NF4 produced NaN activations (KV-cache overflow at layers 38, 44, 51, 54); the resolved result (DRIFT AUROC =0.671 ) is included in Table 4 . Layer selection. We extract hidden states at fractional depths \{0.60,\,0.70,\,0.80,\,0.85\}\times L (Llama-3.3-70B: layers 48, 56, 64, 68), confirmed by Appendix D . Truth-relevant directions reside in the upper network third ( 37 ) (AUROC 0.741 at 20% vs. 0.898 at 85%), final layers are dominated by next-token prediction, and four taps outperform any single tap by +0.017 AUROC. These fractions were fixed before seeing any results and applied uniformly across all twelve architectures. Hidden state extraction. A forward pass over [\text{prompt}+\text{response}] yields hidden states at response token positions. Let T denote the number of response tokens and \mathbf{h}^{(\ell)}_{t}\in\mathbb{R}^{d} the hidden state at layer \ell and position t . The per-layer representation is mean-pooled: \mathbf{h}^{(\ell)}=\frac{1}{T}\sum_{t=1}^{T}\mathbf{h}^{(\ell)}_{t}\;\in\;\mathbb{R}^{d}. (2) The four layer vectors are concatenated to form \mathbf{h}\in\mathbb{R}^{4d} . The full prompt is included so activations are contextualised on the question. Probe architectures. Two probe types are used: a LogisticProbe (L2-regularised logistic regression, 5-fold CV over C\!\in\!\{0.001,0.01,0.1,1\} ) and an MLPProbe (two-layer feedforward, \min(256,d/4) hidden units, ReLU, BCE loss). The LogisticProbe weight vector simultaneously serves as the decision boundary and the hallucination contrastive direction ( 31 ) . Approach A: Perturbation Contrast (Self-Supervised). Four strategies generate a corrupted response r^{\prime} from prompt p and response r (entity swap, numerical corruption, negation flip, boundary violation). An MLPProbe is trained on \Delta\mathbf{h}=\mathbf{h}(p,r)-\mathbf{h}(p,r^{\prime}) . No hallucination labels are required; the perturbation provides supervision. Approach B: Contrastive Activation Addition. For teacher-forced data with paired correct/hallucinated responses, we compute \mathbf{v}_{\text{CAA}}=\frac{1}{N}\sum_{i}\mathbf{h}(p_{i},r^{+}_{i})-\mathbf{h}(p_{i},r^{-}_{i}) and score s_{B}=\cos(\mathbf{h}(p,r),\mathbf{v}_{\text{CAA}}) . † AUROC =1.0 on MedHallu is an artifact of the teacher-forced construction. Approach B requires paired responses for the same prompt (teacher-forced only). In MedHallu, GPT-4 generated pairs with the reference answer visible in the prompt, so \mathbf{v}_{\text{CAA}} encodes surface-text agreement rather than a genuine hallucination signal, and the method does not apply to live-generation corpora. Approach C: Semantic Entropy Probe. Semantic entropy H=-\sum_{c}p_{c}\log p_{c} is computed from N{=}10 stochastic samples ( 18 ) , then an MLPProbe is trained on \mathbf{h} to predict H . This amortises entropy computation to a single inference pass. Approach D: Activation Variance Consistency. Element-wise variance \mathbf{v}=\operatorname{Var}(\mathbf{h}_{1},\ldots,\mathbf{h}_{N}) over N{=}10 stochastic completions is fed to a LogisticProbe, capturing which activation dimensions are uncertain across samples. Approach E: DRIFT. Let \mathcal{L}=\{l_{1},l_{2},l_{3},l_{4}\} denote the four selected layers and \binom{\mathcal{L}}{2} the set of K{=}6 unordered layer pairs. For each pair (l_{a},l_{b}) with l_{a}