CORTEX: Token-Level Hallucination Detection in RAG via Comparative Internal Representations
Kazuaki Furumai, Shuichiro Haruta, Kazunori Matsumoto, Daisuke Kamisaka · 2026
Read CORTEX: Token-Level Hallucination Detection in RAG via Comparative Internal Representations by Kazuaki Furumai, Shuichiro Haruta, Kazunori Matsumoto, Daisuke Kamisaka (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-31033.txt.
Introduction
Kazuaki Furumai, Shuichiro Haruta, Kazunori Matsumoto and Daisuke Kamisaka hunt hallucinations inside retrieval-augmented answers one token at a time. In long RAG outputs, invented content usually sits in local spans, not spread through the whole answer, so whole-response verdicts are too coarse. Their method, CORTEX, rests on a simple intuition: tokens truly grounded in the retrieved documents should be shaped by those documents more than invented tokens are. CORTEX compares the model's internal representations with and without the documents in context.
It also tracks how document influence propagates through earlier tokens, which cuts false alarms on tokens whose evidence was already absorbed. A smoothing step then exploits how hallucination labels tend to persist across neighbouring tokens. On two RAG benchmarks and three language models, CORTEX substantially improved token-level detection, with each component contributing gains.
Catalog record
| CREATOR | Kazuaki Furumai, Shuichiro Haruta, Kazunori Matsumoto, Daisuke Kamisaka |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.31033
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CORTEX: Token-Level Hallucination Detection in RAG via Comparative Internal Representations Kazuaki Furumai, Shuichiro Haruta, Kazunori Matsumoto, Daisuke Kamisaka Abstract In this paper, we propose CORTEX, a token-level hallucination detection method for Retrieval-Augmented Generation (RAG). In long-form RAG outputs, hallucinations often arise in localized spans rather than throughout an entire response. CORTEX therefore identifies ungrounded content at the token level, enabling fine-grained localization of hallucinations. The key intuition behind CORTEX is that tokens grounded in retrieved documents should be more strongly influenced by those documents than hallucinated tokens. To capture this document-induced effect, CORTEX compares internal representations of a large language model (LLM) under two conditions: with and without the retrieved documents. Instead of relying solely on each token’s immediate sensitivity to the retrieved documents, CORTEX also leverages the propagation of document-grounded information through preceding tokens, reducing false positives for tokens whose evidence has already been absorbed into the context. Finally, CORTEX applies post-processing smoothing step that models the tendency of hallucination labels to persist over contiguous spans, reducing local noise and encouraging span-consistent predictions. Experiments on two RAG benchmarks and three LLMs show that CORTEX substantially improves token-level hallucination detection, with each component consistently contributing to performance gains. 1 Introduction Figure 1: Overview of CORTEX for token-level hallucination detection in RAG, using reference-conditioned internal representation comparisons, contextual residual features, and label-persistence smoothing. Retrieval-Augmented Generation (RAG) has been widely adopted to improve the factuality of large language models (LLMs) by grounding generation in external references ( 11 ) . Although RAG mitigates hallucinations to some extent, LLMs may still generate groundless or inconsistent content even when references are provided ( 4 ) . Accurately detecting such hallucinations within generated outputs therefore remains a critical challenge ( 17 ; 13 ; 2 ) . Various hallucination detection methods have been proposed to address this challenge. Self-consistency-based approaches estimate reliability by measuring agreement across multiple outputs from the same prompt ( 15 ) . Prompt-based methods use LLMs as fact-checkers to explicitly verify generated content ( 26 ; 3 ; 5 ) . While these approaches leverage LLMs’ strong language understanding capabilities, they may incur high computational cost due to repeated sampling, and their reliability can depend on prompt design ( 6 ) . More recently, hallucination detection methods based on the internal representations of LLMs have been increasingly studied ( 22 ; 25 ; 23 ) . These methods seek hallucination-related signals in model activations and can offer insight into the internal conditions under which hallucinations arise. This line of work has analyzed attention and feedforward-network states, quantified component contributions to generation, and extracted hallucination-related features from internal representations. However, many existing approaches are not designed specifically for RAG and primarily assess the generated output as a whole, i.e., at the answer level. This is limiting for long-form RAG outputs, where faithful and hallucinated content often coexist. For practical use, detecting hallucinations in such mixed outputs requires a finer-grained formulation than answer-level detection. The appropriate granularity can vary across tasks and annotation protocols, from words and phrases to sentences or paragraphs. Given this variability, token-level detection can serve as a practical approach for localizing ungrounded content and later adapting predictions to the required granularity. In this paper, we propose CORTEX ( C omparative O bserved R eference-based T oken-level EX pressions), a post-hoc token-level hallucination detection method for RAG. Figure 1 illustrates the overall framework. CORTEX builds on the intuition that faithful tokens should exhibit coherent representational changes when references are provided, whereas hallucinated tokens should show weaker or less coherent changes. To measure this effect, CORTEX constructs a paired counterfactual view of the same answer under reference-conditioned and no-reference inputs, rather than probing a single input as in conventional representation-based detectors. The resulting reference-induced changes are then encoded as token-level delta features for hallucination detection. CORTEX further introduces an attention-based contextual residual feature. In long-form generation, reference influence may propagate indirectly through preceding answer tokens, as in reasoning-style or self-referential generation. CORTEX captures this context-mediated reference influence by combining token-level delta features with attention patterns, helping the classifier distinguish tokens that are indirectly grounded in the reference from groundless tokens and thereby reducing false positives. Finally, CORTEX uses post-processing smoothing step to adjust the granularity of token-level scores. Raw token-level predictions can be sensitive to local noise and may produce isolated high-score tokens, whereas human hallucination annotations often appear as contiguous spans. We therefore introduce label-persistence smoothing , which treats raw scores as token-level confidence and models the tendency of hallucination labels to persist across neighboring tokens. This reduces scattered noise and adapts token-level predictions to span-level annotation structure while still retaining token-level scores. Experiments on two RAG benchmarks and three LLMs demonstrate that CORTEX substantially improves token-level hallucination detection, with all components contributing consistently to performance gains. Our contributions are summarized as follows: • We propose CORTEX, which, to the best of our knowledge, is the first token-level hallucination detection method specifically designed for RAG. • CORTEX is a practical post-hoc framework that is easy to implement and can detect hallucinations in outputs from arbitrary LLMs, including API-based closed-weight models. 2 Preliminaries We consider a practical setting in which the answer is produced by a closed-weight LLM whose parameters and internal representations are not accessible, as is the case for many API-based models. Such models are often attractive in deployed applications because of their strong generation quality. We denote this closed-weight LLM by M_{\mathrm{close}} . Given a question q^{\mathrm{text}} , the closed-weight LLM produces an answer as a^{\mathrm{text}}=M_{\mathrm{close}}(q^{\mathrm{text}}). (1) Our goal is to identify hallucinated tokens in the generated answer a^{\mathrm{text}} . Although a direct way to obtain hallucination-related features would be to analyze the internal representations of the closed-weight LLM, we cannot do so for the aforementioned reason. We therefore use a separate open-weight LLM, denoted by M_{\mathrm{open}} , as a post-hoc analysis model. Unlike M_{\mathrm{close}} , M_{\mathrm{open}} exposes internal representations, which allows us to extract features from the answer in relation to the question. Let \mathrm{Tok}(\cdot) denote the tokenizer of M_{\mathrm{open}} . We tokenize the answer as \displaystyle a \displaystyle=\mathrm{Tok}(a^{\mathrm{text}})=(t_{1},t_{2},\ldots,t_{i},\ldots,t_{T}), (2) where T denotes the number of answer tokens and t_{i} denotes the i -th answer token under the tokenizer of M_{\mathrm{open}} . We assume that M_{\mathrm{open}} processes an input containing the question and the answer as M_{\mathrm{open}}(q^{\mathrm{text}}\|a^{\mathrm{text}}) , where \| denotes text concatenation with the appropriate prompt format. M_{\mathrm{open}} produces token-level internal representations over the entire input sequence, which capture the relationships between q^{\mathrm{text}} and a^{\mathrm{text}} . Since hallucination detection is performed over the answer, we use only the internal representations corresponding to the answer tokens. We denote the representation aligned with answer token t_{i} by h_{i}\in\mathbb{R}^{d} . The specific input construction and the extraction of answer-token representations are described in Section 3 . The token-level hallucination detection task is to estimate whether each answer token is hallucinated. Let y_{i}\in\{0,1\} denote the token-level label, where y_{i}=1 indicates that token t_{i} is hallucinated and y_{i}=0 indicates that it is faithful. In our setting, we focus on RAG applications, where references are provided as grounding evidence for answers. This setting reflects common practical deployments in which retrieved documents are used to reduce hallucination and the answer is expected to