Inside-Out: Hidden Factual Knowledge in LLMs
Zorik Gekhman, Eyal Ben David, Hadas Orgad, Eran Ofek, Yonatan Belinkov, Idan Szpektor, Jonathan Herzig, Roi Reichart · 2025
Read Inside-Out: Hidden Factual Knowledge in LLMs by Zorik Gekhman, Eyal Ben David, Hadas Orgad, Eran Ofek, Yonatan Belinkov, Idan Szpektor, Jonathan Herzig, Roi Reichart (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15299.txt.
Introduction
Zorik Gekhman and co-authors show that language models know more facts than they say, in a 2025 paper. They define knowledge as how often a model ranks a right answer above a wrong one. Scores can come from the words the model would say, or from its inner states as it reads.
On three open models, inner scores beat outer scores by about 40 percent on average. Some facts were known perfectly inside yet never came out, even across a thousand sampled tries. That gap caps what extra sampling at test time can buy. The study gives the field a clean way to measure hidden knowledge and a warning about generation limits.
Catalog record
| CREATOR | Zorik Gekhman, Eyal Ben David, Hadas Orgad, Eran Ofek, Yonatan Belinkov, Idan Szpektor, Jonathan Herzig, Roi Reichart |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 78 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-15299
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Inside-Out: Hidden Factual Knowledge in LLMs Zorik Gekhman, Eyal Ben David, Hadas Orgad, Eran Ofek, Yonatan Belinkov, Idan Szpektor, Jonathan Herzig, Roi Reichart Abstract This work presents a framework for assessing whether large language models (LLMs) encode more factual knowledge in their parameters than what they express in their outputs. While a few studies hint at this possibility, none have clearly defined or demonstrated this phenomenon. We first propose a formal definition of knowledge , quantifying it for a given question as the fraction of correct-incorrect answer pairs where the correct one is ranked higher. This gives rise to external and internal knowledge, depending on the information used to score individual answer candidates: either the model’s observable token-level probabilities or its intermediate computations. Hidden knowledge arises when internal knowledge exceeds external knowledge. We then present a case study, applying this framework to three popular open-weight LLMs in a closed-book QA setup. Our results indicate that: (1) LLMs consistently encode more factual knowledge internally than what they express externally, with an average relative gap of 40%. (2) Surprisingly, some knowledge is so deeply hidden that a model can internally know an answer perfectly , yet fail to generate it even once, despite large-scale repeated sampling of 1,000 answers. This reveals fundamental limitations in the generation capabilities of LLMs, which (3) put a practical constraint on scaling test-time compute via repeated answer sampling in closed-book QA: significant performance improvements remain inaccessible because some answers are practically never sampled, yet if they were, we would be guaranteed to rank them first. 1 Introduction Large language models (LLMs) excel at knowledge-intensive tasks, yet fundamental questions remain open about the nature of their factual knowledge. What does it mean for LLMs to know a fact? And equally important, do LLMs store more factual information in their parameters than they express in their outputs? Answering the latter question rigorously can be highly valuable: if such knowledge is stored but not used, we might be able to develop methods to surface it, improving performance and reliability. From a safety perspective, undisclosed knowledge poses risks if it unexpectedly surfaces, revealing sensitive information or producing outputs never meant to be shared. Lastly, beyond practical considerations, this question is key to advancing interpretability: if models encode more knowledge than they express in their outputs, it highlights the need to understand how this knowledge is accessed or suppressed during inference. In this work, our goal is to study whether LLMs encode more factual knowledge in their parameters than they express in their outputs, a phenomenon we refer to as hidden knowledge . So far, prior work has only hinted at its existence ( Burns et al. 2023 ; Orgad et al. 2024 ) , but none have clearly defined it or the conditions required to demonstrate its existence. Consequently, our core contribution is proposing a concrete definition that provides a framework for systematically studying hidden knowledge. To define hidden knowledge we first need to define knowledge , which is challenging since there is no well-defined notion of it for LLMs in the literature ( Fierro et al. 2024 ) . We couple knowledge with the ability to rank correct answers above incorrect ones using a scoring method (e.g., token-level likelihood) and quantify it per-question as the fraction of (correct, incorrect) answer pairs ranked correctly (§ 2.1 ). While other possible definitions could be proposed, ours has three key advantages. First, unlike existing perspectives on LLM’s knowledge ( Fierro et al. 2024 ) , it is specified with a clear computational procedure, making it useful for empirical studies. Second, it addresses limitations in the common practice of measuring knowledge by evaluating performance of a single predicted answer on closed-book question-answering (QA) benchmarks ( Petroni et al. 2019 ; Wei et al. 2024 ) . Lastly, and most importantly, it is designed with the hidden knowledge research question in mind, ensuring that both the knowledge a model expresses externally and the knowledge it encodes internally are measured under a unified definition. To quantify knowledge for a given question, we only need a function that assigns scores to candidate answers using information from the model. Thus, testing for hidden knowledge comes down to measuring knowledge through different scoring functions: external ones, 1 1 1 The term external may suggest elements outside the model, which is not our intention. An alternative name could be observable . We choose to use external since it provides a smooth reading flow in our context. which are restricted to use only observable signals based on the model’s token-level probabilities, and internal ones, which can use intermediate computations. Specifically, we define hidden knowledge as the existence of an internal function that ranks answers more accurately than any external function (§ 2.2 ). Using this framework, we design a targeted study with 1,700 closed-book QA questions. We estimate the set of (correct, incorrect) answer pairs per question using 1,000 model-generated answers, labeled for correctness by an LLM judge that compares against the ground truth ( Wei et al. 2024 ) . For internal scoring, we train a linear classifier to predict correctness from the model’s hidden representations of question-answer pairs ( Hupkes et al. 2018 ; Belinkov & Glass 2019 ) . We then compare the internal knowledge measured by this classifier to external knowledge measured by popular scoring methods that use observable signals based on the model’s token-level probabilities. Our results strongly indicate the presence of hidden knowledge. Across three LLMs, the internal scoring method consistently measures more knowledge than any external one, with all differences statistically significant and an average relative gap of 40%. Our formal definition and framework lay the foundation for further study of this important phenomenon. But how deeply can knowledge be hidden? In 56\% of the questions, the 1,000 model-generated answers are all wrong. So we manually add the ground-truth answer from our dataset to the set of candidate answers, and analyze its impact on the model’s internal knowledge scores. Surprisingly, in 9\% of the questions, the internal scoring method scores the ground-truth answer higher than any incorrect candidate, despite the model failing to generate it even once across 1,000 attempts. This illustrates an extreme case of hidden knowledge: while models may occasionally generate an incorrect answer despite knowing the correct one ( Simhi et al. 2024 ) , it is highly surprising that a perfectly known correct answer is practically never generated, even with large scale repeated sampling. This highlights a fundamental limitation in the generation process of modern LLMs, which we hope future research on decoding mechanisms will explore. Lastly, we leverage our setup and findings to enhance performance in a challenging closed-book QA setting. We obtain 12% average relative improvement over greedy decoding by increasing test-time compute through sampling a large set of answers and selecting the top one based on our internal scoring function. This extends existing evidence on the potential of using verifiers trained on LLMs hidden states to improve performance ( Orgad et al. 2024 ) . However, the more interesting insight is that there is potential for substantial gains of additional 40% relative improvement that remain inaccessible due to the constraints we identified in the LLM generation process: