Unveiling Knowledge Utilization Mechanisms in LLM-based Retrieval-Augmented Generation Conference: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 13–18, 2025; Padua, Italy Proceedings of the 48th International ACM SIGIR Conferenc
Yuhao Wang, Ruiyang Ren, Yucheng Wang, Wayne Xin Zhao, Jing Liu, Hua Wu, Haifeng Wang · 2025
Read Unveiling Knowledge Utilization Mechanisms in LLM-based Retrieval-Augmented Generation Conference: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 13–18, 2025; Padua, Italy Proceedings of the 48th International ACM SIGIR Conferenc by Yuhao Wang, Ruiyang Ren, Yucheng Wang, Wayne Xin Zhao, Jing Liu, Hua Wu, Haifeng Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11995.txt.
Introduction
Yuhao Wang and six co-authors open up a language model to watch it use retrieved knowledge. Their 2025 paper studies retrieval-augmented generation, where the model answers with passages fetched from outside. The puzzle is internal: how do the model's built-in knowledge and the fetched text flow, mix, and fight inside its layers?
Tracing that flow, the team divides the process into four stages: refining the knowledge, eliciting it, expressing it, and a contest stage where sources compete. Passage relevance steers the flow through the stages. At the finer scale, the authors introduce an entropy measure that spots individual neurons tied to internal or to external knowledge. Switching them off shifts the model's reliance between the sources. Attention heads and the feed-forward layers, they find, play complementary parts in forming the knowledge. The work aims at systems whose answers can be audited, not just read.
Catalog record
| CREATOR | Yuhao Wang, Ruiyang Ren, Yucheng Wang, Wayne Xin Zhao, Jing Liu, Hua Wu, Haifeng Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-11995
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Unveiling Knowledge Utilization Mechanisms in LLM-based Retrieval-Augmented Generation Conference: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 13–18, 2025; Padua, Italy Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’25), July 13–18, 2025, Padua, Italy DOI: 10.1145/3726302.3730112 ISBN: 979-8-4007-1592-1/2025/07 CCS: Computing methodologies Natural language processing Yuhao Wang, Ruiyang Ren, Yucheng Wang, Wayne Xin Zhao, Jing Liu, Hua Wu, Haifeng Wang Abstract . Considering the inherent limitations of parametric knowledge in large language models (LLMs), retrieval-augmented generation (RAG) is widely employed to expand their knowledge scope. Since RAG has shown promise in knowledge-intensive tasks like open-domain question answering, its broader application to complex tasks and intelligent assistants has further advanced its utility. Despite this progress, the underlying knowledge utilization mechanisms of LLM-based RAG remain underexplored. In this paper, we present a systematic investigation of the intrinsic mechanisms by which LLMs integrate internal (parametric) and external (retrieved) knowledge in RAG scenarios. Specially, we employ knowledge stream analysis at the macroscopic level, and investigate the function of individual modules at the microscopic level. Drawing on knowledge streaming analyses, we decompose the knowledge utilization process into four distinct stages within LLM layers: knowledge refinement, knowledge elicitation, knowledge expression, and knowledge contestation. We further demonstrate that the relevance of passages guides the streaming of knowledge through these stages. At the module level, we introduce a new method, knowledge activation probability entropy (KAPE) for neuron identification associated with either internal or external knowledge. By selectively deactivating these neurons, we achieve targeted shifts in the LLM’s reliance on one knowledge source over the other. Moreover, we discern complementary roles for multi-head attention and multi-layer perceptron layers during knowledge formation. These insights offer a foundation for improving interpretability and reliability in retrieval-augmented LLMs, paving the way for more robust and transparent generative solutions in knowledge-intensive domains. 1. Introduction Despite advanced capabilities, large language models (LLMs) ( Brown et al. 2020 ; Zhao et al. 2023 ) often struggle with knowledge-intensive challenges such as open-domain question answering (QA) ( Petroni et al. 2020 ) . These limitations arise primarily from LLMs’ reliance on parametric knowledge, which often proves inadequate for real-time queries or domain-specific information, leading to factual hallucinations ( Cheng et al. 2024 ) . To migrate this issue, researchers have developed the retrieval-augmented generation (RAG) technique ( Gao et al. 2023 ) , which first retrieves relevant information from an external knowledge base and then incorporates it as supplementary external knowledge into the input context. This approach not only enhances the model’s parametric knowledge without additional training, but also expands LLMs’ knowledge boundaries and improves their reliability and transparency ( Lewis et al. 2020 ) . As a result, the potential of RAG has been harnessed in domains spanning complex tasks addressing ( Cheng et al. 2025 ) , intelligent information assistants ( Lála et al. 2023 ) , and autonomous agents ( Ren et al. 2025 ) . Recently, researchers have shifted their focus towards understanding how RAG leverages knowledge, moving beyond merely aiming for task-specific performance improvements ( Wang et al. 2024 ) . Existing studies have explored the impact of retrieval augmentation on the knowledge boundaries of LLMs and identified key factors that influence such knowledge boundaries in RAG scenario ( Ren et al. 2023 ) . Additionally, effort examines the conflicts between parametric knowledge and non-parametric knowledge when LLM utilizes external context, proposing methods to prune conflicting modules and mitigate these conflicts ( Jin et al. 2024b ) . Despite recent progress, the field still lacks a comprehensive mechanistic understanding within the RAG framework of how LLMs navigate and reconcile both internal (parametric) and external (retrieved) knowledge ( Wang et al. 2024 ) , particularly at a compositional level. Clear mechanistic insights are crucial for advancing our understanding of RAG’s knowledge utilization and diagnosing potential pitfalls of RAG that emerged with transformer-based architectures ( Vaswani et al. [n. d.] ) . Recent studies on model mechanisms have enhanced our understanding of the Transformer architecture ( Xiao et al. 2017 ) . However, RAG introduces two unique challenges for interpretability, presenting new complexities compared to existing tasks like in-context learning (ICL) or mathematical reasoning ( Hanna et al. 2024 ; Dong et al. 2025 ) . First, RAG operates with unstructured data, which means the external knowledge inputs lack a fixed format. This factor increases the difficulty of analyzing how the model understands the inputs and extracting information as a reference. Second, RAG involves both internal and external sources of knowledge ( Jin et al. 2024a ) . This requires the LLM to integrate and select information from both parametric knowledge and contextual knowledge, demanding a nuanced understanding of how these sources interact. In this study, we propose a comprehensive investigation of the intrinsic mechanisms governing LLMs within the RAG framework, systematically analyzing them from an interpretive perspective. Specially, we develop new analytical strategies that illuminate both macro-level knowledge streaming and micro-level module contributions within the Transformer architecture. Based on these perspectives, we propose two fundamental research questions: (1) How does knowledge stream within the RAG framework? (2) How do LLM modules function in knowledge utilization? To explore the first question, we quantitatively evaluate the stream of knowledge through each layer of the LLM using information flow methodologies. This approach allows us to track how internal and external knowledge streams evolve across layers within the RAG framework. For the second question, we introduce a new approach to analyze the role of neurons, the multi-head attention (MHA) module, and the multi-layer perceptron (MLP) module in knowledge utilization. This enables us to conduct targeted interventions, modulating the LLM’s reliance on internal versus external knowledge and providing deeper insight into the functionality of these modules. We conduct a systematic study, focusing on two main aspects. First, we adopt a diverse range of perspectives, using various methods to explore, hypothesize, and validate specific mechanisms. Second, while uncovering the mechanisms, we also propose methods to leverage them for building controllable RAG systems. From these systematic investigations, we derive the following key insights: (1) Knowledge streaming within the RAG framework can be identified into four distinct stages, regarding to knowledge utilization: knowledge refinement, knowledge elicitation, knowledge expression, and knowledge contestation. Such a stage division can be re-examined by saliency analysis. (2) The relevance degrees of retrieved passages direct the knowledge streaming in RAG. This relevance primarily affects the knowledge elicitation stage. This provides a fundamental explanation for how relevance discrepancies impact RAG’s knowledge utilization and performance. (3) We define a new metric, Knowledge Activation Probability Entropy (KAPE), to identify neurons associated with internal and external knowledge. By specifically deactivating these neurons, we successfully altered the model’s preference for the selection between two knowledge sources. (4) We explore the contributions of MHA and MLP modules to knowledge generation. Our findings validate deep-layer knowledge competition and reveal that MLP layers play a role in verifying knowledge accuracy. In summary, the contributions of this paper are as follows: • For the first time, we unveil the intrinsic knowledge utilization mechanisms of LLMs in RAG scenarios from two perspectives. At the macroscopic level, we examine the trends in