Do Retrieval Augmented Language Models Know When They Don’t Know?
Youchao Zhou, Heyan Huang, Yicheng Liu, Rui Dai, Xinglin Wang, Xingchen Zhang, Shumin Shi, Yang Deng · 2025
Read Do Retrieval Augmented Language Models Know When They Don’t Know? by Youchao Zhou, Heyan Huang, Yicheng Liu, Rui Dai, Xinglin Wang, Xingchen Zhang, Shumin Shi, Yang Deng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-01476.txt.
Introduction
Language models with retrieval can look facts up, yet they still sometimes answer when they should admit ignorance. Youchao Zhou, with seven co-authors, asks whether retrieval-augmented models know when they do not know. Such models should refuse questions their knowledge, internal or retrieved, cannot support, but that refusal skill had not been studied closely.
The team tests calibration across different states of internal and retrieved knowledge. Against expectations, when all retrieved documents were irrelevant, the models still tended to refuse some questions they could have answered correctly on their own. That over-refusal pattern raises a second question about how refusal behavior aligns with calibration quality. Zhou and the co-authors present the study as a probe of a blind spot in hallucination research, where retrieval and refusal have mostly been judged apart.
Catalog record
| CREATOR | Youchao Zhou, Heyan Huang, Yicheng Liu, Rui Dai, Xinglin Wang, Xingchen Zhang, Shumin Shi, Yang Deng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-01476
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Do Retrieval Augmented Language Models Know When They Don’t Know? Youchao Zhou, Heyan Huang, Yicheng Liu, Rui Dai, Xinglin Wang, Xingchen Zhang, Shumin Shi, Yang Deng Abstract Existing Large Language Models (LLMs) occasionally generate plausible yet factually incorrect responses, known as hallucinations. Researchers are primarily using two approaches to mitigate hallucinations, namely Retrieval Augmented Language Models (RALMs) and refusal post-training. However, current research predominantly emphasizes their individual effectiveness while overlooking the evaluation of the refusal capability of RALMs. In this study, we ask the fundamental question: Do RALMs know when they don’t know? Specifically, we ask three questions. First, are RALMs well-calibrated regarding different internal and external knowledge states? We examine the influence of various factors. Contrary to expectations, we find that LLMs exhibit significant over-refusal behavior. Then, how does refusal post-training affect the over-refusal issue? We investigate the Refusal-aware Instruction Tuning and In-Context Fine-tuning methods. Our results show that the over-refusal problem is mitigated by In-context fine-tuning. but magnified by R-tuning. However, we also find that the refusal ability may conflict with the quality of the answer. Finally, we develop a simple yet effective refusal method for refusal post-trained models to improve their overall answer quality in terms of refusal and correct answers. Our study provides a more comprehensive understanding of the influence of important factors on RALM systems. Introduction Existing Large Language Models (LLMs) have demonstrated remarkable performance across various challenging tasks. However, they occasionally generate plausible yet factually incorrect responses, a phenomenon commonly known as hallucination ( Lewis et al. 2020 ; Huang et al. 2025 ) . Prior research primarily addresses the hallucination issue in LLMs using two approaches: retrieval-augmented generation (RAG) ( Lewis et al. 2020 ; Ram et al. 2023 ) and refusal post-training ( Zhang et al. 2024 ; Zhu et al. 2025 ) . RAG leverages external knowledge sources to provide contextual references, allowing retrieval-augmented language models (RALMs) to respond accurately to queries outside their internal knowledge. In contrast, refusal post-training aims to enhance the model’s self-awareness, enabling it to proactively abstain from responding when uncertain. Figure 1: An example of the RALMs knowledge boundary and the corresponding answer correctness. Although both methods are widely adopted, current research predominantly emphasizes their individual effectiveness while overlooking the evaluation of refusal capability within RALMs. Specifically, refusal post-training techniques typically focus on boosting the LLM’s confidence in its internal knowledge and its ability to articulate internal uncertainty. However, given that LLMs are sensitive to the quality and relevance of retrieval contexts ( Park and Lee 2024 ; Cuconasu et al. 2024 ) , a refusal-trained model might mishandle unreliable external information and become uncertain even when it internally possesses correct knowledge. As shown in Figure 1 , we can divide the knowledge state of RALMs into four quadrants based on prior research ( Sun et al. 2025 ) . When RALMs face the question “ When does the 2022 Olympic Winter Game end? ”, they can answer it correctly. However, when retrieved contexts contain misinformation, RALMs may confuse and refuse to answer. Currently, we don’t know when this phenomenon occurs. To address this gap, we investigate how retrieved context influences the refusal capabilities of LLMs. Specifically, in this work, as ask three critical research questions (RQs). First, are retrieval-augmented language models well-calibrated regarding different internal and external knowledge states? (RQ1) Ideally, if RALMs are well-calibrated (know when they don’t known), they can refuse to answer or we can abstain question properly. We categorize knowledge states as shown in Figure 1 and apply uncertainty estimation techniques to quantify the knowledge confidence of RALMs. Contrary to expectations, we find that LLMs exhibit significant over-refusal behavior and decline in confidence, particularly when confronted with exclusively irrelevant contexts. However, we also find models demonstrate improved calibration when supportive context exists within negative context. This indicates a vulnerability to contextual distractions, impairing their ability to effectively distinguish between internal and external knowledge states. As recent studies ( Zhang et al. 2025 ; Zhu et al. 2025 ) primarily elicit the refusal capability of large language models through post-training. Then, given the over-refusal tendency observed in RALMs, we further examine the impact of refusal post-training on this behavior. Accordingly, we pose the second research question: how does refusal post-training affect the over-refusal issue? (RQ2) We implement two instruction tuning based methods: Refusal-Aware Instruction Tuning (R-tuning) ( Zhang et al. 2024 ) and In-Context Fine-Tuning (ICFT) ( Lee, Lin, and Tan 2025 ; Zhu, Panigrahi, and Arora 2025 ) . To provide a comprehensive evaluation, we assess these models using a broader set of metrics, including context utility and refusal confidence. Our results show the over-refusal problem is mitigated by ICFT while magnified by R-tuning . In addition, we also find that the refusal ability conflicts with the answer correctness when positive context exists, due to a degradation of context utility. This limits the reliability and practicality of existing refusal post-training methods for RALMs. Lastly, considering the difficulty to balance proper refusal and context utilization based solely on LLMs themselves, we propose the third research question: can we combine the refusal-aware RALMs and confidence-based answer abstention to mitigate over-refusal? (RQ3) According to previous findings, we can leverage confidence and its variation to determine the knowledge state of RALMs. Then we can choose to answer a question with or without context, or further abstain. Our contributions are as follow: 1) We investigate the confidence calibration of RALMs. We comprehensively examine important factors and identify the over-refusal problems. 2) We explore whether the current refusal instruction tuning will intensify LLMs’ over-refusal and give further explanations. 3) We design a better refusal technique based for RALMs based on the above findings. Related Works Knowledge Boundary of LLMs. Identifying the knowledge boundary helps to recognize the limitations of knowledge. It is also referred to as “knowing what you don’t know,” ( Yin et al. 2023 ; Deng et al. 2024 ) a crucial aspect in determining their practical applicability. Li et al. 2024 formally category knowledge boundary according to prompt and model sensitivity. Gekhman et al. 2024 divide the knowledge type according to hallucination variation after knowledge finetuning. However, they mainly focus on the LLM’s internal knowledge. Hallucination happens when users’ requests are out of LLMs’ knowledge boundary ( Huang et al. 2025 ) . The primary approach to mitigate hallucination includes retrieval-augmented generation (RAG). RAG ( Lewis et al. 2020 ) is a convenient method for inference stage where the retrieved context fills the knowledge gap. recent RAG including adapt retrieval model for LLMs ( Xu, Shi, and Choi 2024 ) , external knowledge injection ( Ovadia et al. 2024 ) and dynamic RAG ( Su et al. 2024 ) . Dynamic RAG leverage uncertainty estimation to determine when to retrieve. However, they ignore the refusal ability of RALMs and assume RALMs are well calibrated. The difference is they focusing on the dynamic generation process while we focus on the static knowledge state. Refusal Method of LLMs. Refusal behavior are mainly researched in post-training stages ( Wen et al. 2025 ) . It mainly considers the instruction tuning ( Kapoor et al. 2024 ) and refusal alignments ( Cheng et al. 2024 ; Sun et