Know Or Not: a library for evaluating out-of-knowledge base robustness
Jessica Foo, Pradyumna Shyama Prasad, Shaun Khoo · 2025
Read Know Or Not: a library for evaluating out-of-knowledge base robustness by Jessica Foo, Pradyumna Shyama Prasad, Shaun Khoo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13545.txt.
Introduction
Jessica Foo, Pradyumna Shyama Prasad, and Shaun Khoo build a library for testing whether a chatbot knows when it does not know. Systems that fetch documents before answering still invent replies when the answer is outside the fetched base. In high-stakes uses, silence beats invention.
Their library, called knowornot, lets users build custom test sets and pipelines for this abstention skill. It needs no hand-written gold answers, which makes tests cheap to build. Users can shape the tests to their own knowledge base and risk level. The method grades whether models refuse out-of-base questions as they should. Such grading is a guardrail for legal, medical, and finance bots. The team releases the tool as open source.
Catalog record
| CREATOR | Jessica Foo, Pradyumna Shyama Prasad, Shaun Khoo |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 82 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-13545
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Know Or Not: a library for evaluating out-of-knowledge base robustness Jessica Foo, Pradyumna Shyama Prasad, Shaun Khoo Abstract While the capabilities of large language models (LLMs) have progressed significantly, their use in high-stakes applications have been limited due to risks of hallucination. One key approach in reducing hallucination is retrieval-augmented generation (RAG), but even in such setups, LLMs may still hallucinate when presented with questions outside of the knowledge base. Such behavior is unacceptable in high-stake applications where LLMs are expected to abstain from answering queries it does not have sufficient context on. In this work, we present a novel methodology for systematically evaluating out-of-knowledge base (OOKB) robustness of LLMs (whether LLMs know or do not know) in the RAG setting, without the need for manual annotation of gold standard answers. We implement our methodology in knowornot , an open-source library that enables users to develop their own customized evaluation data and pipelines for OOKB robustness. knowornot comprises four main features. Firstly, it provides a unified, high-level API that streamlines the process of setting up and running robustness benchmarks. Secondly, its modular architecture emphasizes extensibility and flexibility, allowing users to easily integrate their own LLM clients and RAG settings. Thirdly, its rigorous data modeling design ensures experiment reproducibility, reliability and traceability. Lastly, it implements a comprehensive suite of tools for users to customize their pipelines. We demonstrate the utility of knowornot by developing a challenging benchmark, PolicyBench, which spans four Question-Answer (QA) chatbots on government policies, and analyze its OOKB robustness. The source code of knowornot is available here and PolicyBench is available here . 1 Introduction Large language models (LLMs) are prone to hallucination ( Huang et al., 2025a ) . Retrieval-augmented generation (RAG) ( Lewis et al., 2020 ) has emerged as a key approach to reduce hallucination by leveraging a knowledge base to retrieve relevant context and improve the accuracy of generations. Nonetheless, in real-world deployments of Question-Answer (QA) chatbots, user queries can fall out of scope of the knowledge base. In high-stakes applications where the risk and cost of providing an inaccurate answer is high, LLMs are expected to refrain from relying on its parametric knowledge, and instead abstain from answering queries it does not have sufficient context on ( Anthropic, 2025 ) . In practice, LLMs do persist in answering despite being instructed to only do so when certain. As such, it is necessary to evaluate LLMs’ robustness to out-of-knowledge base (OOKB) queries in order to guide risk management for high-stakes applications. Since hallucinations and OOKB robustness are particularly domain-specific, arising in part from the extent to which the requested information is encoded in the LLM’s parametric knowledge, evaluations have largely remained labor-intensive. A standard practice today is to synthetically generate questions from a given knowledge base, prompt the LLMs for answers, and then require a human to verify whether the answers are supported by the context. However, such a manual process is not scalable. Instead, we require frameworks that are customizable, automated, and reliable to perform robust evaluations. We first introduce a novel methodology for systematically evaluating OOKB robustness of QA chatbots built with LLMs and RAG, without the need for manual annotation of gold standard answers. Our methodology involves constructing an evaluation dataset of QA pairs from a given knowledge base, ensuring that the dataset is grounded, diverse, and informationally distinct. The QA pairs are then systematically removed in a controlled leave-one-out (LOO) experimental set up to ascertain whether an LLM persists in responding despite not having relevant contextual information, allowing us to produce an overall estimate of the LLM’s OOKB robustness. Our second contribution is the development of an open-source library knowornot , which implements the aforementioned methodology. The library enables users to provide source documents to develop their own customized evaluation data and pipelines. The library consists of four main design features. (1) Unified, high-level API streamlining the process of setting up and running robustness evaluations, requiring users to instantiate a single knowornot object containing the necessary methods for executing the pipelines. (2) Modular architecture emphasizing extensibility and flexibility, allowing users to easily integrate their own LLM clients, RAG settings, and evaluation criteria. (3) Rigorous data modeling design to ensure experiment reproducibility, reliability and traceability, including the storage of intermediate experimental outputs. (4) Comprehensive suite of tools for users to customize their pipelines and execute rigorous empirical experiments ablating different models and RAG settings, as well as human validation of automated evaluations. Our third contribution is a novel benchmark, PolicyBench 1 1 1 The dataset is publicly available here and the accompanying code to generate the dataset is available here . , comprising questions from four QA chatbots on Singapore government policies, which can be used to assess OOKB robustness in similar settings where information accuracy is paramount and LLMs should abstain when contextual information is missing. Our empirical experiments with PolicyBench demonstrate ease of using knowornot to build OOKB evaluation pipelines and experiments. 2 Methodology Our methodology focuses on an LLM’s adherence to the provided context and its ability to abstain from answering when the necessary information is missing. This section details the process of (1) generating benchmarks from any text-based knowledge base, (2) designing experiment scenarios to probe LLM behaviors, and (3) evaluating the outcomes using a combination of automated and human-validated techniques. We aim to provide a general framework that enables practitioners to rigorously benchmark the contextual reliability of different LLMs, prompts, and retrieval strategies. 2.1 Knowledge base formalization and test case generation First, we transform unstructured source text into a formalized Knowledge Base (KB) and generate Question-Answer (QA) pairs that are verifiably grounded in this KB. This process ensures that all test cases used in the benchmark originate from, and are answerable by, the original source material. 2.1.1 Atomic fact extraction from source text To formalize the setup, for given source document(s) D , the first step is to decompose the content into granular, verifiable units of information. We term these units "atomic facts". We generate a list of atomic facts F_{D}=[{F_{1},F_{2},...,F_{N}}] through an LLM-assisted process: 1. Sentence segmentation: The input text is segmented into individual sentences using standard natural language processing techniques (i.e., NLTK’s sentence tokenizer). 2. Fact granularization: Each sentence is processed by an LLM (prompt in Appendix A.1.1 ) which extracts one or more self-contained, modular facts from the sentence. 2.1.2 Generation and curation of grounded, diverse and informationally distinct QA pairs Once the KB is formalized as a collection of atomic facts F_{D} , the facts are used to generate an initial set of QA pairs. For each atomic fact, an LLM is instructed (prompt in Appendix A.1.2 ) to formulate (1) a single, objective, and relevant test question where the answer can be directly answered using the given atomic fact, (2) the corresponding correct answer, derived solely from that same atomic fact. The output is a list of QA pairs, (Q_{i},A_{i}) derived from F_{i} , that may contain duplicative or semantically similar questions, as atomic facts may still reference closely related concepts. Hence, we curate this list of QA pairs into a set of diverse and informationally distinct test cases, such that \forall i\neq j,\operatorname{similarity}[(Q_{i},A_{i}),(Q_{j},A_{j})]\approx 0 . Importantly, our methodology aims to