Rule-Based Explanations for Retrieval-Augmented LLM Systems
Joel Rorseth, Parke Godfrey, Lukasz Golab, Divesh Srivastava, Jarek Szlichta · 2025
Read Rule-Based Explanations for Retrieval-Augmented LLM Systems by Joel Rorseth, Parke Godfrey, Lukasz Golab, Divesh Srivastava, Jarek Szlichta (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22689.txt.
Introduction
Explaining why a retrieval augmented language system gave an answer matters when sources drive the output. Joel Rorseth, Parke Godfrey, Lukasz Golab, Divesh Srivastava, and Jarek Szlichta present in 2025 the first proposal to explain such systems with if then rules about which sources were retrieved.
A rule might say that when a ranking article is retrieved, the model ranks a given university first. Testing every source combination would be far too slow. Their optimizations adapt pruning ideas from frequent itemset mining to cut the search. Qualitative and quantitative experiments show the rules are useful and can be generated efficiently.
Catalog record
| CREATOR | Joel Rorseth, Parke Godfrey, Lukasz Golab, Divesh Srivastava, Jarek Szlichta |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-22689
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Rule-Based Explanations for Retrieval-Augmented LLM Systems Joel Rorseth, Parke Godfrey, Lukasz Golab, Divesh Srivastava, Jarek Szlichta Abstract If-then rules are widely used to explain machine learning models; e.g., “if employed = no, then loan application = rejected.” We present the first proposal to apply rules to explain the emerging class of large language models (LLMs) with retrieval-augmented generation (RAG). Since RAG enables LLM systems to incorporate retrieved information sources at inference time, rules linking the presence or absence of sources can explain output provenance; e.g., “if a Times Higher Education ranking article is retrieved, then the LLM ranks Oxford first.” To generate such rules, a brute force approach would probe the LLM with all source combinations and check if the presence or absence of any sources leads to the same output. We propose optimizations to speed up rule generation, inspired by Apriori-like pruning from frequent itemset mining but redefined within the scope of our novel problem. We conclude with qualitative and quantitative experiments demonstrating our solutions’ value and efficiency. keywords Explainability, Rule-Based Explanations, Large Language Models, Retrieval-Augmented Generation 1 Introduction Advances in large language models (LLMs) have yielded increasingly sophisticated artificial intelligence (AI) systems capable of accomplishing a wide variety of tasks. These advances are enabled not only by the increasing size and architectural complexity of the models, but also by innovations in how LLMs incorporate knowledge. One such prominent innovation is retrieval-augmented generation (RAG), by which the static, pre-trained knowledge of an LLM is augmented with dynamic, arbitrary knowledge retrieved at inference time. Although RAG enables more control over an LLM’s data, the provenance of information in the LLM’s response is further complicated as a result. The emerging field of explainable artificial intelligence (XAI) has been addressing issues related to understanding the behavior of complex models, including LLMs. Many XAI methods focus on the simpler problem of local explanation, which attempts to explain specific model predictions rather than general model behavior. One popular local XAI methodology is feature attribution , which explains a model output by scoring the relative importance of the input features. For RAG systems, context attribution can assign scores to each piece of information retrieved by an LLM system at inference time ( Cohen-Wang et al. 2024 ) . However, feature attribution is not actionable (e.g., to achieve recourse). Counterfactual explanations address this problem, explaining a model’s output by citing a minimal input perturbation that would induce an alternative output ( Wachter et al. 2017 ) . In RAG, a counterfactual explanation might reveal that the LLM would have generated a different answer if some piece of information were not provided or retrieved at runtime ( Rorseth et al. 2024 ) . However, there may be many minimal counterfactuals for a given prediction, which may overwhelm end users. Explanations using if-then rules mitigate these problems by summarizing patterns observed across a model’s predictions ( Ribeiro et al. 2018 ; Macha et al. 2022 ; Guidotti et al. 2019 ) , but have not yet been considered in the context of LLMs with RAG. We fill this research gap and present a rule framework for explaining the output of black-box RAG systems using rules that formulate input predicates based on the presence of RAG sources provided to the LLM. Let us illustrate using a healthcare example. Suppose that a clinic is using a RAG system connected to a medical database, and a medical research assistant working at the clinic asks the LLM, “What treatment is most effective for Long COVID fatigue?” Here, a rule could reveal that “Whenever ‘Document 123’ is included in the RAG sources, the LLM always recommends an unsafe Long COVID treatment.” (A rule’s antecedent could likewise be when a given document is omitted from the RAG sources.) This rule helps curators of the medical database flag documents such as Document 123 that, unbeknownst to them, advocate for unsafe treatments. In this case, Document 123 may advocate for a treatment that was thought to be effective until recently, while newer studies have shown it is both ineffective and unsafe for certain demographics. Our framework empowers developers, auditors, policy makers, and other end users to seek rules for any identifiable output condition, such as responses that are incorrect, contain misinformation, or reflect negative sentiment ( Han et al. 2024 ) . Our main contributions are as follows. 1. We propose a rule-based explanation formulation for RAG, the first of its kind. We offer two variants that predicate on the basis of retaining or omitting features (RAG sources). A rule explains the output returned by an LLM by identifying conditions that consistently hold in practice whenever certain features are retained or omitted. 2. We design efficient search algorithms to mine rule-based explanations, exploiting intuitions of rule validity to prune the search space and enable early termination of the search. 3. We conduct an experimental evaluation of our explanation search algorithms, presenting case studies showing the value of rule-based explanations of RAG systems for question answering, and highlighting the efficiency improvements due to our pruning techniques. The rest of the paper is structured as follows: Section 2 reviews related work; Section 3 introduces our rule formulation; Section 4 illustrates our rule mining algorithms; Section 5 presents an experimental evaluation; and Section 6 concludes with future research directions. 2 Related Work In the XAI literature, many methods have been proposed to explain individual model predictions, collectively known as local explanations. Many local methods adopt an approach known as feature attribution, exemplified by methods such as LIME ( Ribeiro et al. 2016 ) and SHAP ( Lundberg and Lee 2017 ) , which explain a model by scoring input features according to their relative influence on the model’s output. Though intuitive, these explanations are not directly actionable, and have been shown to be potentially unreliable, misleading, and manipulable ( Kindermans et al. 2019 ; Slack et al. 2020 ; Hooshyar and Yang 2024 ) . In contrast, counterfactual explanations explain a model by perturbing the input features in a minimal way to change the original output ( Wachter et al. 2017 ; Verma et al. 2024 ) ; e.g., a declined loan application would have been approved if the applicant’s income were 20 percent higher. Counterfactual perturbations aim to produce actionable insights that facilitate recourse, but many valid counterfactuals may exist, potentially contradicting each other and overwhelming users. Some methods therefore seek to sample diverse counterfactual perturbations or prioritize those deemed more feasible ( Mothilal et al. 2020 ; Tsirtsis and Gomez Rodriguez 2020 ) . At the cost of increased computational complexity, rules mitigate the above problems by summarizing patterns observed across a model’s predictions ( Ribeiro et al. 2018 ; Macha et al. 2022 ; Guidotti et al. 2019 ; Rudin and Shaposhnik 2023 ) ; e.g., “if employed = no, then loan application = rejected,” regardless of the values of the other features. Recent works have established that rules essentially subsume counterfactuals, forming a duality, since both make assertions over a model’s input features ( Geng et al. 2022 ; Guidotti et al. 2024 ) . Rule-based explanations have been defined as conjunctions of predicates over the features of an input, for which any matching input