Evaluating the False Trust Engendered by LLM Explanations
Vardhan Palod, Upasana Biswas, Subbarao Kambhampati · 2026
Read Evaluating the False Trust Engendered by LLM Explanations by Vardhan Palod, Upasana Biswas, Subbarao Kambhampati (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10930.txt.
Introduction
Vardhan Palod, Upasana Biswas, and Subbarao Kambhampati test whether language-model explanations help users spot wrong answers or merely talk them into false trust. Reasoning traces, their summaries, and after-the-fact explanations all resemble provenance, yet the evidence shows traces may not mirror what the model actually computed.
In a between-subject study where users could not check answers themselves, the team compared explanation types and added a contrastive setting that argues both for and against the answer. Reasoning traces and post-hoc explanations persuaded users without informing them, inflating trust in wrong answers. Designers of critical AI tools learn that explanations must be judged by user discrimination, not by how convincing they read.
Catalog record
| CREATOR | Vardhan Palod, Upasana Biswas, Subbarao Kambhampati |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59569 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.10930
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluating the False Trust Engendered by LLM Explanations Vardhan Palod, Upasana Biswas, Subbarao Kambhampati Abstract Large Language Models (LLMs) and Large Reasoning Models (LRMs) are increasingly used for critical tasks, yet they provide no guarantees about the correctness of their solutions. Users must decide whether to trust the model’s answer, aided by reasoning traces, their summaries, or post-hoc generated explanations. These reasoning traces, despite evidence that they are neither faithful representations of the model’s computations nor necessarily semantically meaningful, are often interpreted as provenance explanations. It is unclear whether explanations or reasoning traces help users identify when the AI is incorrect, or whether they simply persuade users to trust the AI regardless. In this paper, we take a user-centered approach and develop an evaluation protocol to study how different explanation types affect users’ ability to judge the correctness of AI-generated answers and engender false trust in the users. We conduct a between-subject user study, simulating a setting where users do not have the means to verify the solution and analyze the false trust engendered by commonly used LLM explanations — reasoning traces, their summaries and post-hoc explanations. We also test a contrastive dual explanation setting where we present arguments for and against the AI’s answer. We find that reasoning traces and post-hoc explanations are persuasive but not informative: they increase user acceptance of LLM predictions regardless of their correctness. In contrast, dual explanation is the only condition that genuinely improves users’ ability to distinguish correct from incorrect AI outputs. 1 Introduction With the advent of Large Language models (LLMs), the field of AI has been revolutionized. LLMs have demonstrated impressive capabilities across a wide array of tasks, from planning and reasoning to instruction following and story writing [ 1 ] . Unlike traditional AI systems, these generative language models are broad and general purpose models, leading to their rapid adoption by lay users. However, this adoption comes with a caveat. Large Language Models confidently answer any question put to them, regardless of whether the answer is correct. As these systems are increasingly used for critical decisions in healthcare, education, and legal domains, two critical questions arise: how do we develop methods to engender appropriate rather than uncritical trust in end users, and how do we evaluate the effectiveness of such methods. Traditional Explainable AI (xAI) techniques were designed for deep and narrow, task-specific AI systems that were experts in one domain [ 2 , 3 ] . Since they did not share the same internal representations as humans, these systems were reliable but opaque to users. The goal of developing explanation techniques was to increase user trust by explaining the system’s decision-making process in terms which the users can understand [ 4 , 5 ] . The xAI challenge was “how do we get users to trust a system which doesn’t speak the end-user’s language ?”. LLMs change the setting quite a bit. They communicate in natural language and readily provide a palatable explanation of their generated answers, while not providing any guarantees of correctness or faithfulness [ 6 , 7 , 8 , 9 ] . We observe this in a new crop of language models called Large Reasoning Models (LRMs) which generate a sequence of intermediate tokens (colloquially or perhaps fancifully called a “Chain of Thought” or “reasoning trace”) before producing an answer sequence [ 10 ] . These reasoning traces have often been interpreted as the intermediate steps which the model took to arrive at the final answer, leading the community to treat them as provenance explanations [ 11 ] . However, some researchers have cautioned against interpreting reasoning traces as faithful representations of the model’s internal computations [ 6 ] , and have demonstrated that these traces do not need to have any semantic meaning or end-user interpretability to produce correct answers [ 12 , 13 , 14 ] . The frontier model makers– OpenAI [ 15 ] , Google [ 16 ] , Anthropic [ 17 ] - do not show the actual reasoning traces which the models produced rather expose the users to what seems like a summarized version of the reasoning traces. GPT-OSS models [ 18 ] show commentary tokens which are interpretable for the end user, and are admittedly distinct from the actual intermediate tokens. The commentary tokens might very well have been generated post-hoc to explain the LLM’s predicted answer. These are the different kinds of explanations - reasoning traces, their summaries or post hoc explanations - which users are exposed along with the predicted answer when they interact with these systems. With the widespread adoption of LLMs, especially in safety critical domains, the need to study the science of appropriate trust - trusting correct answers and distrusting incorrect answers - when interacting with a noisy system is paramount. Typical studies are often anecdotal and fail to disentangle the connection between the user’s ability to verify the solution and effectiveness of the provided explanation [ 19 ] . If the user can independently verify the correctness of the predicted answer–either through their own knowledge or with access to external tools, they are not dependent on the explanation to aid them and thus, they often introduce bias in evaluating the explanations, may even choosing to ignore them. In such studies [ 20 ] , it is extremely difficult to convincingly isolate the effects of the provided explanations on user trust. To address this gap, we propose an evaluation protocol for measuring how LLM-generated explanations affect user trust. The protocol simulates a setting where users do not have the ability or means to independently verify the LLM generated answer. For each problem, we expose users to different types of explanations alongside AI-generated answers and the participants need to judge whether the AI’s predicted answer is correct or incorrect. We instantiate this protocol with a between-subjects user study on challenging math, physics, and chemistry problems from the JEE-Bench dataset [ 21 ] . Our study reveals that reasoning traces and post-hoc explanations significantly increased false trust engendered in users, as compared to our baseline where no explanation was provided. In an attempt to mitigate this critical issue, we evaluate a post hoc explanation generation paradigm - dual explanations, similar to contrastive explanation generation in [ 22 ] , where the user is exposed to the merits and demerits of the LLM generated answer. Overall, we find that the dual explanations have the lowest rate of engendering false trust. We also observe that reasoning traces and dual explanations achieve the highest rates of appropriate trust, though for fundamentally different reasons. Reasoning traces tend to increase user confidence in LLM-generated answers overall, leading to high accuracy in identifying correct answers but poor performance in detecting incorrect ones. In contrast, dual explanations exhibits a more balanced effect: users maintain relatively high accuracy in identifying both correct and incorrect answers. Our protocol also enables us to evaluate how the familiarity of the user with a problem instance changes the engendered trust rates. We found that when a user is familiar with a problem instance, reasoning trace or their summaries seem to be most helpful but when users are not confident in their ability to solve the problem, dual