Human Decision-Making with Persuasive and Narrative LLM Explanations

Laura R. Marusich, Mary Grace Kozuch Dhooghe, Jonathan Z. Bakdash, Murat Kantarcioglu · 2026

Read Human Decision-Making with Persuasive and Narrative LLM Explanations by Laura R. Marusich, Mary Grace Kozuch Dhooghe, Jonathan Z. Bakdash, Murat Kantarcioglu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-23867.txt.

Introduction

Laura Marusich and colleagues run a large behavioral experiment. They test whether persuasive story-like explanations from language models actually help people decide better. Varying explanation persuasiveness around AI predictions for classification tasks, they find accuracy barely moves beyond a bare prediction. That matches classic explainable AI results. Stories do raise reliance on the AI, for right and wrong predictions alike. The most persuasive stories may slow responses and blur the line between correct and incorrect advice. That tradeoff is worth weighing.

Catalog record

CREATORLaura R. Marusich, Mary Grace Kozuch Dhooghe, Jonathan Z. Bakdash, Murat Kantarcioglu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56263
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.23867

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Human Decision-Making with Persuasive and Narrative LLM Explanations Laura R. Marusich, Mary Grace Kozuch Dhooghe, Jonathan Z. Bakdash, Murat Kantarcioglu Abstract Large language models (LLMs) have the potential to aid and improve human decision-making in classification tasks, not only by providing fairly accurate predictions, but also in their ability to generate cogent narrative explanations of those predictions. Prior work has demonstrated that people generally find AI narrative explanations to be understandable, trustworthy, and convincing for changing beliefs and opinions; however, less is known about the impact of narrative explanations on objective human decision-making performance. Here we conduct a large-scale human behavioral experiment to evaluate decision-making performance with LLM-generated narrative explanations of varying persuasiveness. We found the degree of persuasiveness, or lack thereof, for LLM-based explanations did not meaningfully impact decision accuracy over a simple AI prediction alone, in agreement with typical results with explainable AI based on feature importance. We found evidence that narratives increased reliance on AI, but both when the AI prediction was correct and incorrect. Exploratory analyses also indicated that the more persuasive narratives may have had a detrimental effect on decision response times and the ability to discriminate between a correct and incorrect AI prediction. Overall, this work indicates that including narrative explanations with AI predictions may involve tradeoffs for decision-making performance, and more work is needed to determine how and when narrative explanations impact human decision-making. 1 Introduction Joint human/AI decision-making can potentially combine the strengths of AI with those of humans in a way that leads to better performance than what can be attained by either alone. There is substantial work exploring the factors that increase human trust in and reliance upon AI predictions, and to a lesser extent, factors that improve the accuracy performance of joint human/AI decisions. Recent advancements in large language models (LLMs) make interactions with AI systems increasingly intuitive and understandable to human users. For example, LLMs are very well-suited for generating narrative text explanations for AI predictions (compared to lists of decision rules, model visualizations, e.g.). They can also be prompted to generate highly persuasive arguments that can shift opinions in human users ( 43 ) . However, there is limited prior work on the the impact of persuasive and narrative AI explanations on objective human decision-making performance. For example, prior work with narrative explanations found increased human understanding of AI model reasoning yet no change in subjective human decisions 32 . Human decision-making using rule/feature-based explainable AI has been widely evaluated, and it typically does not improve objective decision-making performance 44 ; 3 ; although results may be task-dependent 30 . Given the growing use of narrative AI explanations, it is critical to evaluate how the persuasiveness, or lack thereof, for LLM narrative explanations impacts human reliance on AI and also objective decision-making performance. This topic reflects a key gap in the literature, and in the work described here, we are among the first to evaluate objective human decision-making with persuasive and narrative explainable AI. Our results show narrative explainable AI, regardless of persuasiveness, did not meaningfully change human decision-making accuracy over a simple AI prediction alone. However, we did find evidence for increased reliance on AI with narrative explanations for both correct and incorrect AI predictions. In this paper, we first provide background on narrative explanations for explainable AI, persuasion and the growing body of work on the persuasive capabilities of LLMs, and the more limited research assessing joint human/AI decision-making . Section 3 describes our hypotheses and experimental design, including the datasets and model prompts we used to generate our experimental stimuli. We report the findings from the pre-registered, large sample size behavioral experiment comparing human decision making accuracy with only AI predictions vs. AI predictions with narrative explanations of varying persuasiveness. Finally, we conclude by discussing the implications of our results, with limitations and recommendations for future work. 2 Background and related work 2.1 Narrative AI explanations 23 introduce narrative explanations as a framework in which AI explanations are tailored to the target audience through adaptive content, presentation style, and user feedback that can be used to iteratively improve the system. Explingo, introduced and evaluated by 50 , employs two LLMs: a narrator, which generates narrative explanations of SHAP outputs, and a grader, which evaluates these narratives along the dimensions of accuracy, completeness, fluency, and conciseness. Their results show that LLMs can produce high-quality narrative explanations, performing particularly well in accuracy and completeness, although sometimes with reduced conciseness. 32 create XAIStories using LLM narratives to improve user understanding of traditional AI explanation techniques. They find that for SHAP and CF explanations, general users find the narratives more convincing than the explanations alone in 93% and 90% of cases, respectively. Furthermore, when shown XAIStories based on SHAP instead of raw SHAP outputs, users’ objective understanding of the AI model’s reasoning tended to greatly improve, but this did not impact whether the user and AI (subjective) decisions were in agreement. 2.2 Persuasion Social scientists have been interested in questions of persuasion for nearly a century, and it continues to be an area of active study. There are a number of widely accepted definitions and models of persuasion (see, for example, 11 ; 37 ; 39 ; 28 ; 19 ; 10 ). Persuasion is distinct from other types of influence in that it is intentional, not coercive, and involves symbolic action - typically language ( 20 ) . In fields such as psychology and political science, persuasion is a form of communication that intentionally aims to change or influence people’s beliefs, attitudes, and/or behaviors ( 11 ) . In considering specifically language-based persuasion, researchers have detailed a number of features of communication that affect the persuasiveness of the message (for an overview, see 45 ). These features can be categorized into aspects of the content , the structure , and the style of the communication. Applying these message features to manipulate text persuasiveness is fairly intuitive for narrative-based explanations; less so for rule/feature-based explanation types. This presents an interesting opportunity to assess how narrative explanations of varying persuasiveness can affect human decision-makers’ performance. It should be noted, however, that it can be difficult to produce text samples that differ purely on the dimension of persuasiveness. For example, the use of exaggerated or flowery language is a common persuasion technique ( 11 ) widely used in U.S. advertisements ( 1 ; 4 ) and often represents a natural confound. In the context of LLMs, persuasive ability has recently attracted significant attention, generating both excitement and concern. LLMs can match the effectiveness of human persuasion, although it depends on the content, structure, and style of communication 24 . 42 survey the literature on factors affecting the persuasive capabilities of LLMs across literature, finding that multi-shot prompting is generally more effective than single-shot ( 13 ; 18 ) , although increased engagement with static text can be similarly effective 2 . This is further supported in recent work showing that in a multi-shot scenario, ChatGPT-4 is more persuasive than a human ( 43 ) . Personalization techniques can often increase persuasiveness ( 35 ; 34 ; 47 ; 42 ) , although 22 find that microtargeted messaging is not more

Read the complete text