Classifying Unreliable Narrators with Large Language Models

Anneliese Brei, Katharine Henry, Abhisheik Sharma, Shashank Srivastava, Snigdha Chaturvedi · 2025

Read Classifying Unreliable Narrators with Large Language Models by Anneliese Brei, Katharine Henry, Abhisheik Sharma, Shashank Srivastava, Snigdha Chaturvedi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-10231.txt.

Introduction

Anneliese Brei, Katharine Henry, Abhisheik Sharma, Shashank Srivastava, and Snigdha Chaturvedi test whether language models can spot unreliable narrators, in a two thousand twenty-five paper mixing literary theory and machine learning. An unreliable narrator unintentionally misrepresents events in a first-person account.

Drawing on narratology, the authors define several unreliability types and build TUNa, an expert-annotated dataset spanning blog posts, forum posts, hotel reviews, and literature. Classification tasks cover unreliability within a narration, between narrators, and across texts. Popular open and proprietary models struggle, though few-shot prompting, fine-tuning, and curriculum learning all show promise. The dataset and code are released for future work.

Catalog record

CREATORAnneliese Brei, Katharine Henry, Abhisheik Sharma, Shashank Srivastava, Snigdha Chaturvedi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE92 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-10231

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Classifying Unreliable Narrators with Large Language Models Anneliese Brei, Katharine Henry, Abhisheik Sharma, Shashank Srivastava, Snigdha Chaturvedi Abstract Often when we interact with a first-person account of events, we consider whether or not the narrator, the primary speaker of the text, is reliable. In this paper, we propose using computational methods to identify unreliable narrators, i.e. those who unintentionally misrepresent information. Borrowing literary theory from narratology to define different types of unreliable narrators based on a variety of textual phenomena, we present TUNa , a human-annotated dataset of narratives from multiple domains, including blog posts, subreddit posts, hotel reviews, and works of literature. We define classification tasks for intra-narrational, inter-narrational, and inter-textual unreliabilities and analyze the performance of popular open-weight and proprietary LLMs for each. We propose learning from literature to perform unreliable narrator classification on real-world text data. To this end, we experiment with few-shot, fine-tuning, and curriculum learning settings. Our results show that this task is very challenging, and there is potential for using LLMs to identify unreliable narrators. We release our expert-annotated dataset and code at https://github.com/adbrei/unreliable-narrators and invite future research in this area. 1 Introduction Imagine that you are on social media warning your friends about a recent shopping experience, and before submitting the post, you wonder if the presentation of your writing undermines your credibility. In another window, you are writing a cover letter. You recount a critical learning experience from your past job and wonder if your present voice sounds reliable to the reader. In the next room, your family is discussing the debate transcript between political candidates. Your sister thinks one of the candidates speaks like a villain from a novel she read, and your family differs on how reliable the candidate actually is. For each of these situations, it would be useful to have an automatic tool that identifies unreliability. Figure 1: Real-world text with first-person narrators, such as the narrative shown (left), can be analyzed to determine the unreliability of the narrator. We separately classify three types of unreliability (right): intra-narrational, inter-narrational, and inter-textual. Readers of personal accounts, such as reviews, online comments, cover letters, and college application essays, often implicitly question the reliability of the narrator: Can I trust how this narrator has perceived and is describing the event? Meanwhile, writers who wish to defend their points are concerned about how they textualize their ideas: Am I sharing information in a reliable way? Answering such questions is critical for the safe transmission of information Nünning 2015 . However, answering these questions is not a simple task. That is because unreliability cues are often subtle and context-dependent Hansen 2007 . They might be scattered across the text or involve a deeper understanding beyond what is explicitly stated. Sometimes it is necessary to draw abstract inferences about the emotional and mental state of the narrator. Also, a text might have many readers, some of whom focus on different aspects of these cues. From a writer’s perspective, it is important to pay attention to all of these cues to ensure the writing sounds reliable to all readers. Narratology has explored these questions by attempting to define the unreliable narrator , a first-person speaker who unintentionally describes situations misleadingly Booth 1961 . Hansen 2007 considers leading definitions and observes “the unreliable narrator is a concept covering very diverse textual phenomena” and accordingly proposes a taxonomy containing different forms of unreliable narrators with “conceptual distinction.” These forms include intra-narrational, inter-narrational, and inter-textual unreliability. The first form, intra-narrational unreliability is the classical definition that focuses on the presence of verbal tics (textual cues that indicate uncertainty). The left half of Figure 1 shows an example: an excerpt from a blog post where the writer narrates an experience with another person, Dorian, at a bar. The text in orange font indicates content that is narrated in an intra-narrationally unreliable manner because the narrator admits having trouble remembering details due to being inebriated. Consequently their narration is possibly unreliable. The second form, inter-narrational unreliability occurs when a secondary voice presents a contrasting version of events. For example, in Figure 1 , highlighted in green, Dorian does not agree with the narrator regarding the whereabouts of a notebook. Such a contradiction indicates that either the narrator or Dorian must be wrong and raises reader’s doubts about the reliability of the narrator. The third form, inter-textual unreliability involves pattern-matching the narrator with established unreliable character tropes Riggan Jr 1978 . In Figure 1 , highlighted in purple, the reader questions the narrator’s reliability because they seem cunning as they bribe the bartender (fitting the trope of pícaro ). More detailed definitions of each type of unreliability are outlined in Section 3 . Identifying these three forms of unreliable narrators requires picking up on subtle cues that range from specific lexical choices (e.g., a direct statement such as “it’s hard to remember”) to increasingly abstract inferences (e.g., drawing inferences through statements and actions that a narrator has cunning and self-interest). These forms may contain overlapping characteristics; however, they are classified and determined separately. Hence, a narrator might be unreliable in one of these forms but not another. It is valuable to analyze narrators in this way because it provides in-depth views of the narrator from lexical to abstract contextual levels. Determining narrator unreliability ultimately considers all three forms since together they provide a more complete picture. In this work, we borrow these definitions from narratology and introduce the task of automatically identifying three forms of unreliable narrators. We pose this problem as a set of binary/multi-class classifications corresponding to the three types of unreliability (shown in the right of Figure 1 ). We propose that these ideas from the theoretical field of narratology can be used more broadly to identify unreliability across diverse real-world domains. We observe that as of date there has been no work on analyzing narrator unreliability with automatic methods, and there are no available resources or labeled datasets. Hence, we introduce TUNa , a collection of personal anecdotes from blog posts, subreddit posts, online reviews, and fiction. We hire expert annotators obtaining honors undergraduate or graduate degrees in English literature to annotate these accounts for the three forms of unreliability mentioned above. To identify unreliable narrators automatically, we explore using large language models (LLMs). We conduct experiments with 6 open and closed-source LLMs of a variety of sizes. We try zero/few-shot settings, fine-tuning, and curriculum learning Bengio et al. 2009 . With these methods, we attempt to learn from labeled data from fiction and generalize this knowledge to real-world text. We observe that classifying unreliable narrators is a very difficult task and encourage future research to further explore its nuances and challenges. Our contributions are as follows: • We introduce the task of automatically identifying unreliable narrators; • We borrow narratological definitions for unreliable narrator (i.e., we consider three diverse and increasingly abstract forms: intra-narrational, inter-narrational, and inter-textual); • We introduce TUNa , an expert annotated dataset of unreliable first-person accounts spanning four different text domains; • We experiment with multiple methods that learn how to identify unreliable narrators in snippets from fiction and transfer this knowledge to common text read in everyday situations. 2 Background and

Read the complete text