Leveraging Language Models for Interpretable Analysis of Narratives in a Large Corpus
Eric A. Bai, Minling Zhou, Ricardo Henao, Kyle M. Schwing, Lawrence Carin · 2025
Read Leveraging Language Models for Interpretable Analysis of Narratives in a Large Corpus by Eric A. Bai, Minling Zhou, Ricardo Henao, Kyle M. Schwing, Lawrence Carin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-18599.txt.
Introduction
Eric A. Bai, Minling Zhou, Ricardo Henao, Kyle M. Schwing, and Lawrence Carin measure story lines in a huge text corpus in a way readers can inspect. Narratives steer human acts and lie at the core of geopolitics, yet they have dodged counts that track their overlap and shift over time. Full model reads of a vast corpus cost too much and stay opaque.
Their model joins a word count topic code with a question and answer story model built on a language model. Both share one latent kernel space. Gradient steps for the fit stay readable and echo the self attention of transformers in form. A context based trick forecasts question outcomes for unasked texts. The mix aims at cheap, clear narrative gauges at corpus scale.
Catalog record
| CREATOR | Eric A. Bai, Minling Zhou, Ricardo Henao, Kyle M. Schwing, Lawrence Carin |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 103 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-18599
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Leveraging Language Models for Interpretable Analysis of Narratives in a Large Corpus Eric A. Bai, Minling Zhou, Ricardo Henao, Kyle M. Schwing, Lawrence Carin Abstract Narratives drive human behavior and lay at the core of geopolitics, but have eluded quantification that would permit measurement of their overlap and evolution. We present an interpretable model that integrates an established bag-of-words (BoW) topical representation and a novel LLM-based question answering (Q&A) narrative model, which share a latent Reproducing Kernel Hilbert Space representation, to quantify written documents. Our approach mitigates the cost, interpretability, and generalization challenges of using a LLM to analyze large corpora without full inference. We derive efficient functional gradient descent updates that are interpretable and structurally analogous to the self-attention mechanism in Transformers. We further introduce an in-context Q&A extrapolation method inspired by Transformer architectures, enabling accurate prediction of Q&A outcomes for unqueried documents. I Introduction I-A Motivation and Framing Large Language Models (LLMs) 5 ; 30 ; 8 have demonstrated remarkable capabilities in analyzing and summarizing large corpora of text. Given a document, an LLM can answer questions, extract themes, and even generate high-level insights. However, while these outputs are often fluent and compelling, the underlying reasoning process remains opaque. This black-box nature of LLMs poses a barrier to trust, especially in high-stakes domains such as policy analysis, legal review and healthcare, where decisions must be transparent, auditable, and justifiable to human stakeholders ( 15 ; 24 ) . In this work, we propose a structured and interpretable alternative: a double-softmax narrative model that leverages LLMs for question-answering but delegates the generalization and reasoning to a transparent latent model. Rather than relying on the LLM to analyze the corpus end-to-end, we use it to generate or answer a set of interpretable questions. These question and answer (Q&A) pairs are then modeled using a principled latent factor model, where each latent dimension – termed a “narrative” – is a collection of probability mass functions (PMFs) over answers to each question. This structure allows us to interpret each narrative as a coherent perspective on the corpus, grounded in observable Q&A behavior. This approach offers several advantages. First, it systematizes the use of LLMs: questions are designed and answered in a controlled, reproducible manner. Second, it enables interpretability at multiple levels: the questions themselves are human-readable, the latent narratives are composed of interpretable PMFs, and the document-level representations are convex combinations of these narratives. We also demonstrate that inference in the model is interpretable. Third, our proposed model supports extrapolation: by learning a mapping from document embeddings to narrative distributions, we can predict Q&A behavior for unseen documents without re-querying the LLM. This reduces cost, ensures consistency, and enables active learning strategies. Importantly, our framework can incorporate an additional layer of interpretability by asking the LLM to provide provenance information – that is, to specify which parts of the document support each answer. This source attribution can be stored alongside the Q&A data and used to audit or explain the model’s inferences. When combined with our structured narrative model, this enables a hybrid system that is not only interpretable in its latent structure but also traceable in its use of evidence. This is particularly valuable in domains where accountability and transparency are paramount ( 17 ) . Our model aligns with the growing emphasis on explainable AI (XAI) in high-stakes decision-making. As recent work has emphasized, interpretability is not merely a technical desideratum – it is a prerequisite for ethical, legal, and institutional accountability ( 15 ; 24 ) . By combining the expressive power of LLMs with the transparency of structured latent models, we offer a hybrid framework that is both powerful and trustworthy. In doing so, we reframe the role of the LLM: not as an opaque analyst, but as an oracle embedded within a transparent reasoning system. As illustrated in Figure 1 , the process begins with BoW-based topic modeling over the entire corpus. An LLM interprets the learned topics and identifies those aligned with the user’s interests. Documents with high probability under the selected topics are forwarded for Q&A analysis. One LLM designs a set of categorical questions, and a second LLM answers them for each document. This results in a structured representation of each selected document as a vector of Q&A responses, which serves as input to the narrative model. The narrative, structured here as Q&A responses, is the framing of the topic, which is scoped by the BoW. This relationship codifies the linguistic significance of narrative, a cognitive framework that provides social meaning to communication 12 . The topic and narrative models may be learned jointly or separately. In both cases, we model document-dependent distributions over topics and narratives using latent functions in a reproducing kernel Hilbert space (RKHS) 25 . These functions are updated via a functional gradient descent procedure that we show is interpretable from a Bayesian perspective. Specifically, the updates correspond to posterior-minus-prior shifts, aggregated across documents via a kernel that encodes document similarity. We show that this RKHS-based setup can be interpreted as a soft, deterministic analog of hierarchical Bayesian topic models such as HDP 27 ; 19 . Figure 1 : Overview of our coupled topic and narrative modeling system. The bottom layer shows BoW-based topic modeling: each topic is a PMF over words, and each document is represented by a PMF over topics. Based on user interest, an LLM selects one or more relevant topics, and the corresponding documents (in blue) are selected for further analysis. An LLM generates Q categorical questions, and a second LLM answers them for each selected document. Each narrative is defined by Q PMFs — one per question — and each document is assigned a PMF over narratives. The narratives and topics are learned jointly, and subsequently interpreted by an LLM. I-B Summary of Contributions This paper makes the following key contributions: • Interpretable Narrative Modeling: We introduce a novel paradigm in which each document is represented by a distribution over latent narratives — interpretable patterns in LLM-derived question-answer responses. Each narrative consists of a set of probability mass functions (PMFs) over answers to a set of user-selected questions, enabling structured and transparent perspectives across the corpus. These narratives distinguish among the claims of the documents. We note that our model determines discrete narrative differences that often are conflicting statements of causality, which have been a challenge to capture systematically and quantitatively 20 . • Joint RKHS-Based Topic and Narrative Modeling: We formulate both topic and narrative models within a shared reproducing kernel Hilbert space (RKHS). This design enables interpretable, modular learning via functional gradient descent, and provides a principled Bayesian interpretation of the inference procedure. • Transformer-Inspired Q&A Extrapolation: To reduce reliance on costly LLM inference, we develop a Transformer-style mechanism for in-context extrapolation. This allows the model to predict Q&A outcomes for unseen documents, generalizing narrative structure from a limited set of annotated examples. • Agentic and Auditable Analysis Framework: We operationalize our model within an agentic system, where LLMs perform interpretable subtasks (e.g., question generation, answering, provenance attribution), and the latent model encodes, extrapolates, and audits these outputs. This hybrid framework is designed for transparency, multilingual flexibility, and real-world scalability. I-C Related