Identifying economic narratives in large text corpora An integrated approach using Large Language Models
Tobias Schmidt, Kai-Robin Lange, Matthias Reccius, Henrik Müller, Michael Roos, Carsten Jentsch · 2025
Read Identifying economic narratives in large text corpora An integrated approach using Large Language Models by Tobias Schmidt, Kai-Robin Lange, Matthias Reccius, Henrik Müller, Michael Roos, Carsten Jentsch (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-15041.txt.
Introduction
Tobias Schmidt and five co-authors test whether large language models can find economic stories in news text, in a study from 2025. Such stories, like tales about why prices rise, shape how people and markets act. Older software chains could pull out who did what, but they often missed the deeper sense that makes a text a true story.
The team set a strict rule for what counts as a story and ran a leading model on Wall Street Journal and New York Times pieces about inflation. They then set the model output side by side with stories marked by trained experts. Early results suggest the model can match expert work well. The authors argue this simpler path could replace complex chains of older tools.
Catalog record
| CREATOR | Tobias Schmidt, Kai-Robin Lange, Matthias Reccius, Henrik Müller, Michael Roos, Carsten Jentsch |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 105 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-15041
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Identifying economic narratives in large text corpora An integrated approach using Large Language Models Tobias Schmidt, Kai-Robin Lange, Matthias Reccius, Henrik Müller, Michael Roos, Carsten Jentsch Abstract Identifying economic narratives in large text corpora – An integrated approach using Large Language Models Tobias Schmidt 1 (TU Dortmund University), Kai-Robin Lange 1 (TU Dortmund University), Matthias Reccius 1 (Ruhr-University Bochum), Henrik Müller (TU Dortmund University), Michael Roos (Ruhr-University Bochum) and Carsten Jentsch (TU Dortmund University) Abstract As interest in economic narratives has grown in recent years, so has the number of pipelines dedicated to extracting such narratives from texts. Pipelines often employ a mix of state-of-the-art natural language processing techniques, such as BERT, to tackle this task. While effective on foundational linguistic operations essential for narrative extraction, such models lack the deeper semantic understanding required to distinguish extracting economic narratives from merely conducting classic tasks like Semantic Role Labeling. Instead of relying on complex model pipelines, we evaluate the benefits of Large Language Models (LLMs) by analyzing a corpus of Wall Street Journal and New York Times newspaper articles about inflation. We apply a rigorous narrative definition and compare GPT-4o outputs to gold-standard narratives produced by expert annotators. Our results suggests that GPT-4o is capable of extracting valid economic narratives in a structured format, but still falls short of expert-level performance when handling complex documents and narratives. Given the novelty of LLMs in economic research, we also provide guidance for future work in economics and the social sciences that employs LLMs to pursue similar objectives. JEL-Code: C18, C55, C87, E70 Key words: economic narratives, natural language processing, large language models 1 : equal contribution Acknowledgments: This study is part of a project of the Dortmund Center for data-based Media Analysis (DoCMA) at TU Dortmund University and the Narrative Economic Alliance Ruhr (NEAR) project, supported by the Mercator Research Center Ruhr (MERCUR) with project number Ko-2022-0015. It was also partially funded by the Reality Check incubator project at the Research Center for Trustworthy Data Science and Security. 1 Introduction Macroeconomic policy decisions are not made in a vacuum. More and more, central banks and other institutions recognize the role that public discourse and widely shared narratives play in shaping expectations and thus economic behavior. The central bank communications literature in particular has described the active role of policy makers in explaining their policies to great detail ( Gurkaynak et al. 2005 ; Hansen et al. 2017 ; Hansen et al. 2019 ; Gorodnichenko et al. 2023 ) . But neither fiscal nor monetary policy discourses are top-down processes. People’s economic views and expectations are increasingly shaped by the media ( Fiore et al. 2025 ) . Particularly in times of uncertainty, stories about rising inflation, looming recessions, or job market disruptions spread rapidly and uncontrollably. As economic dynamics are driven by the beliefs and expectations of households and firms, monitoring the narratives that dominate public discourse is crucial. Quantifying economic narratives in mass media can be challenging, however, as the amount of information transmitted is vast and it’s rhetorical packaging diverse. Many researchers perform qualitative analyses, which require expert evaluations of texts. While such procedures produce reliable results, they are not scalable to large text corpora. Increasing numbers of articles, reports, opinion pieces and even comments on social media are released every day, so purely relying on qualitative research is not feasible when analyzing the narratives that circulate in an economy. Therefore, economists need efficient, quantitative methods of extracting and presenting narratives from texts. State-of-the-art language models such as BERT ( Devlin et al. 2019 ) and its variations have shown considerable success in tasks like sentiment analysis, named entity recognition, and semantic role labeling, which are seen by many as foundational to the identification of economic narratives. However, while these models excel at narrow linguistic tasks, they often fall short when tasked with the deeper, more nuanced understanding required to distinguish between complex economic narratives and simpler textual structures. Pipeline approaches that mix different methods also introduce compounding error risks due to their multiple processing stages. Hence, traditional NLP models address the scalability problem but lack the integrated language comprehension needed to fully capture the contextual meaning and economic relevance of narratives. This gap presents an opportunity for exploring alternative approaches that can offer greater contextual understanding. The advent of Large Language Models (LLMs), with both commercial models such as GPT ( Brown et al. 2020 ) , Claude Sonnet and Google Gemini and open source models like Llama ( Dubey et al. 2024 ) , has opened up entirely new possibilities for narrative extraction. LLMs have demonstrated remarkable proficiency in a wide array of tasks, excelling especially in those that require a deeper comprehension of language and context ( Shahriar et al. 2024 ) . These models are trained on vast amounts of data and can process complex narratives with a level of understanding closer to that of human readers. But how should economists leverage LLMs to accurately measure complex phenomena such as narratives? And can LLMs match expert annotators in identifying and extracting economic narratives from newspaper articles? In this paper, we use a set of complex instructions and expert-labeled examples to evaluate the proficiency of LLMs at extracting economic narratives. We utilize GPT-4o ( OpenAI et al. 2024a ) , the leading commercial LLM at the time of analysis, for the extraction of inflation narratives from a corpus of Wall Street Journal and New York Times newspaper articles. We further demonstrate how complex concepts like economic narratives can be operationalized for human coding, and then adapted for coding by LLMs. While such models enable new avenues for quantitative economic research, the litmus test for their usefulness lies in the rigorous validation of their outputs. Hence, we compare the results of the LLMs’ analysis against a gold-standard set of narratives, representing the consensus of three trained annotators with domain expertise. A detailed and rigorous annotator codebook was developed through multiple iterations and refinements. We find that expert annotators reach notably different results extracting narratives, pointing to the inherent subjectivity involved in narrative sense-making. To refine and adjust our evaluation of GPTs’ performance to the subjectivity of the task, we not only consider the gold-standard data set as a set of optimal answers, but also the deviations made by the expert annotators, which we call “expected” deviations. We also compare the results attainable through a set of different LLM prompting strategies recently found to be the most effective by research in Artificial Intelligence (AI). The rest of the paper is structured as follows. In Section 2 , we shed light on previous research that has aimed to define economic narratives or automatically extract them from texts. We also provide a short overview of inflation-related literature, as inflation is the center of our evaluation strategy. In Section 3 , we introduce the data as well as the LLM we use in our experiments. Section 4 covers our narrative extraction codebook as well as a description of the process of creating a gold-standard data set. In Section 5 we explain our prompting strategy and compare it to other approaches used in the field of AI to maximize LLM performance. The results of