Narratives to Numbers: Large Language Models and Economic Policy Uncertainty

Ethan Hartley · 2025

Read Narratives to Numbers: Large Language Models and Economic Policy Uncertainty by Ethan Hartley (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-17866.txt.

Introduction

Ethan Hartley evaluates large language models as tools for measuring economic policy uncertainty from news text, a widely used index based on word counting rules. Better measurement matters because the index feeds many studies of how uncertainty affects markets and policy.

Modern language model classifiers beat dictionary rules, match human audit judgments more closely, and handle noisy old and multilingual news. Hartley builds a nineteenth century index for the United States from hundreds of millions of newspaper articles, plus exploratory indexes for other countries. The study argues that such models should be treated as explicit measurement tools in economics.

Catalog record

CREATOREthan Hartley
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-17866

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Narratives to Numbers: Large Language Models and Economic Policy Uncertainty Ethan Hartley Abstract This study evaluates large language models as estimable classifiers and clarifies how modeling choices shape downstream measurement error. Revisiting the Economic Policy Uncertainty index, we show that contemporary classifiers substantially outperform dictionary rules, better track human audit assessments, and extend naturally to noisy historical and multilingual news. We use these tools to construct a new nineteenth-century U.S. index from more than 360 million newspaper articles and exploratory cross-country indices with a single multilingual model. Taken together, our results show that LLMs can systematically improve text-derived measures and should be integrated as explicit measurement tools in empirical economics. \draftSpacing 1.5 \shortTitle Narratives to Numbers \pubMonth 11 \pubYear 2025 \pubVolume Vol \pubIssue Issue \JEL C45, C38, D8, E6 \Keywords large language models, deep learning, natural language processing, policy uncertainty Uncertainty is fundamental. It shapes how firms invest, how households spend, and how policymakers design and time interventions. Policy-related uncertainty is especially central in current debates about trade tensions, unconventional monetary policy, and healthcare, yet it is not directly observed. It is a latent feature of the environment that we infer from behavior and from communication about policy. Text offers a natural way to make this latent object measurable: newspaper articles, government communications, and other narratives record when policy is contested, delayed, or unexpectedly changed. Turning those narratives into statistically relevant data requires a mapping from high-dimensional text to scalars. We show that the design of this mapping materially affects the text-based indices used in empirical work and that modern, empirically driven methods yield markedly better measures. In macroeconomics and finance, the leading example is the Economic Policy Uncertainty (EPU) index of Baker, Bloom and Davis 2016 (henceforth BBD), constructed by counting newspaper articles that mention a small dictionary of policy-related and uncertainty-related terms. That keyword-based design made it possible to measure policy uncertainty at scale and has been widely adopted, but it also hard-wires a particular, hand-crafted mapping from text to an index. This raises a basic question: how sensitive are the resulting series (and the empirical conclusions that rely on them) to the choice of text classifier, and what do economists gain by using modern language models instead? We address these questions within the BBD framework, comparing large language models (LLMs)—high-dimensional models of natural language trained on massive text corpora to predict the next word—to the canonical keyword-based approach and show that the choice of classifier materially changes both article-level performance and the resulting index. Our empirical analysis proceeds in three steps. We first compare keyword-based models, support vector machine (SVM) classifiers, and fine-tuned LLMs on the human-audited BBD corpus. We then embed these models in the standard EPU construction pipeline to study how classifier performance, decision threshold, and aggregation choices affect resulting indices. Finally, we apply these models to hundreds of millions of historical U.S. articles and millions of multilingual articles, constructing a nineteenth-century U.S. index, a GDP-weighted African index, and a Bangladesh EPU index. Three findings emerge. First, LLMs substantially outperform keyword-based models at the article level, delivering 46% relative improvements in F1 score and closer alignment with human auditor assessments. Second, these gains matter for the resulting indices. Alternative classifiers and aggregation choices produce noticeably different series, and these design decisions can be used by the econometrician to balance data quality against data scarcity and shape the resulting pattern of measurement error. Third, learned representations transfer well. They allow us to process noisy articles dating back to the 1800s and classify text in 29 languages using a single mapping learned from English articles. Our analysis adds to the growing text-as-data literature in economics and finance. Classic applications map text to numbers using dictionaries and term frequencies ( Tetlock 2007 ; Gentzkow and Shapiro 2010 ; Loughran and McDonald 2011 , e.g.,) . Recent work uses machine learning to digitize and structure large corpora from newspapers and other textual sources, greatly expanding available datasets ( Dell et al. 2023 ; Silcock et al. 2024 ) . A more closely related strand applies LLMs to economic prediction problems, such as forecasting labor market transitions from structured resumes ( Vafa et al. 2022 ; Athey et al. 2024 ) . We extend this agenda to measurement, using LLMs as calibrated classifiers that map text into policy-uncertainty labels and treating them as explicit, estimable components of the data-generating process for text-based indices. Within the specific literature on measuring policy uncertainty, most indices continue to rely on static keyword sets or rule-based classifiers, sometimes augmented with sentiment weighting ( Charemza, Makarova and Rybiński 2022 ) , co-occurrence constraints ( Benguria et al. 2022 ) , or topic models ( Azqueta-Gavaldón et al. 2023 ) . These approaches are attractive because they are transparent and easy to implement, but they embed strong assumptions about how uncertainty is expressed in text and how it should be aggregated into a univariate series. Supervised methods, including SVMs trained on labeled articles ( Tobback et al. 2018 ; Noailly, Nowzohour and van den Heuvel 2022 ) and more recent work that uses fine-tuned or prompted language models to construct policy-related uncertainty indices ( Trust, Zahran and Minghim 2023 ; Ito, Sato and Ota 2025 ; Audrino, Maly and Stalder 2024 ) , illustrate gains in performance but rely on narrow domains, automated or weak labels, and limited validation. By training directly on large-scale human labels and evaluating against the original audit design in Baker, Bloom and Davis 2016 , we provide a transparent benchmark for what is achievable with modern, probabilistic, context-aware models and a methodology that can be ported to other policy domains and text-as-data applications. Finally, we contribute to the large empirical literature employing policy uncertainty indices (and other variables constructed from unstructured data) as regressors in macroeconomic and financial models ( Caldara and Iacoviello 2022 ; Jahan-Parvar et al. 2024 ; Kong et al. 2022 , e.g.,) . These indices are typically treated as observed covariates, even though they are generated by an underlying model and measured with error. LLM-based classifiers produce probabilistic article-level assessments, allowing researchers to evaluate how threshold and aggregation choices shape the resulting series and downstream measurement error. This focus on measurement is complementary to recent work on robust inference with machine-learned predictions, such as the MARS framework of Carlson and Dell 2025 and the LLM analysis of Ludwig, Mullainathan and Rambachan 2025 . We show that the choice of classifier and calibration can materially alter text-based signals that enter downstream models, and that recognizing this endows researchers with a transparent set of tuning parameters for measuring, decomposing, and mitigating the resulting uncertainty. 1 Identifying Policy Uncertainty As proposed in BBD, constructing a text-based EPU index proceeds in two steps. First, articles are labeled using a simple bag-of-words (BOW) classifier: an article is labeled EPU-related if it contains at least one term from each of three keyword groups: • Economic : “ economic ” or “ economy ” • Policy : “ Congress ,” “ deficit ,” “ Federal Reserve ,” “ legislation ,” “ regulation ,” or “ White House ” • Uncertainty : “ uncertain ” or “ uncertainty ” Let a_{ijt}^{\mathrm{EPU}} denote the indicator that

Read the complete text