Making Interpretable Discoveries from Unstructured Data: A High-Dimensional Multiple Hypothesis Testing Approach
Jacob Carlson · 2025
Read Making Interpretable Discoveries from Unstructured Data: A High-Dimensional Multiple Hypothesis Testing Approach by Jacob Carlson (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-01680.txt.
Introduction
Jacob Carlson builds an automated pipeline for discovering findings in unstructured data, such as the free-text answers in surveys of economic beliefs. Traders of interpretation are the bottleneck: a researcher reading text chooses what to look for, which invites data snooping and missed angles.
The framework first uses AI interpretability methods to convert raw text into high-dimensional measurements of interpretable concepts. It then states concept-level hypotheses, tests them with algorithms backed by new high-dimensional multiple-testing results, and writes up the surviving discoveries in plain language, again automatically. With few researcher degrees of freedom, the method supports fast sensitivity checks and replication. Revisiting recent economics studies, Carlson's pipeline reproduced known discoveries, added nuance to others, and surfaced entirely new ones.
Catalog record
| CREATOR | Jacob Carlson |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 77 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-01680
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Making Interpretable Discoveries from Unstructured Data: A High-Dimensional Multiple Hypothesis Testing Approach Jacob Carlson Abstract Social scientists are increasingly turning to unstructured datasets to unlock new empirical insights, e.g., estimating causal effects on text outcomes, measuring beliefs from open-ended survey responses. In such settings, unsupervised analysis is often of interest, in that the researcher does not want to pre-specify the objects of measurement or otherwise artificially delimit the space of measurable concepts; they are interested in “discovery.” This paper proposes a general and flexible framework for pursuing discovery from unstructured data in a statistically principled way. The framework leverages recent methods from the literature on machine learning interpretability to map unstructured data points to high-dimensional, sparse, and interpretable “dictionaries” of concepts; computes (test) statistics of these dictionary entries; and then performs selective inference on them using newly developed statistical procedures for high-dimensional exceedance control of the k k -FWER under arbitrary dependence. The proposed framework has few researcher degrees of freedom, is fully replicable, and is cheap to implement—both in terms of financial cost and researcher time. Applications to recent descriptive and causal analyses of unstructured data in empirical economics are explored. An open source Jupyter notebook is provided for researchers to implement the framework in their own projects. 1 Introduction Empowered by recent developments in machine learning and AI, researchers in the social sciences are increasingly leveraging sources of unstructured data—such as text, images, videos, and audio—in quantitative analyses. In economics, interest in unstructured data sources is especially widespread, ranging from using speech recordings from FOMC meetings to better understand monetary policy [ Gorodnichenko et al. 2023 ] ; to using videos of start-up pitches to study entrepreneurship and investment [ Hu and Ma 2025 ] ; to using open-ended survey questions to probe the economic behavior and beliefs of individuals [ Haaland et al. 2024 ] ; to using mug shots to study judicial decisionmaking [ Ludwig and Mullainathan 2024 ] ; to using visual art or written narratives to infer long-run living standards [ Gorin et al. 2025 , Lagakos et al. 2025 ] ; to using interview transcripts to better understand the impacts of RCT treatments [ Bergman et al. 2024 , Krause et al. 2025 ] . The promise of these unstructured data, fundamentally, is that they offer new opportunities to measure social phenomena that were previously unmeasurable. This includes both measures that a researcher is capable of pre-specifying as well as those that a researcher is not, and instead would like to “discover.” When modern AI/ML methods are brought to bear on these goals, the former falls under the heading of “supervised learning,” whereby the researcher typically makes predictions of their ex-ante known quantity of interest using the unstructured data as features, and the latter falls under the heading of “unsupervised learning,” which is less amenable to a prediction-based framework, and demands that AI/ML methods uncover latent structure in the unstructured data with minimal researcher input. Though new statistical and econometric frameworks have recently become available for interpretable and statistically principled analyses of unstructured data in the supervised learning setting (e.g., Angelopoulos et al. 2023 , Ludwig et al. 2024 , Carlson and Dell 2025 ), it is an open question as to how best perform interpretable and statistically principled analyses of unstructured data in unsupervised learning settings that emphasize discovery. The framework proposed in this paper provides one possible answer to this open question. Specifically, this paper proposes a general framework for conducting interpretable and rigorous statistical inference on unstructured data in the following way, starting from an unstructured dataset \{Z_{i}\}_{i=1}^{n} : 1. Convert unstructured data to dictionary feature vectors: The researcher converts the unstructured data instances Z_{i}\in\mathcal{Z} into high-dimensional, sparse, binary vectors Y_{i}\in\{0,1\}^{p} using recently developed methods in dictionary learning for mechanistic interpretability of large language models (LLMs) and other deep neural networks (DNNs), i.e., Y_{i}:=\mathtt{Dict}(Z_{i}) . Both theory and empirics corroborate that vector entries Y_{ij} correspond to the presence of a monosemantic feature (i.e., a feature with a single, interpretable meaning) or “concept” j in the given unstructured data instance i . There are p\gg n dictionary features, e.g., we assume that n may be on the order of 10^{2} or 10^{3} , and p is on the order of 10^{4} or 10^{5} (or larger). The breadth and depth of concepts spanned by these dictionary features is vast, as can be gleaned from the literature on dictionary learning for LLMs, and as will be observed concretely in the empirical applications of this paper. Intuitively, the dictionary should contain all possible (monosemantic) concepts that the LLM has learned for the purpose of next token prediction over a massive internet-scale corpus. However, due to sparsity imposed by design and suggested by machine learning theory, only a handful of these dictionary features will be present (‘‘activate’’) for any given unstructured data point. 1 1 1 For more discussion of what “monosemanticity” means and its importance for LLM interpretability, see, e.g., the discussion in Bricken et al. 2023 . 2. Compute test statistics from the dictionary feature vectors: The researcher computes from this newly created dictionary feature indicator dataset \{Y_{i}\}_{i=1}^{n} a vector of p test statistics, T_{n}\in\mathbb{R}^{p} . These test statistics are intended to test p hypotheses, one for each dictionary feature, and may be flexibly chosen by the researcher. Especially important and simple choices explored in this paper’s empirical examples include hypotheses testing means and differences-in-means, feature by feature. 3. Perform high-dimensional selective inference using the test statistics: Finally, the researcher invokes a high-dimensional selective inference procedure newly developed in this paper to control the k -FWER of the selected set of rejected hypotheses (“discoveries”). This procedure is asymptotically valid for small k , requires only very mild assumptions on the data generating process (which are met in the applications of interest), and is well-powered in the empirical examples considered. This procedure may also be of independent interest in other high-dimensional selective inference settings. The output of this framework is therefore a set of interpretable discoveries (rejections of dictionary feature derived tests) based on some set of unstructured data \{Z_{i}\}_{i=1}^{n} with generalized familywise error rate guarantees, which may be relaxed or tightened as the researcher sees fit to permit more or fewer discoveries. Two notable use cases for this framework explored in the empirical applications of this paper include: 1. Estimating average treatment effects on dictionary features in RCTs, or testing, for treatment indicator D_{i} and fixed probability of treatment \pi , H_{0,j}:E[Y_{ij}(1)-Y_{ij}(0)]=E[H_{i}Y_{ij}]=0,\quad H_{i}=\frac{D_{i}-\pi}{\pi(1-\pi)}. 2. Estimating the probability of the presence of various concepts (measured by monosemantic dictionary features) in the population from which the unstructured dataset was sampled, or testing H_{0,j}:E[Y_{ij}]=0 and then inverting these tests to form a “generalized” confidence set [ Romano and Wolf 2007 ] . This particular framework for discovery from unstructured data conveys several scientific benefits. The first is that of having few researcher degrees of freedom . In many settings featuring exploratory analysis of unstructured data, there is concern that researchers may cherry pick aspects of their data to measure or cherry pick choices of measurements in order to reverse engineer a specific conclusion. By design, however, the framework proposed by this paper has very little room