Optimal Post-Hoc Theorizing
Andrew Y. Chen · 2025
Read Optimal Post-Hoc Theorizing by Andrew Y. Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-10370.txt.
Introduction
Chen asks when economists should build theory before, or after, seeing their data. For many questions, results only matter if they are strong, which changes the value of theorizing first. This 2025 paper models the choice in a Bayesian frame, weighing two forces.
Theorizing first gives a Darwinian benefit, since weak ideas die early. Looking at data first gives a statistical benefit, since effort flows to what the data support. Chen argues that with mature theory and huge data sets, the second force usually wins. Post-hoc theorizing then becomes the better sequence. The paper is short and formal, aimed at research practice rather than a single market or policy.
Catalog record
| CREATOR | Andrew Y. Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-10370
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Optimal Post-Hoc Theorizing Thanks: email:[email protected]. I thank Irene Caracioni for excellent research assistance, and Alejandro Lopez-Lira, Matt Ringgenberg, Mish Velikov, and Tom Zimmermann for helpful comments. The views expressed herein are those of the authors and do not necessarily reflect the position of the Board of Governors of the Federal Reserve or the Federal Reserve System. Andrew Y. Chen Abstract For many economic questions, the empirical results are not interesting unless they are strong. For these questions, theorizing before the results are known is not always optimal. Instead, the optimal sequencing of theory and empirics trades off a “Darwinian Learning” effect from theorizing first with a “Statistical Learning” effect from examining the data first. This short paper formalizes the tradeoff in a Bayesian model. In the modern era of mature economic theory and enormous datasets, I argue that post hoc theorizing is typically optimal. JEL Classification : B41, C18, C11 Keywords : Publication Bias, Machine Learning, Predictivism vs Accommodation, HARKing 1 Introduction Theories formed after observing empirical results ( post hoc theories), are viewed with suspicion by social scientists (e.g. Kerr 1998 ; Harvey 2017 ). Yet some of the most successful theories in all of science were formed this way (e.g. gravity, quantum mechanics). 1 1 1 Newton 1726 even said “whatever is not deduced from the phenomena… … have no place in experimental philosophy.” Consistent with this confusion, the philosophy literature has long debated the merits of post hoc vs a priori theorizing ( Barnes 2022 ) This paper provides a Bayesian model for understanding this “paradox.” It shows post hoc theory is clearly suboptimal if the sole goal of research is unbiased empirical results. Given statistics’ 100-year obsession with unbiasedness ( Efron 2001 ), it is perhaps unsurprising that post hoc theory is viewed suspiciously. However, the goal of research is typically more than unbiased empirical results. Another ubiquitous goal of research is to find “a good idea,” whether the idea is an investment strategy, health intervention, or model of human language. In such settings, statistical bias may matter little, as long as research provides a powerful solution. If the goal is a “good idea,” then the optimal research method trades off a Darwinian Learning effect with a Statistical Learning effect. Darwinian Learning comes from weeding out bad theories by subjecting them to prediction competitions. Statistical Learning simply comes from theorists improving their ideas after looking at data. If Statistical Learning is stronger than Darwinian Learning, then post hoc theorizing is optimal. In the modern world of enormous datasets and massive computing power, Statistical Learning is becoming more and more powerful. At the same time, the economic sciences have become mature, and Darwinian Learning has arguably run its course. For these reasons, I argue that post hoc theorizing is, in most cases, optimal. 1.1 Related Literature My model is an extension of the publication bias models ( Hedges 1984 ; Brodeur et al. 2016 ; Andrews & Kasy 2019 ; Abadie 2020 ; Chen & Zimmermann 2020 ; Jensen et al. 2023 ; Kasy & Spiess 2024 ). In these papers, it is unclear whether post hoc theory is harmful. In fact, the models in these papers exhibit the irrelevance result found in Hempel 1966 ; Lakatos 1970 ; and elsewhere (see Section 2.4 ). Building on the insights of from the philosophy literature (namely Maher 1988 ), I show how heterogeneous theories breaks this irrelevance. Breaking this irrelevance helps us understand the empirical findings of Chen et al. 2024 , which shows that asset pricing theories add little value for selecting strategies with strong post-research performance (see also Chen & Dim 2025 ). In the philosophy literature, Maher ( Maher 1988 , Maher 1990 ) and Kahn, Landsburg, and Stockman ( Kahn et al. 1992 ; Kahn et al. 1996 ) (KLS) study post hoc theorizing under heterogeneous theories. They document the selection effect that I call Darwinian Learning, and conclude that a priori theorizing is optimal, at least in normal scientific settings. Amid the centuries of debate (e.g. Leibniz 1969 ; Newton 1726 ; Keynes 1921 ), Barnes 1996 describes Maher’s analysis as “the closest thing to an illuminating account of predictivism in existence.” Predictivism is the view that a priori theorizing is optimal. My paper builds on Maher and KLS by showing how there is an offsetting effect to Darwinian Learning, namely Statistical Learning. This effect is ruled out by the assumptions in Maher and KLS. Statistical Learning is perhaps a natural extension of one of Howson & Franklin 1991 ’s ( Howson & Franklin 1991 ) criticisms of Maher ( Maher 1988 ) and ( Maher 1990 ) , though Maher 1993 also points out flaws in Howson & Franklin 1991 ’s ( Howson & Franklin 1991 ) criticisms. My paper provides clarity to this debate. Also unlike Howson and Franklin, I show how to connect Maher’s and KLS’s ideas to models of publication bias, and the broader statistics literature on large scale inference ( Efron 2012 ). 2 A Very Simple Model of Research Idea i\in\left\{1,2,...,N\right\} has quality \mu_{i} , which is unknown to the research community. Researchers may observe the measured quality of i \displaystyle\hat{\mu}_{i} \displaystyle=\mu_{i}+\varepsilon_{i} (1) where E\left(\varepsilon_{i}\right)=0 . i may be a real-world choice for readers (e.g. an investment strategy), in which case \mu_{i} is the realized, quality of i after the research is finished (“post-research”). Or i may be an explanation for some phenomenon (e.g. a model of obesity in adolescents), in which case \mu_{i} is the explanation’s fit to the phenomenon, post-research. In either case, higher \mu_{i} is better. Theory rules out some ideas, and is characterized by a set \displaystyle S\subseteq\left\{1,2,...,N\right\} (2) and \displaystyle\text{$i$ is consistent with theory if }i\in S. (3) Theorizing turns S into a selected idea i^{\ast} , and theorizing is either a priori or post hoc : • a priori : the researcher selects randomly from S , and announces the selected i^{\ast} . (In this simple model, all ideas are equally consistent with theory.) • post hoc : the researcher first examines the data (observes \{\hat{\mu}_{1},\hat{\mu}_{2},...\hat{\mu}_{N}\} ). Then the researcher chooses \displaystyle i^{\ast}=\arg\max_{i\in S}\hat{\mu}_{i}. (4) (The researcher chooses the idea with the largest measured quality, subject to the idea being consistent with theory.) 2.1 A Priori Theorizing is the Unbiased Ideal If the sole goal of research is to find an unbiased estimate of the idea quality, then a priori theorizing achieves this goal. The expected \hat{\mu}_{i} from a priori theorizing satisfies \displaystyle E\left(\hat{\mu}_{i}\mid i\in S\right) \displaystyle=E\left(\mu_{i}\mid i\in S\right). (5) where i is randomly selected from S . In contrast, the expected \hat{\mu}_{i} from post hoc theorizing is clearly biased: Lemma 1 . \displaystyle E\Bigl(\hat{\mu}_{i}\big|i=\arg\max_{j\in S}\hat{\mu}_{j}\Bigr) \displaystyle>E\Bigl(\mu_{i}\big|i=\arg\max_{j\in S}\hat{\mu}_{j}\Bigr). (6) Proof. The LHS can be written as \displaystyle E\left(\mu_{i}\big|i=\arg\max_{j\in S}\hat{\mu}_{j}\right)+E\bigg(\varepsilon_{i}\big|i\in S,\{\varepsilon_{i}>\hat{\mu}_{j}-\mu_{i},\quad\forall j\in\left(S\setminus\{i\}\right)\}\bigg) The first term is the RHS of Equation ( 5 ). Thus we just need to show the second term is positive. The second term is positive because E\left(\varepsilon_{i}\big|i\in S\right)=0 , and because the second condition on \varepsilon_{i} cuts off the lower tail of the distribution. ∎ Intuitively, \hat{\mu}_{i} contains both \mu_{i} and measurement error. Selecting on large \hat{\mu}_{i} then selects for positive measurement error, leading to a biased estimate.