Evaluating Meta-Regression Techniques: A Simulation Study on Heterogeneity in Location and Time
Jonathan Gendron, Ali Habibnia · 2025
Read Evaluating Meta-Regression Techniques: A Simulation Study on Heterogeneity in Location and Time by Jonathan Gendron, Ali Habibnia (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-16696.txt.
Introduction
Jonathan Gendron and Ali Habibnia race methods that pool past studies when the pooled set shifts by place and time. Such pooled reads, called meta-regressions, can skew when groups differ in both ways at once. Their 2025 mock study builds fake person-level data and shifts the gaps step by step. Old pooled forms face seven new forms that hold both gaps in view, one of them the pair's own.
Each form is scored on punch, skew, and grit as gaps grow. Forms that hold both gaps beat forms that hold one, the runs show. A form with place locks and a time trend earns a close look in their picks. Plain pick rules close the study. The guide serves fields from health to trade, where pooled proof rules big calls.
Catalog record
| CREATOR | Jonathan Gendron, Ali Habibnia |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 122 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-16696
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluating Meta-Regression Techniques: A Simulation Study on Heterogeneity in Location and Time Jonathan Gendron, Ali Habibnia Abstract In this paper, we conduct a simulation study with subject level data to evaluate conventional meta-regression approaches (study-level random, fixed, and mixed effects) against seven methodology specifications new to meta-regressions that control joint heterogeneity in location and time (including a new one that we introduce). We systematically vary heterogeneity levels to assess statistical power, estimator bias and model robustness for each methodology specification. This assessment focuses on three aspects: performance under joint heterogeneity in location and time, the effectiveness of our proposed settings incorporating location fixed effects and study-level fixed effects with a time trend, as well as guidelines for model selection. The results show that jointly modeling heterogeneity when heterogeneity is in both dimensions improves performance compared to modeling only one type of heterogeneity. Keywords: Meta-regression, simulation, heterogeneity, random-effects, fixed-effects, meta-analysis JEL Codes: B41; C15; C40; C52 Code availability: Github link Competing Disclosure: To the best of our knowledge, no competing interests exist Funding Statement: This research was not funded 1 Introduction A meta-analysis is a useful tool that combines results from independent studies with similar design to examine whether there is consistency in the results across studies or if those results conflict ( McGrath et al., (2019) ; Bakbergenuly & Kulinskaya, (2018) ; Rice et al., (2018) ; Kelley, (2012) ) . Meta-regressions are commonly referred to as meta-analyses (and vice versa), since the only difference introduced by a meta-regression is that it transcends the univariate model of a meta-analysis by including identifiable study and/or subject characteristics as covariates. Meta-regression data looks very similar to regression data with covariates, a variable of interest, and group-level controls. The only difference is that the estimators are estimating the common effect sizes after controlling for the study-level heterogeneity, instead of a effect size for a specific data set. Data from the different studies under consideration can be at the subject level or aggregate level using the summary statistics of each study; this manuscript will focus on the subject level due to the many issues inherent in aggregate level data ( Yamaguchi et al., (2014) ; Cooper & Patall, (2009) ; Lambert et al., (2002) ; Stewart & Clarke, (1995) ) . Consequently, thousands of meta-analyses have been performed, primarily in the medical, pharmaceutical, ecology, and social sciences literature ( Borenstein et al., (2009) ) . The methodology used for meta-analyses is designed to accurately estimate the effect sizes and mitigate any heterogeneity that may be present between each of the studies in the meta-analysis. The standard methodology uses study-level effects: most commonly random effects, fixed effects, and on rare occasions mixed effects ( Gambarota & Altoè, (2024) ; T. Stanley et al., (2023) ; Sugasawa & Noma, (2021) ; Richardson et al., (2021) ; Pateras et al., (2018) ; T.D. Stanley & Doucouliagos, (2017) ) . But what about heterogeneity in location or time? It is common for meta-analyses to include decades worth of studies from many locations around the world, which introduces the likely variation in both the time and location in which each study was conducted ( Thöni & Volk, (2021) ; Van Den Akker et al., (2020) ; Engel, (2011) ; Oosterbeek et al., (2004) ) . In the experimental economic and meta-analysis simulation literature, almost all meta-analyses are conducted at the study level (dominantly using random effects), with only a small percentage controlling for location variation and none controlling for time variation. 1 1 1 Table 4 in Appendix A shows a representative sample of meta-regressions in experimental economics and how they control for heterogeneity: only about 30% control for location heterogeneity, only about 15% control for precise country-level heterogeneity, and only about 5% control for time heterogeneity (implicitly). This manuscript tests three hypotheses when the data follows the standard assumptions: 1. When facing joint location and time heterogeneity, specifications that control both location and time will perform better than those that only control for one dimension (i.e., location or time). • We posit this because ignoring cross-sectional variation will likely induce the omitted variable bias. 2. Study-level specifications will perform worse than specifications that explicitly control location and time when heterogeneity is at the location and time level. • We propose this because if studies share a location or year, study-level controls may fail to capture that shared variation. 3. When location or time heterogeneity is present, even methodology specifications that control that heterogeneity in location or time will perform worse. • We posit this because even when study-level heterogeneity exists, study-level specifications tend to under perform - consistent with meta-analysis simulation findings (see Section 2 ). This manuscript provides three key contributions that are novel for both the meta-regression simulation literature and practitioners in any discipline that use meta-regressions. First, the performance of seven methodology specifications new to meta-regressions is tested in a simulation environment with various degrees of location and time-level heterogeneity. This is very important because meta-regressions are inherently at high risk of joint heterogeneity in location and time, and location heterogeneity as well as joint heterogeneity have not been tested in a simulation environment 2 2 2 The only exception are spatiotemporal heterogeneity meta-analyses, but these measure this heterogeneity very differently. Please see the literature review in Section 2 for more details . Second, performance of two particular methodologies is tested in our simulation, location-level fixed effects from the experimental economic literature, and study-level fixed effects with a trend from the meta-analysis simulation literature. This testing is also important because it will answer the appropriateness of each methodology specification in the face of joint location and time heterogeneity. Third, since thousands of meta-regressions are regularly conducted in a large variety of literature, we have outlined steps for optimal model selection that any practitioner can follow. As the manuscript will show, model selection should be carefully considered depending on the characteristics of the data, especially whether there is joint heterogeneity in location and time or not. This research aims to test which meta-regression methodology performs optimally when joint heterogeneity in the location and time between studies in a meta-regression is present in varying degrees. To achieve our research objective, we conduct a simulation study that tests the performance of ten meta-regression methodology specifications that include the standard methodology and the specifications that control for heterogeneity in location and/or time. Note that most of the methodology specifications are fixed-effect specifications since the assumption of random effects is a phenomenon that rarely holds in practice (which is required for random or mixed effect models, please see Section 5.1 for further details) ( Clark & Linzer, (2015) ; Wooldridge, (2010) ; Baltagi, (2005) ; Hausman & Taylor, (1981) ; Mundlak, (1978) ) . In summary, our simulation found that the standard methodology performed the best and the specifications that control for joint heterogeneity performed a close second best. This excludes study-level fixed effects with a trend proposed in the meta-analysis simulation literature because this provided inaccurate trend estimates, but includes a new specification inspired by the literature and introduced by this manuscript: location-level fixed effects with a trend. Accordingly, specifications that did not jointly control for location and time