Building an Atlas of Social Experiments to Link Studies, Reconcile Conflicts, and Bridge Gaps
Jiawei Zhang, Honglin Bao, Pengda Wang, Alex Yan, Xiao Liu, James A. Evans · 2026
Read Building an Atlas of Social Experiments to Link Studies, Reconcile Conflicts, and Bridge Gaps by Jiawei Zhang, Honglin Bao, Pengda Wang, Alex Yan, Xiao Liu, James A. Evans (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27153.txt.
Introduction
Zhang and co-authors turn a pile of social experiments into an atlas that shows what is known, what clashes, and what is missing. For a target study, their ExAtlas tool finds close prior studies and asks whether their effects can be combined to predict the target effect. A match links the target to past evidence, a mismatch prompts a search for factors that explain the clash, and a failed combination prompts a bridge experiment. On held-out targets, the tool gets the effect direction right in 98.6% of cases.
Catalog record
| CREATOR | Jiawei Zhang, Honglin Bao, Pengda Wang, Alex Yan, Xiao Liu, James A. Evans |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57946 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.27153
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Building an Atlas of Social Experiments to Link Studies, Reconcile Conflicts, and Bridge Gaps Jiawei Zhang, Honglin Bao, Pengda Wang, Alex Yan, Xiao Liu, James A. Evans Abstract Social and behavioral science runs thousands of experiments each year, yet their findings rarely accumulate into a coherent map of what is known, what conflicts, and what remains missing. We introduce ExAtlas , a framework for turning an archive of experiments into an atlas: a structured map in which studies link, conflict, or leave bridgeable gaps. Given a target study, ExAtlas searches for prior studies that are locally close in treatment and outcome space and asks whether their observed effects can be composed to predict the target effect. This yields three cases. If composition succeeds and agrees with the observed result, ExAtlas links the target to consistent prior evidence. If composition succeeds but disagrees, ExAtlas reconciles the conflict and proposes candidate moderators or higher-level theories that could explain it. If the composition fails, ExAtlas proposes bridge experiments to close the gap. We provide an error bound for composition under local smoothness of the treatment-effect surface. On held-out targets certified as locally supported, ExAtlas recovers effect direction in 98.6% of cases. Human evaluations further suggest that its proposed bridge experiments are plausible and exhibit connectedness, and that its conflict explanations are useful for theory generation. These results suggest that the archive of social experiments contains more latent structure than current practice extracts—and that making this structure explicit can guide both future theory and future experimentation. 1 Introduction Figure 1: Illustration of how ExAtlas links, reconciles, and bridges experimental evidence. Social and behavioral science has long suffered what Watts called an “incoherency problem”: scholars examine related phenomena without integrating them, competing findings coexist without being adjudicated, and isolated studies accumulate without theories that explain how they relate ( 26 ) . Research often proceeds one experiment at a time, leaving the literature fragmented even after decades of productive research ( 2 ) . Addressing this problem requires turning fragmented experimental evidence into a coherent research landscape. Such a landscape should make three things visible: which experiments speak to the same phenomenon, where they reinforce or contradict each other, and where gaps between them lie. Done well, it serves not as a summary but as an atlas—a map to guide the next theory and the next experiment. Figure 1 illustrates this vision: an experimental atlas links consistent experiments, reconciles conflicts, and bridges gaps in fragmented literature. We use atlas literally: a structured collection of local maps that together cover a territory, where each map is a region of treatment-outcome space in which experiments compose to predict each other’s effects, and where seams between maps are themselves objects of study. The organization of an experimental landscape is inherently compositional. A target finding may bear little surface resemblance to any single prior study, yet be clearly situated within the literature when several studies are interpreted together. Consequently, matching experiments by surface similarity alone is insufficient. Directly prompting an LLM does not address the problem either. What is required is the ability to reason across many studies at once: to detect compositions, conflicts, and gaps, and to determine how the evidential landscape changes when studies are evaluated jointly rather than in isolation. We introduce ExAtlas , a framework for constructing such an atlas from joint experimental evidence. At the core of ExAtlas is a composition step that asks whether a target experiment can be reconstructed from prior observed experiments. Consider a target study asking whether a 15-minute mindfulness break before an exam improves performance for first-generation college students. No single prior experiment may study this exact treatment-outcome pair. But the archive may contain studies of shorter mindfulness interventions before tasks, of study-skills supports targeted at first-generation students, and of brief contemplative practices in academic settings. If the target sits inside the region these studies span—close in treatment and outcome meanings—its effect can be predicted by an appropriately weighted average of theirs. The method is rejectable by design. Theoretically, the composition error decomposes into three terms: extrapolation (how far the target sits from its reconstruction), local model bias (how curved the treatment-effect surface is in the neighborhood), and unobserved residual (variation in factors no embedding captures). When all three are small, the composition is trustworthy. Empirically, we evaluate ExAtlas by holding out experiments from an archive and reconstructing their observed treatment effects from the remaining studies. The certification is the contribution, and the accuracy is its consequence. On targets that ExAtlas certifies as locally supported, the predicted direction matches the observed direction in 71 of 72 cases (98.61%). Based on the composition result, ExAtlas identifies conflicts and gaps. When a target is composable from nearby experiments, but the composition disagrees with the observed effect, ExAtlas treats the disagreement as a theoretical opportunity. Prior work and the new result imply different effects despite being close in treatment-outcome space, and an LLM is prompted with both to propose a reconciliation that could explain the divergence. When a target cannot be composed, the gap itself is the finding. ExAtlas iteratively proposes bridge experiments until the target enters the composable region. In this way, the same machinery that predicts known effects also distinguishes two cases: Kuhnian anomalies ( 15 ) , which are existing puzzles in the literature where experiments can be composed but yield conflicting predictions, and Lakatosian extensions ( 16 ) , which mark the field’s next frontiers where existing experiments cannot yet be composed. Human raters judge ExAtlas ’s bridge suggestions more plausible and more connected to the source experiments than forced composition of existing experiments. This suggests that the bridge module can turn gaps in the archive into actionable experiment designs. For composable targets with conflicting evidence, qualitative expert evaluation further shows that ExAtlas can turn disagreements into candidate theoretical explanations. For example, ExAtlas surfaces a hidden 2\times 2 moderator structure in which the effect of candidate age changes direction across cells, and the LLM articulates a reconciliation that experts judge plausible and useful for interpreting the conflict. Our contributions are as follows: (1) We introduce ExAtlas , a rejectable framework for composing experimental evidence, whose composition procedure turns experimental archives into an atlas of links, conflicts, and gaps. (2) We provide a theoretical error bound for composition under local smoothness, and show empirically that when ExAtlas certifies local support, composed effects recover the observed effect direction in 98.6% of cases. (3) We show that the atlas’s two failure modes (i.e., composition-with-conflict and non-composition) are themselves generative. The first yields theoretical proposals that human raters find substantive, the second yields bridge-experiment proposals that human raters find feasible, plausible, and well connected to the source literature. 2 Related Work LLMs for literature understanding and scientific discovery. A growing line of work asks LLMs to generate research ideas and hypotheses from the literature ( 21 ; 27 ; 4 ) , with 24 finding that LLM-generated NLP ideas can match or exceed human experts in novelty. This direction has been extended to experiment design ( 19 ; 10 ) and end-to-end research automation ( 22 ) . Most of these methods, however, retrieve literature shallowly, relying on keyword search and feeding