Identifying Non-Replicable Social Science Studies with Language Models

Denitsa Saynova Kajsa Hansson, Bastiaan Bruinsma Annika Fredén Moa Johansson · 2025

Read Identifying Non-Replicable Social Science Studies with Language Models by Denitsa Saynova Kajsa Hansson, Bastiaan Bruinsma Annika Fredén Moa Johansson (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-10671.txt.

Introduction

Denitsa Saynova and co-authors test whether language models can flag social science studies that will not replicate, in work from 2025. Failed replication shakes trust in behavioral research. A cheap early screen could steer costly rechecks to the shakiest claims.

Their data held fourteen past studies, nine that replicated and five that did not. Four instruction-tuned models each faked survey samples and re-estimated the original effects. The best model matched human replication verdicts with a strong score. A top commercial model scored lower than the best open model. Sampling settings mattered too: low randomness biased effect sizes. The method is a screen, not a verdict.

Catalog record

CREATORDenitsa Saynova Kajsa Hansson, Bastiaan Bruinsma Annika Fredén Moa Johansson
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-10671

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Identifying Non-Replicable Social Science Studies with Language Models Denitsa Saynova Kajsa Hansson, Bastiaan Bruinsma Annika Fredén Moa Johansson Abstract In this study, we investigate whether LLMs can be used to indicate if a study in the behavioural social sciences is replicable. Using a dataset of 14 previously replicated studies (9 successful, 5 unsuccessful), we evaluate the ability of both open-source (Llama 3 8B, Qwen 2 7B, Mistral 7B) and proprietary (GPT-4o) instruction-tuned LLMs to discriminate between replicable and non-replicable findings. We use LLMs to generate synthetic samples of responses from behavioural studies and estimate whether the measured effects support the original findings. When compared with human replication results for these studies, we achieve F1 values of up to 77 % 77\% with Mistral 7B, 67 % 67\% with GPT-4o and Llama 3 8B, and 55 % 55\% with Qwen 2 7B, suggesting their potential for this task. We also analyse how effect size calculations are affected by sampling temperature and find that low variance (due to temperature) leads to biased effect estimates. * * footnotetext: Equal contribution. 1 Introduction Since the early 2000s, several studies have shown that many findings in the behavioural social sciences are less replicable than previously thought ( Ioannidis 2005 ; Simmons et al. 2011 ) . This has since led to a widespread debate within the community about the reliability and robustness of their research designs ( Open Science Collaboration 2015 ; Camerer et al. 2016 ) . One of the first steps taken is to conduct large-scale replication projects, such as the Open Science Collaboration and Many Labs initiatives, to test the robustness of some of the more well-known studies. However, while these initiatives provide valuable insights, they are hampered by the significant financial and time demands they place on researchers, particularly early career researchers who often lack the resources to undertake such large-scale studies ( Bahlai et al. 2019 ; Whitaker and Guest 2020 ) . Figure 1: Illustration of a successful LLM replication (after Tversky and Kahneman 1981 ) A potential way to support these efforts is to use LLMs as a “silicon sample” to indicate replicability of studies in the social sciences. This would be in line with the growing interest in the social sciences in using LLMs to simulate human participants in behavioural experiments ( Park et al. 2024 ; Cui et al. 2024 ; Lippert et al. 2024 ; Hewitt et al. 2024 ) . And while it is clear that naively replacing humans with LLMs is not a viable approach, as LLMs suffer from problems such as hallucinations, prompt sensitivity, bias due to language, phrasing and response order ( Gallegos et al. 2024 ; Perković et al. 2024 ; Elazar et al. 2021 ; Huang et al. 2025 ) , there is evidence that they can capture relevant social information in some contexts, while they have difficulties in others ( Wang et al. 2024 ) . We do not hypothesise that LLM answer distributions capture the same distributions as observed in humans. Rather that the relative strengths of the underlying associations learned by the models may align with human behaviour enough to make them useful for this task. Here we propose to use LLMs to indicate whether a study should be considered for further investigation and replication efforts with human subjects . This means that the LLM serves the purpose to indicate which studies should be looked at further, for a replication using human participants to be carried out. The LLM, then, can be used as a way of justifying why additional experiments are worth conducting. To demonstrate our approach, we use 14 studies from the Many Labs 2 replication project that use human participants to see if using an LLM would lead to similar conclusions. A study is replicable (or successfully replicated), if the conclusion we draw from the replication aligns with the original finding. This applies when we use either human or synthetic (LLM) samples. To determine how well LLMs perform we compare the results from the LLM to the human replication. Previous studies have found low variance in LLM responses (including a so-called “correct-answer effect”, where a large part of all responses fall within the same response category ( Park et al. 2024 ) ). This can lead to an inability to detect differences between experimental conditions and biases in the statistical tests used to measure effect size. Thus, we test how temperature affects the model’s ability to indicate whether a study can be successfully replicated with a human sample. This work makes three main contributions: • Using samples from LLM generated responses, we find LLMs exhibit low replicability for unsuccessful human replications, while successfully replicating a proportion of the studies that have been successfully replicated with humans, suggesting their potential for predicting the replicability of a study; • We find no substantial difference in performance between smaller open-source models (Llama, Qwen and Mistral) and larger proprietary models (GPT-4o); • We find higher temperatures produce less usable samples (due to noise), however, lower temperatures produce a very narrow response distribution leading to zero or near-zero variation in responses and an overestimation of effect sizes; 2 Methods 2.1 Data We take 14 studies (out of 28 ) that were replicated in the Many Labs 2 replication project ( Klein et al. 2018 ) (see Appendices A and B for details). We exclude the other 14 as they contain data other than text and are therefore unsuitable for our analysis. The aim of Many Labs 2 is to confirm the results of the previous study under the same or similar conditions using (in most cases) a larger sample size. A replication is considered successful if the estimated effect is in the same direction as in the original study and is found to be significant. Out of the 14 studies, 9 are found replicable (successfully replicated) and 5 are found non-replicable (either no significant effect or opposite effect). We use the same criterion and same statistical tests for our replications (for details see Appendix C ). An example ( Figure 1 ) of a study we include is the study by Tversky and Kahneman 1981 . In the original version, the authors presented 181 participants with a scenario in which they were buying one of two items: a relatively cheap one ( \$30 ) or a more expensive one ( \$250 ). Participants are asked if they are willing to travel 20 minutes to another store for a \$20 discount. Both the original study and the Many Labs replication (with 7,228 participants), found people more willing to travel when the discounted product was the cheaper item. 2.2 Sampling We query LLMs with the study questions to generate synthetic samples from their responses. We use four models: three open-source: meta-llama/Meta-Llama-3-8B-Instruct (Llama), Qwen/Qwen2-7B-Instruct (Qwen), and mistral-ai/Mistral-7B-Instruct-v0.3 (Mistral), and one proprietary: gpt-4o-2024-08-06 (GPT-4o). To align with the query format of GPT-4o, we use instruction-tuned open-source models. To generate the responses from the LLM, we follow the approach by Park et al. 2024 . We pass the questions in a study to the model and ask it to fill in the blanks with answers to all questions. 1 1 1 Code will

Read the complete text