Large language model-based agents for automated research reproducibility: an exploratory study in Alzheimer’s Disease

Nic Dobbins, PhD, Christelle Xiong, BS, Kristine Lan, MS, Meliha Yetisgen, PhD · 2025

Read Large language model-based agents for automated research reproducibility: an exploratory study in Alzheimer’s Disease by Nic Dobbins, PhD, Christelle Xiong, BS, Kristine Lan, MS, Meliha Yetisgen, PhD (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23852.txt.

Introduction

Reproducing a published study by hand is slow, so Nic Dobbins and co-authors hired a simulated research team of AI agents to try. Given five Alzheimer's studies, the dataset they used, and only each paper's abstract, methods, and data dictionary, the agents wrote and ran their own analysis code.

Across thirty-five headline findings, the agents approximately reproduced about half per study. Exact numbers and ranges often drifted. The agents also chose different statistical settings than the originals, though trends and significance sometimes matched. Failures traced to coding flaws and missing method details. The authors call the system a promising, scalable rigor check, not yet a replacement for human reproduction.

Catalog record

CREATORNic Dobbins, PhD, Christelle Xiong, BS, Kristine Lan, MS, Meliha Yetisgen, PhD
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-23852

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Large language model-based agents for automated research reproducibility: an exploratory study in Alzheimer’s Disease Nic Dobbins, PhD, Christelle Xiong, BS, Kristine Lan, MS, Meliha Yetisgen, PhD Abstract Objective : To demonstrate the capabilities of Large Language Models (LLMs) as autonomous agents to reproduce findings of published research studies using the same or similar dataset. Materials and Methods : We used the "Quick Access" dataset of the National Alzheimer’s Coordinating Center (NACC). We identified highly cited published research manuscripts using NACC data and selected five studies appeared reproducible using this dataset alone. Using GPT-4o, we created a simulated research team of LLM-based autonomous agents tasked with writing and executing code to dynamically reproduce the findings of each study, given only study Abstracts and Methods sections and data dictionary descriptions of the dataset. Results : We extracted 35 key findings described in the Abstracts across 5 Alzheimer’s studies. On average, LLM agents approximately reproduced 53.2% of findings per study. Numeric values and range-based findings frequently differed between studies and agents. Also, the LLM agents applied statistical methods or parameters that varied from the original studies, though overall trends and significance were sometimes similar. Discussion : In certain cases, LLM-based agents accurately replicated research techniques and findings. But in others, they failed because of implementation flaws or missing methodological information. These discrepancies demonstrate the current constraints of LLMs’ capacity to completely automate reproducibility evaluations. However, this preliminary investigation highlights the potential of structured agent-based systems to provide scale assessment of scientific rigor. Conclusion : This exploratory work demonstrates both the promise and the limitations of LLMs as autonomous agents for automating reproducibility in biomedical research. We demonstrate the potential for LLMs to validate, critique, and potentially reproduce published research. Journal: arXiv 1 Introduction In recent years, Large Language Models (LLMs) have demonstrated remarkable capabilities in many traditional biomedical natural language processing (NLP) tasks, such as information extraction [ 1 ] , ontology generation [ 2 ] , concept normalization [ 3 ] , as well as relatively recent, emerging tasks such as diagnostic reasoning [ 4 ] , radiology reporting [ 5 ] , cohort discovery [ 6 ] , code generation [ 7 ] and self-reflection and correction [ 8 ] . These tasks often require high degrees of reasoning, interpretation, and analysis of complex biomedical data. A subset of these tasks [ 7 , 8 ] also require a reasonable degree of planning, experimentation, engineering, and dealing with ambiguity in data or instructions. Taken together, these findings indicate that, under certain circumstances, modern LLMs can capably perform a number of tasks typically expected of human scientists. At the same time, the scientific community has been said to be undergoing a "reproducibility crisis" [ 9 ] , with an estimated 75% to 90% of published studies estimated to not be reproducible [ 10 ] . While the reasons for this are complex and not necessarily indicative of consistently poor study design or falsification of results, the inability of scientists to reproduce others’ findings in many cases stands as a clear challenge to future scientific progress. Moreover, within biomedical informatics (as well as other computationally-driven research domains), the process of reproducing published study results - reading a manuscript, acquiring study data, establishing an environment to conduct experiments, and often writing or reviewing computer code following study methods - is highly time- and resource-intensive, precluding large-scale study replication. We hypothesized that LLMs may potentially be applied to automate certain aspects of reproducibility and sought to explore their limitations and capabilities in this context. As LLMs model language - literally outputting sequences of text strings (or audio, images or video) - an additional external component is needed to enable interaction with datasets and code execution: the "wrapping" of an LLM for iterative, turn-based interaction and enablement of external tool usage, such as web search, code interpreters, application programming interfaces (APIs) and so on. Models equipped in such a fashion are called "agents" [ 11 , 12 ] . In this study, we explore the use of an LLM agent-based system for reproduction of highly cited peer-reviewed manuscripts using the National Alzheimer’s Coordinating Center’s (NACC) "Quick Access" dataset. NACC serves as a central repository and organizing hub for 35 Alzheimer’s Disease Research Centers (ADRCs) across the United States which collect detailed, high-quality data from tens of thousands of patients at risk for or with suffering from Alzheimer’s Disease (AD) and other neurological disorders. As NACC data have been widely used for research purposes for decades [ 13 ] and are freely available to research groups around the world, published studies using NACC data lend themselves in many ways to automated reproducibility. We aimed to explore whether a group of LLM agents assigned various roles and instructed to develop a plan, explore data, and write and execute code could reproduce published study findings on Alzheimer’s Disease. All code used in the study as well as transcripts of LLM agent responses, actions and code are available at https://github.com/ndobb/nacc_paper_llm_reproducer 1 1 1 Code will be made available upon article acceptance . 2 Background 2.1 LLM agents Autonomous agents have long been a research interest in the artificial intelligence and computer science communities [ 14 , 15 , 16 ] . Prior to widespread use of deep learning-based models from the 2010s, agentic systems were often designed as rule-based software programs driven by relatively simple heuristic policy functions aimed at determining an optimal sequence of actions toward some larger goal. Such systems typically depended on hand-written rule curation and syntactic parsing of natural language [ 17 ] , limiting their potential utility relative to humans in open-domain settings with evolving, complex objectives and various means of interacting with a given environment. More recently, LLMs have been increasingly studied as core engines for goal interpretation, user interaction, planning, action selection, data analysis and other components of potential agentic systems [ 18 ] . Exploration in the use of LLM agents has been done in various domains, including social simulation of daily life and cognitive development [ 19 , 20 , 21 ] , jurisprudence and legal interpretation [ 22 , 23 ] and research assistants [ 24 , 25 ] . Common features of these LLM agentic systems include capabilities for planning smaller sequences of steps to solve larger, more complex or abstract goals, storage and retrieval of past actions and observations as "memories" to be used in future decision-making, tool intergration (e.g., APIs or knowledge bases) and flexible response capabilities while interacting with a given environment. See Wang et al [ 18 ] for a detailed examination of these trends. 2.2 Automated Study Reproducibility Study replication is typically a highly time- and resource-intensive task, typically done manually. Individual research studies are, by design, generally unique and heterogeneous, necessitating experimental setup parameters, datasets, computing needs and more specific to a given research manuscript or project. While short of full automation, many researchers have nonetheless explored methods, representations, and tools aimed toward facilitating reproducibility. These efforts have tended to focus either on abstraction , the expression of methodology and steps in generalizable patterns, and source code execution to recreate experiment results in environments outside of the originating lab. In the biochemical and

Read the complete text