Neurodata Without Boredom: Benchmarking Agentic AI for Data Reuse

Ling-Qi Zhang, Kristin Branson · 2026

Read Neurodata Without Boredom: Benchmarking Agentic AI for Data Reuse by Ling-Qi Zhang, Kristin Branson (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12808.txt.

Introduction

Zhang and Branson benchmark agents on neuroscience data reuse. Neuroscience data are fragmented across labs and formats, and reuse demands manual effort. Flexible formats are rarely self-explanatory, while datasets demand documentation few labs sustain. Given data, code and paper, agents reformat data to train a neural decoder. The team pairs outcome metrics with a rubric scored by two evaluators. Agents succeed on subtasks but rarely produce an error-free solution, and judges are unreliable.

Catalog record

CREATORLing-Qi Zhang, Kristin Branson
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE128624
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.12808

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Neurodata Without Boredom: Benchmarking Agentic AI for Data Reuse Ling-Qi Zhang, Kristin Branson Abstract Neuroscience data are highly fragmented across labs, formats, and experimental paradigms, and reuse often requires substantial manual effort. A persistent roadblock to data reuse and integration is the need to decipher bespoke and diverse data formatting choices. Common data formats have been proposed in response, but the field continues to struggle with a fundamental tension: formats flexible enough to accommodate diverse experiments are rarely descriptive enough to be self-explanatory, and sufficiently descriptive formats demand detailed documentation and curation effort that few labs can sustain. Agentic AI is a natural candidate to solve this problem: LLMs read code and text faster and with sustained attention to the low-level details humans tend to skim over. To measure how well agentic AI performs on this task, we selected eight recent papers studying large-scale mouse neural population recordings that shared both data and code, spanning diverse recording modalities, behavioral paradigms, and dataset formats (e.g., NWB, specialized APIs, and general-purpose Python or MATLAB files). We provided agents with the data, code, and paper, and prompted them to load, understand, and reformat the data for a common downstream task: training a decoder from neural activity to task or behavioral variables. General-purpose coding agents commonly used by scientists performed well on each sub-task, but rarely strung together a fully error-free end-to-end solution. We characterize the types of mistakes agents made and the dataset properties that elicited them, and propose data-sharing best practices for the agentic-AI era. We further find that agents-as-judges are unreliable at catching errors, especially without ground-truth references, so interactive, human-in-the-loop coding remains necessary. 1 Introduction 1 1 footnotetext: Code available at https://github.com/kristinbranson/data-format Understanding the brain requires investigating a wide range of behaviors, neural circuits, and computations. Individual experiments necessarily probe only a narrow part of this space. This diversity of experimental approaches is compounded by the diversity of neural recording technologies, such as electrophysiology and calcium imaging. As a result, neuroscience data are inherently heterogeneous. At the same time, progress in neuroscience increasingly depends on integrating data across experiments, recording modalities, and laboratories. Theorists and computational neuroscientists routinely reanalyze published datasets to test models and hypotheses beyond the scope of the original study (e.g., [ 1 , 2 , 3 ] ). This need has become even more acute with recent interest in foundation-scale models of brain and behavior [ 4 , 5 , 6 ] , where preparing heterogeneous datasets into an appropriate format for downstream analysis has emerged as a bottleneck. Making neuroscience data reusable in the face of this heterogeneity has been the focus of substantial community effort, including standardization initiatives such as Neurodata Without Borders (NWB) [ 7 ] and consortium-scale curation projects such as the International Brain Lab (IBL) and the Allen Brain Observatory (ABO) [ 8 , 9 , 10 ] . These efforts have moved the field meaningfully closer to reusable data, but a fundamental tension persists. A format flexible enough to accommodate the full diversity of neuroscience experiments inevitably leaves room for ambiguity. For example, trials is an NWB-defined object for storing per-trial information, but the semantic meaning of its columns relies on user-defined column names, and contextual knowledge is still required to interpret fields such as trial_instruction or trial_outcome [ 11 ] . Fully specifying the semantic meaning of each data field, and how those fields are used in downstream analyses, requires detailed documentation. Producing and maintaining such documentation for data sharing imposes a burden that few individual labs can sustain. Even within consortia, where curation is centralized and resourced, the interfaces required to make data usable to outsiders are non-trivial to learn. The result is that shared data is often technically available but practically expensive to reuse. Recent progress in agentic AI offers a possible path forward. General-purpose coding agents are increasingly applied, typically with humans in the loop, to tasks that demand reading unfamiliar code, parsing heterogeneous data, and synthesizing information across long technical documents [ 12 , 13 ] . They can do so quickly and with apparent attention to low-level details that human readers may skim. These are exactly the capabilities that data reuse demands. If coding agents can reliably read a paper, inspect its data and code, and produce a faithful reformatting for a downstream analysis, the practical cost of reusing shared neuroscience data could drop substantially. To test this, we selected eight recent papers studying large-scale mouse neural population recordings that released both data and code. These datasets span a variety of recording modalities, behavioral paradigms, and storage formats, including NWB files, consortium APIs, and custom Python or MATLAB files. For each paper, we asked general-purpose coding agents (Claude Code and Codex) to study the released materials and reformat the data for a common downstream task: training a decoder from neural activity to task or behavioral variables selected from the original study. Evaluating these solutions requires more than checking whether the code runs. Many conversion decisions are under-specified, and multiple choices can be defensible. We therefore evaluated agents using both outcome-level metrics, such as dataset statistics and decoder performance, and a process-based rubric in which we manually scored the agents’ decisions for each subtask of the data conversion pipeline. We found that agents performed well on each individual subtask, but rarely strung together a fully error-free end-to-end solution. Many agent failures were subtle, superficially reasonable, and difficult to detect from aggregate metrics alone. Moreover, agents were unreliable when asked to judge these solutions themselves, suggesting that human-in-the-loop review remains necessary for scientific data reuse. Our contributions are the following: • We introduce a realistic benchmark framework for evaluating coding agents on scientific data reuse, spanning eight recently released neuroscience datasets with heterogeneous data formats, experimental paradigms, and recording modalities. • We evaluate two widely used coding agents across repeated runs using both outcome-level metrics and a process-based manual rubric. The rubric decomposes the data conversion process into domain-relevant subtasks, revealing failures and concerning decisions that aggregate output statistics alone can miss. • We characterize agent failure modes through detailed case studies, showing that many failures are subtle, superficially reasonable, and difficult to detect without careful inspection. These observations suggest practical guidelines for future human-in-the-loop agent workflows and for improving agent-oriented data-sharing practices. • We show that agents are unreliable judges of the quality of data conversion pipeline, especially without a ground-truth reference, supporting the continued need for human-in-the-loop review. 2 Related work Agentic AI benchmarks for science. A growing body of work benchmarks AI agents on scientific tasks. Closest to ours are SUPER [ 14 ] and CORE-Bench [ 15 ] , which prompt agents to install dependencies, run code, and recover a paper’s reported results. PaperBench [ 16 ] tasks the agent to replicate a paper without its code base. In our setting, we do not ask the agent to reproduce the paper’s analysis but to understand the data and code well enough to produce something new. The agent must do the conceptual work of reading the paper and code, but is not graded on whether the original code runs. MLE-bench [ 17 ] and MLAgentBench [ 18 ] evaluate agents on ML engineering

Read the complete text