Can we automatize scientific discovery in the cognitive sciences?
Akshay K. Jagadish, Milena Rmus, Kristin Witte, Marvin Mathony, Marcel Binz, Eric Schulz · 2026
Read Can we automatize scientific discovery in the cognitive sciences? by Akshay K. Jagadish, Milena Rmus, Kristin Witte, Marvin Mathony, Marcel Binz, Eric Schulz (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-20988.txt.
Introduction
Jagadish and co-authors chart what an automated science of the mind would take. AI tools now begin to automate parts of scientific discovery, including in psychology. They frame the field around four challenges. First, experiments must be represented in a form machines can use. Second, synthetic behavior must be generated well enough to stand in for human data. Third, models of that behavior must be built and compared. Fourth, the loop must close, so the system can move from data and models to new psychological theories.
Catalog record
| CREATOR | Akshay K. Jagadish, Milena Rmus, Kristin Witte, Marvin Mathony, Marcel Binz, Eric Schulz |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 18744 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.20988
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Can we automatize scientific discovery in the cognitive sciences? Akshay K. Jagadish, Milena Rmus, Kristin Witte, Marvin Mathony, Marcel Binz, Eric Schulz Abstract The cognitive sciences aim to understand intelligence by formalizing underlying operations as computational models. Traditionally, this follows a cycle of discovery where researchers develop paradigms, collect data, and test predefined model classes. However, this manual pipeline is fundamentally constrained by the slow pace of human intervention and a search space limited by researchers’ background and intuition. Here, we propose a paradigm shift toward a fully automated, in silico science of the mind that implements every stage of the discovery cycle using Large Language Models (LLMs). In this framework, experimental paradigms exploring conceptually meaningful task structures are directly sampled from an LLM. High-fidelity behavioral data are then simulated using foundation models of cognition. The tedious step of handcrafting cognitive models is replaced by LLM-based program synthesis, which performs a high-throughput search over a vast landscape of algorithmic hypotheses. Finally, the discovery loop is closed by optimizing for “interestingness”, a metric of conceptual yield evaluated by an LLM-critic. By enabling a fast and scalable approach to theory development, this automated loop functions as a high-throughput in-silico discovery engine, surfacing informative experiments and mechanisms for subsequent validation in real human populations. Introduction The cognitive sciences aim to understand intelligence using computational models [ 6 ] . These models formally ground theories about cognitive processes by specifying interpretable operations and dynamics underlying human reasoning, learning, and decision-making [ 18 ] . The standard model of discovery in the cognitive sciences typically follows a virtuous cycle of four stages [ 10 ] . First, researchers develop a paradigm to test a particular feature of human cognition. Second, a sufficient volume of data is collected from human participants. Third, a predefined class of models is handcrafted and tested against the collected data, which corresponds to targeted hypothesis testing of the involved cognitive processes. Finally, the results of the experiment are reviewed by the scientific community, who then innovate on the paradigm, model class, and model comparisons involved. While this process has led to general progress in our understanding of cognition, it remains constrained by several structural disadvantages. The primary constraint is the inherent slowness of a cycle that relies on human involvement at every step, where the time elapsed between initial hypothesis and final publication can span years and thus decouple the pace of theory development from the velocity of data acquisition. Furthermore, this manual pipeline restricts the space of hypothesis that gets explored, which is often restricted to familiar experimental designs and model classes rather than the vast landscape of all possible paradigms and algorithmic solutions. This reliance on researcher-driven designs introduces subtle biases and limits the scalability of experimentation and theory development. These limitations raise the question: can we remove the bottlenecks imposed by human intervention and genuinely scale up scientific discovery in the cognitive sciences? [ 9 , 11 ] We argue that we should seriously consider accelerating the discovery process by automating the steps involved in the process [ 13 ] . We are currently at a critical inflection point where models capable of automating each step of the standard discovery cycle are becoming available [ 3 ] . In the following sections, we examine how the four constituent stages of this cycle, i.e., proposing experiments, generating data, synthesizing models, and closing the loop through iterative refinement, can be implemented in silico. By enabling a new approach that is remarkably fast, scalable, and widely applicable, an automated science of the mind is no longer a distant fiction, but an imminent reality. Figure 1: An automated cycle of scientific discovery in the cognitive sciences. An experimentalist proposes experiments, a cognitive foundation model generates behavioral data. A modeller proposes and tests different computational models. Finally, a critic judges the “interestingness” of the results and parsed that signal back to the experimentalist for further optimization. Proposing experiments Automated discovery begins by defining the space of possible experiments. If machine algorithms are to generate new paradigms rather than merely optimize existing ones, a formal language for describing experimental structures and how their constituent components can be combined is required. A natural starting point are generative grammars over experiments [ 14 ] . For instance, a grammar over Markov Decision Processes (MDPs) [ 12 ] could provide an initial foundation. Because MDPs are formally defined by environmental states, transition dynamics, and reward structures, they can capture a vast class of decision-making tasks used in psychology and neuroscience, ranging from simple multi-armed bandits [ 19 ] to multi-step planning paradigms [ 8 ] . Yet, this approach immediately exposes a representational bottleneck: automated discovery is fundamentally constrained by the expressivity of its underlying grammar. Ideally, the task language should be able to express a broad range of existing cognitive experiments while allowing flexible integration of new experiments with previously unexplored factors. Building such a language remains a key scientific challenge. We propose that a promising strategy in the future is to use an LLM itself as an intelligent experiment sampler. In this framework, an LLM directly proposes a set of experiments that are best positioned to elicit a specific type of behavior or test a specific hypothesis, and critically, it can further refine them based on feedback, adding or removing key experimental factors as needed. Generating data Once a task is specified, the cycle proceeds to the generation of synthetic behavioral data. This is achieved using recent iterations of the Centaur model [ 2 ] , a foundation model of human cognition. These models can generate data for any cognitive experiment that can be expressed in language format. For example, MDP-like tasks can be described as a task in which the agent needs to choose between multiple options repeatedly to maximize rewards. Once the agent made a choice, the reward of the chosen option gets generated, the choice and its outcome are appended to the trial history that is then re-submitted as a new prompt to generate the next choice, and so forth [ 5 ] . Unlike traditional cognitive models that require manual parameter tuning, foundation models, like Centaur, can generate de novo data that are often indistinguishable from human behavior. While the ability of these models to generalize to entirely novel experimental structures remains a subject of ongoing debates, current evidence suggests they can (at least to some extent) successfully extrapolate to tasks outside their initial training distribution [ 4 ] . In recent efforts, we aim to scale the underlying datasets for these models by an order of magnitude and integrate extensive subject-specific metadata. By incorporating demographics (e.g., age, gender, nationality) and questionnaire scores (e.g., psychiatric symptom scales), the data generation process can be conditioned on specific individual differences. This allows the system to simulate not just a generic agent, but a targeted participant profile, for instance, simulating the behavior of a 30-year-old individual with high scores in obsessive-compulsive traits performing a specific planning task. By parsing both the proposed experiment and these subject-specific features into the generative engine, we can produce high-fidelity behavioral data that reflects the high-dimensional complexity of real-world populations. This data can the be submitted to the next