AInsteinBench: Benchmarking Coding Agents on Scientific Repositories
2025
Read AInsteinBench: Benchmarking Coding Agents on Scientific Repositories (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-21373.txt.
Introduction
Titouan Duston and a large team of co-authors introduce AInsteinBench, a benchmark for testing whether language model agents can do real scientific software work. Existing benchmarks mostly test textbook reasoning or ordinary coding tasks. AInsteinBench instead places agents inside real scientific software projects and asks them to carry out full development work.
Its tasks come from real pull requests written by the maintainers of six widely used scientific codebases, covering quantum chemistry, quantum computing, molecular dynamics, numerical relativity, fluid dynamics, and cheminformatics. Each task passed multi-stage filtering and expert review to confirm it is scientifically hard, well tested, and fairly graded. Because the tasks run in executable environments with test-driven checks, the benchmark exposes real failure modes. It measures whether a model can move past surface-level code generation toward the skills that research actually requires.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 104 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-21373
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AInsteinBench: Benchmarking Coding Agents on Scientific Repositories Abstract We introduce AInsteinBench , a large-scale benchmark for evaluating whether large language model (LLM) agents can operate as scientific computing development agents within real research software ecosystems. Unlike existing scientific reasoning benchmarks which focus on conceptual knowledge, or software engineering benchmarks that emphasize generic feature implementation and issue resolving, AInsteinBench evaluates models in end-to-end scientific development settings grounded in production-grade scientific repositories. The benchmark consists of tasks derived from maintainer-authored pull requests across six widely used scientific codebases, spanning quantum chemistry, quantum computing, molecular dynamics, numerical relativity, fluid dynamics, and cheminformatics. All benchmark tasks are carefully curated through multi-stage filtering and expert review to ensure scientific challenge, adequate test coverage, and well-calibrated difficulty. By leveraging evaluation in executable environments, scientifically meaningful failure modes, and test-driven verification, AInsteinBench measures a model’s ability to move beyond surface-level code generation toward the core competencies required for computational scientific research. † † affiliation: ByteDance Seed † † affiliation: Princeton University † † contribution: Full author list in Contributions † † Project Page: https://github.com/ByteDance-Seed/AInsteinBench Figure 1 : Overall performance score across frontier models among all difficult tiers 1 Introduction Modern scientific discovery is increasingly mediated by large-scale computational systems. Across disciplines such as chemistry, physics, and materials science, new scientific insights are no longer derived solely from analytical derivations or isolated simulations, but from sustained interaction with complex scientific software ecosystems. Researchers routinely formulate hypotheses, encode physical or mathematical assumptions into simulation code, and iteratively refine implementations based on empirical feedback from tests, solvers, and large-scale simulation runs. As illustrated in Figure 2 , such a workflow follows an iterative cycle of reading literature, experimenting, and solving questions , with active interactions with the environment: the academic institution and compute infrastructure. In this setting, scientific discoveries depend not only on the literal understanding of the subject, but also on the ability to implement the theory into production codebases and execute in mass production. Recent advances in large language models (LLMs) have growing interest in their potential to assist scientific computing research [ 1 , 2 , 3 ] . To function as genuine research assistants, LLM-based agents must be able to navigate long-lived scientific repositories, understand domain-specific invariants and numerical constraints, and implement or debug code in ways that preserve scientific correctness. Whether current models are capable of the transition from surface-level scientific reasoning toward more analytic and physically based reasoning remains an open question. As shown in Fig. 2 , analogous to human researchers, the LLM agents engage in a workflow of loading files into context, executing bash commands, and committing changes, with active interaction with its environment: usually a Docker container we built. Current scientific reasoning benchmarks evaluate high-level conceptual understanding [ 4 , 5 , 6 , 7 , 8 ] or isolated coding questions [ 9 ] , but they stop short of requiring LLMs to navigate, interpret, and modify real scientific repositories. As a result, these benchmarks fundamentally underrepresent the skills needed to be a successful scientist. Across nearly all of academia, there are numerous groups whose primary workflow entails developing ideas within large scientific repositories. This is no better demonstrated than by looking through the author lists of popular scientific repositories ( AMReX ; astropy ; EinsteinToolkit ; pyscf ; qchem6 ; terrachem ; openmm ; qiskit ; rdkit ) . Existing SWE benchmarks ( jimenez2024swebench ; zan2025multiswebench ) are not much better, as they do not reflect the coding paradigms, numerical stability constraints, cross-language heterogeneity, and HPC execution environments that dominate scientific computing. Although SWE-bench includes one scientific repository, Astropy, this is by no means a representative sampling. As a result, a fundamental gap remains: no benchmark currently measures an LLM’s ability to solve issue or write new features through patch synthesis inside real, large-scale scientific software ecosystems. Figure 2 : Scientist vs LLM agent workflow In computational sciences like physics, chemistry and biology, the daily work of a graduate student and postdoc usually consists of such a process: Reading papers and textbooks for understanding of both science and coding; Experimenting with computer simulations or actual machines or laboratories to gain feedback and iterate; Solving problems or implementing new features to a project or the computational code repository it self. Throughout the process the scientist would actively interact with the computing allocations and the school/laboratory resources. On the other hand, in this paper, we setup docker container with terminal access as an environment for the LLM agent to interact with. Allowing the LLM agent to view files , execute bash commands, and commit their changes. Through this processes, the human researches and the LLM agents contribute their ability to solve real-world problems in scientific computing projects across different disciplines including RDKit, Qiskit, EinsteinToolkit, PySCF, OpenMM, and AMReX. To address this gap, we introduce AInsteinBench , a new benchmark designed to evaluate performance in scientific computing development as practiced in real research environments . Rather than extending existing software-engineering benchmarks, AInsteinBench is constructed to measure how well an agent can operate as a scientific researcher working through code: developing, diagnosing, and maintaining computational implementations that embody modern scientific theory. The benchmark is grounded in large, production-grade scientific repositories spanning quantum chemistry (PySCF), quantum computing (Qiskit), cheminformatics (RDKit), fluid dynamics (AMReX), numerical relativity (Einstein Toolkit), and molecular dynamics simulation (OpenMM). Unlike standard software projects, these codebases encode domain-specific scientific structure: from density-functional theory and self-consistent field formulations to symplectic integration, constrained optimization, and PDE discretizations. Successfully modifying such systems requires reasoning not only about code, but about the scientific assumptions, invariants, and numerical behavior. By curating tasks from these environments, AInsteinBench moves towards whether models can translate scientific understanding into correct and stable computational practice. This focus aligns the benchmark with the realities of scientific programming, where progress is measured by preserving physical correctness and numerical robustness. We construct this benchmark through a rigorous multi-phase pipeline inspired by SWE-bench but extended to meet scientific reproducibility standards. First, we identify high-impact scientific repositories with active developer communities and well-maintained test suites, ensuring both scientific relevance and executability. Second, we collect scientifically meaningful issue-linked pull requests and extract before/after code states, environment specifications, and domain-specific metadata. Third, we build fully containerized execution environments, capturing dependencies such as MPI, domain-specific compilers, and scientific development stacks. Fourth, we apply a strict filtering criterion which isolate issues that exhibit clear numerical or logical bugs with reproducible failing tests. Finally, we perform domain-expert manual verification to eliminate unsuitable instances. Contributions. Our contributions are threefold: • We formalize scientific computing as a research capability and introduce AInsteinBench , a benchmark that evaluates whether LLM agents can translate scientific understanding into correct and numerically stable computational practice. • The dataset is explicitly difficulty-calibrated and spans tasks ranging from the implementation of new computational modules requiring scientific and mathematical reasoning to the identification and correction of known repository issues, including numerical instabilities, solver non-convergence, violated invariants, and cross-language integration errors. • We provide a systematic evaluation of state-of-the-art models, revealing that while current agents can resolve isolated bugs, they struggle with preserving scientific invariants, coordinating multi-file changes, and maintaining correctness in complex scientific algorithms. 2 Related Work Benchmarks for mathematical and