CodeDistiller: Automatically Generating Code Libraries for Scientific Coding Agents
Peter A. Jansen, Samiah Hassan, Pragnya Narasimha · 2025
Read CodeDistiller: Automatically Generating Code Libraries for Scientific Coding Agents by Peter A. Jansen, Samiah Hassan, Pragnya Narasimha (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-01089.txt.
Introduction
Peter Jansen, Samiah Hassan, and Pragnya Narasimha present CodeDistiller in a 2025 paper on scientific coding agents. Automated scientific discovery systems generate and run code-based experiments on their own. Their reach is limited by the code they can reliably produce from built-in knowledge alone.
Current systems either mutate a few hand-crafted examples or work purely from what the model already knows. CodeDistiller automatically generates code libraries for such agents, distilling dependable routines they can reuse. A good library would raise both the quality and reach of automated experiments, letting agents attempt studies that free-form code generation cannot sustain.
Catalog record
| CREATOR | Peter A. Jansen, Samiah Hassan, Pragnya Narasimha |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-01089
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CodeDistiller: Automatically Generating Code Libraries for Scientific Coding Agents Peter A. Jansen, Samiah Hassan, Pragnya Narasimha Abstract Automated Scientific Discovery (ASD) systems can help automatically generate and run code-based experiments, but their capabilities are limited by the code they can reliably generate from parametric knowledge alone. As a result, current systems either mutate a small number of manually-crafted experiment examples, or operate solely from parametric knowledge, limiting quality and reach. We introduce CodeDistiller , a system that automatically distills large collections of scientific Github repositories into a vetted library of working domain-specific code examples, allowing ASD agents to expand their capabilities without manual effort. Using a combination of automatic and domain-expert evaluation on 250 materials science repositories, we find the best model is capable of producing functional examples for 74% of repositories, while our downstream evaluation shows an ASD agent augmented with a CodeDistiller generated library produces more accurate, complete, and scientifically sound experiments than an agent with only general materials-science code examples. 1 1 1 Video: http://youtu.be/RQxSGFbGZSc , Repository: www.github.com/cognitiveailab/codedistiller 1 Introduction Figure 1: CodeDistiller distills a large collection of Github repositories into a library of reusable scientific code, allowing Code-RAG style scientific discovery agents to perform tasks beyond their parametric knowledge. While automated scientific discovery has been explored for decades Simon et al. 1981 , particularly in the context of literature-based discovery Swanson 1986 ; Yang et al. 2024 and data-driven discovery Langley and Zytkow 1989 ; Majumder et al. 2024 , recent advances in code generation models ( Chen et al. 2021 ; Jiang et al. 2024 , e.g.) have spurred the development of experiment-driven discovery agents in domains such as computer science where experiments can be run primarily through code. Typically these agents are supplied with a research task, then must generate (and iteratively debug) code that runs a computational experiment before writing a report describing the results and conclusions, as shown in Figure 1 . These systems have recently been demonstrated to produce novel, if incremental, scientific discoveries Weng et al. 2025 ; Zhang et al. 2025 ; Jansen et al. 2025 . A central challenge is that scientific experiments typically require highly specific methods, measurements, and protocols, whereas coding agents are limited to generating experiment code using whatever parametric knowledge they acquired during training. Existing automated scientific discovery agents therefore either mutate pre-existing experiment code ( Lu et al. 2024 , e.g.) or require the experimenter to pre-generate a library of existing vetted code examples they can combine ( Jansen et al. 2025 , e.g.) . In this work we address this limitation by building a system that can automatically build a library of functional domain-specific code examples at scale, allowing systems to automatically increase their domain-specific capabilities while reducing their reliance on manual code construction. The contributions of this work are: 1. CodeDistiller , a system for automatically converting Github code repositories in specialized scientific domains into debugged, working examples suitable for incorporation in automated scientific discovery systems. 2. An evaluation in the materials science domain, showing that the best and worst performing base models are capable of successfully distilling between 26% to 74% of 250 repositories, with different price/performance tradeoffs. 3. A downstream evaluation showing that an automated discovery system augmented with a library of code automatically built with CodeDistiller generates more accurate, complete, and scientifically sound output than a baseline model with only general materials science code examples. 2 Related Work Automated Scientific Discovery: Agents for automated scientific discovery can be divided into those focusing on literature-based discovery ( Swanson 1986 ; Yang et al. 2024 , e.g.) , data-driven discovery ( Majumder et al. 2024 ; Mitchener et al. 2025 , e.g.) , and experiment-driven discovery—the latter being the focus of this work. Several agents make use of solely computational experiments for discovery. AI Scientist Lu et al. 2024 mutates the code of existing experiment repositories to create novel experiments. AgentLab Schmidgall et al. 2025 generates experiment code after reviewing relevant literature. Some approaches rely on expert-generated tool interfaces to domain-specific tools, such as in chemistry ( Bran et al. 2024 , ChemCrow ;) or biology ( Huang et al. 2025 , Biomni ;) . CodeDistiller addresses the approach taken by CodeScientist Jansen et al. 2025 , where a Code-RAG agent retrieves multiple relevant code examples from a vetted code library that it can combine to build complex computational experiments, and reduce reliance on parametric knowledge. While the 6 discoveries made by CodeScientist relied on a code library built through a combination of manual generation and expert curation of LLM-generated examples, CodeDistiller allows for augmenting Code-RAG libraries through automatically generated and vetted examples. Tasks: Several existing tasks are similar to the example distillation task. The SUPER benchmark Bogin et al. 2024 evaluates agents’ capacity to set up and replicate specific research results from a Github repository, with current best performance at 16%. RexBench Edwards et al. 2025 and TM-Bench Wölflein et al. 2025 demonstrate automatically repurposing 12 and 15 research repositories (respectively) from Papers-With-Code to new research tasks. ENVBENCH Eliseeva et al. 2025 addresses LLM configuration of development environments. ResearchCodeBench Hua et al. 2025 evaluates whether LLMs can implement 212 coding challenges drawn from 20 research articles using only parametric knowledge, finding the best model achieved less than 40% success, supporting the notion that distilling examples from repositories may increase recall. GISTIFY Lee et al. 2025 requires generating fully self-contained examples from 5 Github repositories whose output passes sets of predefined pytests . In contrast, CodeDistiller focuses on building a code library for CodeRAG -style automated discovery systems, which can require integrating code across multiple repositories. CodeDistiller does not require repositories to contain unit tests, and is evaluated at greater scale than the above systems—both using automatic LLM-as-a-judge metrics Zheng et al. 2023 , as well as human domain expert evaluation in our domain of interest (materials science). Figure 2: An overview of the core stages of the CodeDistiller workflow, including identifying the core purpose of the repository, identifying files relevant for building an example, and the example generation and debugging process. Agent Base Model Metric GPT-OSS-120B GPT-5 Claude 4-5 Overall Performance (Auto) Successfully Completed (LLM-as-a-judge) 61.6% 70.4% 75.6% (Manual) Code Executed without Error \dagger 29.6% 69.0% 75.6% (Manual) Demonstrates Repository Functionality \dagger 29.6% 69.0% 75.6% (Manual) Correct Functionality \dagger 25.9% 60.5% 74.1% Runtimes Avg. Runtime (successful cases) 13.8 mins 20.3 mins 19.0 mins Avg. Runtime (unsuccessful cases) 16.3 mins 47.0 mins 21.9 mins Avg. Debug Iterations (successful cases) 2.4 2.2 1.9 Costs Avg. Cost (successful cases) $0.09 $0.70 $1.71 Avg. Cost (unsuccessful cases) $0.13 $1.68 $2.26 Table 1: Summary statistics of CodeDistiller performance as a function of using different base models. \dagger Manual analysis was completed by a domain expert on a subsample of 50 repositories that were marked as successfully completed by the agent, and normalized to have a common denominator with automatic results. 3 System Overview The CodeDistiller workflow is shown in Figure 2 . First, the core purpose of the repository is determined, followed by identifying code, documentation, and other files relevant to that purpose. Then, those highly-relevant files are used to generate and iteratively debug a working example of the core repository functionality. 3.1 Purpose