AutoSDT: Scaling Data-Driven Discovery Tasks Toward Open Co-Scientists

Yifei Li, Hanane Nour Moussa , Ziru Chen, Shijie Chen, Botao Yu,, Mingyi Xue, Benjamin Burns, Tzu-Yao Chiu, Vishal Dey, Zitong Lu,, Chen Wei, Qianheng Zhang, Tianyu Zhang, Song Gao, Xuhui Huang,, Xia Ning, Nesreen K. Ahmed, Ali Payani, Huan Sun · 2025

Read AutoSDT: Scaling Data-Driven Discovery Tasks Toward Open Co-Scientists by Yifei Li, Hanane Nour Moussa , Ziru Chen, Shijie Chen, Botao Yu,, Mingyi Xue, Benjamin Burns, Tzu-Yao Chiu, Vishal Dey, Zitong Lu,, Chen Wei, Qianheng Zhang, Tianyu Zhang, Song Gao, Xuhui Huang,, Xia Ning, Nesreen K. Ahmed, Ali Payani, Huan Sun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-08140.txt.

Introduction

Yifei Li and co-authors auto-build practice tasks for AI science helpers. Their 2025 paper tackles a data drought: co-scientist models need many real coding tasks from data-driven discovery, and few exist. Their pipeline, AutoSDT, uses language models to hunt sources, pick lifelike tasks, and write sound instructions plus code answers. The yield is a set of thousands of tasks across four science fields and hundreds of Python tools.

Experts judged the bulk of sampled tasks lifelike and the bulk of programs correct. Models trained on the set leapt on two discovery benchmarks, with the largest open model matching a top closed rival on one. Code and data are open.

Catalog record

CREATORYifei Li, Hanane Nour Moussa , Ziru Chen, Shijie Chen, Botao Yu,, Mingyi Xue, Benjamin Burns, Tzu-Yao Chiu, Vishal Dey, Zitong Lu,, Chen Wei, Qianheng Zhang, Tianyu Zhang, Song Gao, Xuhui Huang,, Xia Ning, Nesreen K. Ahmed, Ali Payani, Huan Sun
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-08140

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AutoSDT: Scaling Data-Driven Discovery Tasks Toward Open Co-Scientists Yifei Li, Hanane Nour Moussa , Ziru Chen, Shijie Chen, Botao Yu,, Mingyi Xue, Benjamin Burns, Tzu-Yao Chiu, Vishal Dey, Zitong Lu,, Chen Wei, Qianheng Zhang, Tianyu Zhang, Song Gao, Xuhui Huang,, Xia Ning, Nesreen K. Ahmed, Ali Payani, Huan Sun Abstract Despite long-standing efforts in accelerating scientific discovery with AI, building AI co-scientists remains challenging due to limited high-quality data for training and evaluation. To tackle this data scarcity issue, we present AutoSDT, an automatic pipeline that collects high-quality coding tasks in real-world data-driven discovery workflows. AutoSDT leverages the coding capabilities and parametric knowledge of LLMs to search for diverse sources, select ecologically valid tasks, and synthesize accurate task instructions and code solutions. Using our pipeline, we construct AutoSDT-5K, a dataset of 5,404 coding tasks for data-driven discovery that covers four scientific disciplines and 756 unique Python packages. To the best of our knowledge, AutoSDT-5K is the only automatically collected and the largest open dataset for data-driven scientific discovery . Expert feedback on a subset of 256 tasks shows the effectiveness of AutoSDT: 93% of the collected tasks are ecologically valid, and 92.2% of the synthesized programs are functionally correct. Trained on AutoSDT-5K, the Qwen2.5-Coder-Instruct LLM series, dubbed AutoSDT-Coder, show substantial improvement on two challenging data-driven discovery benchmarks, ScienceAgentBench and DiscoveryBench. Most notably, AutoSDT-Coder-32B reaches the same level of performance as GPT-4o on ScienceAgentBench with a success rate of 7.8%, doubling the performance of its base model. On DiscoveryBench, it lifts the hypothesis matching score to 8.1, bringing a 17.4% relative improvement and closing the gap between open-weight models and GPT-4o. 1 1 1 Our code and data are publicly available at https://osu-nlp-group.github.io/AutoSDT/. 1 Introduction Accelerating scientific research and development has long been an aspirational goal in AI research ( Langley et al. 1987 ; Waltz and Buchanan 2009 ) . Since the 1960s, there have been continuous efforts in developing AI methods for scientific discovery, e.g., by constructing massive knowledge bases ( Buchanan and Feigenbaum 1978 ) , designing expert rules and heuristics ( Langley et al. 1981 ) , and learning representations from large-scale data ( Jumper et al. 2021 ) . Nonetheless, they are tailored for very specific tasks with constrained solution spaces. The realization of an AI assistant for open-ended scientific discovery still seems distant. Figure 1: Performance of our AutoSDT-Coder in comparison to other open-weight and proprietary LLMs on ScienceAgentBench. AutoSDT-Coder-32B achieves the same level of performance as GPT-4o (2024-05-13). Recently, large language models (LLMs) bring new light to fulfill this ambition and have piqued significant interest in building “AI co-scientists” ( Boiko et al. 2023 ; Gottweis et al. 2025 ) that assist in scientific discovery. Specifically, due to their digital nature, most AI co-scientist agents focus on data-driven discovery workflows ( Hey et al. 2009 ; Majumder et al. 2024 ) , including scientific computation and analysis ( Tian et al. 2024 ; Chen et al. 2025 ) , symbolic regression ( Shojaee et al. 2025a ; Shojaee et al. 2025b ) , and hypothesis generation ( Majumder et al. 2025 ; Mitchener et al. 2025 ) . While some agents have shown promising results ( Gottweis et al. 2025 ; Lu et al. 2024 ) , they mostly rely on proprietary LLMs, which impedes their adoption in subjects that require transparency and data privacy, such as social science ( Étienne Ollion et al. 2024 ) and medicine ( Zhang et al. 2024 ) . Thus, there is a strong need for AI co-scientists powered by open-weight LLMs. To build AI co-scientists, one critical bottleneck is the absence of large-scale, high-quality data for training and evaluation. Commonly formulated as code generation problems ( Chen et al. 2025 ; Majumder et al. 2025 ; Mitchener et al. 2025 ) , data-driven discovery tasks require AI agents to derive scientific insights by processing, analyzing, and visualizing data. On one hand, automatically mining data-driven discovery tasks is particularly challenging. Unlike software engineering tasks ( Jimenez et al. 2024 ) , which can often be extracted from code changes in pull requests, data-driven discovery tasks require complete, file-level code that operates on real-world scientific datasets and solves domain-specific problems , which cannot be directly crawled from existing code repositories. On the other hand, manual task annotation is quite time-consuming. It takes trained graduate students at least 2.5–3 hours to annotate one task ( Chen et al. 2025 ; Majumder et al. 2025 ) , not to mention extra time for paper searching and task validation. In this paper, we present AutoSDT, a pipeline for auto matically s caling d ata-driven discovery t asks to tackle the data scarcity issue from three aspects. (1) Source Diversity : Our pipeline overcomes the lack of source diversity in manually annotated datasets ( Chen et al. 2025 ; Majumder et al. 2025 ) by using LLM-based query augmentation to systematically search for code repositories that contain data-driven discovery tasks. (2) Task Ecological Validity : We exploit LLMs’ parametric knowledge to locate programs that resemble real-world data-driven discovery tasks and generate scientifically accurate task instructions. (3) Code Quality : Each selected program goes through multiple rounds of adaptation and validation by an LLM to ensure standalone executability and functional equivalence to the original code. We use our pipeline to create AutoSDT-5K, a dataset of 5,404 data-driven discovery tasks, which costs only 0.55 USD per task on average. To our best knowledge, AutoSDT-5K is so far the largest coding dataset for data-driven discovery, covering 756 unique Python packages of computational tools in four disciplines: Bioinformatics, Computational Chemistry, Geographical Information Science, and Psychology and Cognitive Neuroscience. We also engage 9 subject matter experts from these disciplines, including Ph.D. students and professors, to examine a subset of 256 tasks. The experts report that 93% of the tasks are scientifically authentic and represent parts of their data-driven discovery workflows, and 92.2% of the generated programs are deemed correct solutions to the tasks, validating the high quality of AutoSDT-5K. Figure 2: Our AutoSDT collects data-driven discovery tasks in three steps: (1) AutoSDT-Search generates a list of keywords for each discipline and searches for relevant repositories. (2) AutoSDT-Select identifies programs that represent data-driven discovery tasks and extracts their execution dependency folders. (3) AutoSDT-Adapt modifies the selected programs to be independently executable and generates their corresponding task instructions. Through comprehensive experiments, we further demonstrate the utility of AutoSDT-5K. We fine-tune Qwen2.5-Coder-Instruct ( Hui et al. 2024a ) on AutoSDT-5K and obtain AutoSDT-Coder, a series of LLMs with improved coding capabilities for data-driven discovery. We evaluate AutoSDT-Coder on two challenging data-driven discovery benchmarks, ScienceAgentBench ( Chen et al. 2025 ) and DiscoveryBench ( Majumder et al. 2025 ) . As shown in Figure 1 , AutoSDT-Coder-32B reaches the same level of performance as GPT-4o (2024-05-13) with a 7.8% success rate (SR) on ScienceAgentBench, double the performance of the base model (3.9% SR). On DiscoveryBench, AutoSDT-Coder-32B also brings a 17.4% relative improvement over its base LLM, lifting the hypothesis matching score from 6.9 to 8.1. These results illustrate how AutoSDT can propel the advancement toward open AI co-scientists by automatically scaling high-quality data-driven discovery tasks. 2 AutoSDT Since manual annotation requires extensive

Read the complete text