D3-Gym: Constructing Real-World Verifiable Environments for Data-Driven Discovery

Hanane Nour Moussa, Yifei Li, Zhuoyang Li, Yankai Yang, Cheng Tang, Tianshu Zhang, Nesreen K. Ahmed, Ali Payani, Ziru Chen, Huan Sun · 2026

Read D3-Gym: Constructing Real-World Verifiable Environments for Data-Driven Discovery by Hanane Nour Moussa, Yifei Li, Zhuoyang Li, Yankai Yang, Cheng Tang, Tianshu Zhang, Nesreen K. Ahmed, Ali Payani, Ziru Chen, Huan Sun (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-27977.txt.

Introduction

Moussa and colleagues present D3-Gym, a set of test tasks for agents doing real science work. Hundreds of tasks from real research code bases come with plain instructions, setups that run, sample answers and auto scoring that matches human marks closely. Training on runs from these tasks lifted scores on a science-agent test, closing much of the gap to top closed systems.

Catalog record

CREATORHanane Nour Moussa, Yifei Li, Zhuoyang Li, Yankai Yang, Cheng Tang, Tianshu Zhang, Nesreen K. Ahmed, Ali Payani, Ziru Chen, Huan Sun
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68406
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.27977

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

D3-Gym: Constructing Real-World Verifiable Environments for Data-Driven Discovery Hanane Nour Moussa, Yifei Li, Zhuoyang Li, Yankai Yang, Cheng Tang, Tianshu Zhang, Nesreen K. Ahmed, Ali Payani, Ziru Chen, Huan Sun Abstract Despite recent progress in language models and agents for scientific data-driven discovery, further advancing their capabilities is held back by the absence of verifiable environments representing real-world scientific tasks. To fill this gap, we introduce D3-Gym, the first automatically constructed dataset with verifiable environments for scientific D ata- D riven D iscovery. D3-Gym comprises (1) 565 tasks sourced from 239 real scientific repositories across four disciplines where (2) each task is equipped with a natural language instruction, an executable environment with pre-installed dependencies, input dataset and artifact previews, a reference code solution, and an automatically synthesized evaluation script. Rigorous evaluation of the quality of the verification signal in D3-Gym confirms that our evaluation scripts achieve 87.5% agreement with human-annotated gold standards and strong alignment in domain-specific evaluation logic, showing their scientific soundness. Further, training on trajectories sampled from D3-Gym yields consistent and substantial gains across Qwen3 models of varying sizes on ScienceAgentBench, boosting Qwen3-32B by 7.8 absolute points and substantially shrinking the gap with strong proprietary models. All D3-Gym artifacts (environments, creation workflow, trajectories, and models) can be found at https://github.com/OSU-NLP-Group/D3-Gym . (a) (b) Figure 1: (a) Overview of D3-Gym. (b) Success Rate (SR@3) on ScienceAgentBench (SAB) ( 6 ) of D3-Gym trained models based on Qwen3-14B and Qwen3-32B and other larger open-weight and proprietary reasoning models. 1 Introduction Language agents ( 30 ; 31 ) are beginning to reshape scientific research. Powered by large language models (LLMs), they can now meaningfully assist researchers in their workflows, from literature synthesis ( 28 ; 3 ) to domain-specific scientific reasoning ( 4 ; 18 ) . In particular, there has been significant research interest in developing language agents for data-driven discovery ( 9 ; 15 ) , where agents can programmatically test new hypotheses ( 16 ; 17 ) and derive scientific insights from data ( 6 ; 26 ; 27 ) . Despite this progress, further advancing these capabilities, particularly for open-weight LLMs ( 43 ) , is held back by the absence of suitable infrastructure for scientific tasks. A critical component in this infrastructure is real-world verifiable environments . Recent work such as Autoresearch ( 11 ) highlights their value: given a manually prepared task instruction, a dataset, an evaluation script, and a repository with pre-installed dependencies, coding agents can use the resulting executable feedback loop to improve the task performance and run dozens of experiments in a few hours for faster iteration. This result highlights how powerful task-specific verification signals can be for developing stronger models and agents. However, it also reveals a key bottleneck: constructing such environments currently requires substantial manual effort, making them difficult to scale beyond a small number of tasks. In this paper, we fill this gap by presenting an automated pipeline that constructs real-world, verifiable coding environments for data-driven discovery. Unlike software engineering, where unit tests already present in repositories can be directly repurposed as verification signals ( 37 ; 10 ; 21 ) , automatically constructing environments for scientific tasks is particularly challenging because related repositories rarely contain pre-existing evaluation scripts, and programs produce domain-specific outputs whose correctness cannot be assessed by any universal criterion. To address this challenge, we start from a large pool of candidate tasks collected using AutoSDT ( 13 ) and design a series of rigorous filtering steps, followed by task execution and output validation, to retain only the highest-quality and most ecologically valid task instances. Moreover, we address a key challenge, evaluation script generation, to create verifiable environments around these task instances. Specifically, we use LLMs to synthesize evaluation scripts in two stages: first writing a detailed, task-specific evaluation plan, and then translating that plan into an executable script to complete environment construction. The resulting dataset, D3-Gym, comprises 565 verifiable environments sourced from 239 unique scientific repositories spanning bioinformatics, computational chemistry, geographic information science, and psychology and cognitive neuroscience ( 1(a) ). Each instance includes an executable environment with pre-installed dependencies, input data files and their previews, a natural language task instruction, a reference solution adapted from the original repository, and a task-specific evaluation script with appropriate metrics and acceptance criteria. To the best of our knowledge, D3-Gym is the first automatically constructed dataset with verifiable environments for scientific data-driven discovery. Through comprehensive experiments, we demonstrate the quality of synthesized evaluation scripts and the utility of D3-Gym in improving open-weight LLMs. On a set of 50 tasks with human-annotated gold evaluation scripts, our synthesized evaluation scripts achieve 87.5% agreement on pass/fail verdicts and closely match the underlying evaluation logic, confirming that they are both accurate and methodologically sound. Furthermore, rejection sampling fine-tuning ( 38 ) on D3-Gym substantially improves the Qwen3 family models ( 32 ) on ScienceAgentBench ( 6 ) . Notably, our 32B model improves its success rate over three trials by 7.8 absolute points, surpassing Qwen3-235B-A22B and approaching strong proprietary models such as OpenAI o1 ( 1(b) ). Overall, D3-Gym provides a scalable foundation for training and evaluation for data-driven discovery in real-world, verifiable environments. 2 Constructing Verifiable Environments Figure 2: Overview of the D3-Gym construction workflow. Candidate tasks from scientific repositories ( 13 ) undergo quality filtering and dataset preview creation, then execution and validation to obtain reference outputs. These provide the context needed to generate high-quality, task-specific evaluation scripts via a planning-then-coding approach. As shown in Figure 2 , we construct D3-Gym through a four-stage workflow. The first three stages (§ 2.1 ) collect candidate tasks from scientific repositories, filter them for data integrity, and execute them to obtain reference outputs. These stages serve the purposes of ensuring task quality and produce the essential context that enables the core contribution of our workflow: automated generation of task-specific evaluation scripts with scientifically grounded metrics and acceptance criteria (§ 2.2 ). We validate the reliability of these scripts against human-annotated gold standards in § 2.3 . Further details are in Appendix C . 2.1 Task Pre-processing Candidate Task Collection. We collect an initial pool of candidate tasks using AutoSDT ( 13 ) , which crawls scientific research repositories from GitHub, applies multi-step filtering to identify files implementing data-driven scientific workflows, adapts them into reference code solutions, and pairs each with a natural language task instruction. The pipeline also extracts a dependency folder for each task containing all file dependencies from the original repository. Expert evaluation confirms that 93% of collected tasks are scientifically meaningful ( 13 ) . We ensure that all repositories used in ScienceAgentBench ( 6 ) are excluded , as it serves as our evaluation set. Filtering and Dataset Preview Creation. A limitation of AutoSDT is that its code adaptation stage may substitute synthetic data when an input file is absent from the original repository ( 13 ) . Since our goal is to construct high-fidelity environments grounded in real scientific workflows, we discard such tasks with synthetic data and retain only those whose reference solutions operate exclusively on real data files present in the original repository. We implement this filtering using Claude Code ( 1 ) , which verifies

Read the complete text