ReX-MLE: The Autonomous Agent Benchmark for Medical Imaging Challenges These authors contributed equally.
Roshan Kenia, Xiaoman Zhang, Pranav Rajpurkar · 2025
Read ReX-MLE: The Autonomous Agent Benchmark for Medical Imaging Challenges These authors contributed equally. by Roshan Kenia, Xiaoman Zhang, Pranav Rajpurkar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-17838.txt.
Introduction
Roshan Kenia, Xiaoman Zhang, and Pranav Rajpurkar ask whether today's coding agents can handle real medical imaging work. They built ReX-MLE, a benchmark of 20 challenges drawn from major medical imaging contests. The tasks span many scan types and demand full workflows, from raw data to a finished submission.
The team tested leading agents paired with several large language models. Most entries landed near the bottom when ranked against human experts. The failures came from missing medical knowledge and weak engineering under tight time and compute limits. The benchmark gives builders a clear map of what domain aware agents still lack.
Catalog record
| CREATOR | Roshan Kenia, Xiaoman Zhang, Pranav Rajpurkar |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-17838
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ReX-MLE: The Autonomous Agent Benchmark for Medical Imaging Challenges These authors contributed equally. Roshan Kenia, Xiaoman Zhang, Pranav Rajpurkar Abstract Autonomous coding agents built on large language models (LLMs) can now solve many general software and machine learning tasks, but they remain ineffective on complex, domain-specific scientific problems. Medical imaging is a particularly demanding domain, requiring long training cycles, high-dimensional data handling, and specialized preprocessing and validation pipelines, capabilities not fully measured in existing agent benchmarks. To address this gap, we introduce ReX-MLE , a benchmark of 20 challenges derived from high-impact medical imaging competitions spanning diverse modalities and task types. Unlike prior ML-agent benchmarks, ReX-MLE evaluates full end-to-end workflows, requiring agents to independently manage data preprocessing, model training, and submission under realistic compute and time constraints. Evaluating state-of-the-art agents (AIDE, ML-Master, R & \& D-Agent) with different LLM backends (GPT-5, Gemini, Claude), we observe a severe performance gap: most submissions rank in the 0th percentile compared to human experts. Failures stem from domain-knowledge and engineering limitations. ReX-MLE exposes these bottlenecks and provides a foundation for developing domain-aware autonomous AI systems. 1 1 footnotemark: 1 keywords Autonomous Coding Agents, Medical Imaging, Benchmarking Figure 1: Performance of SOTA AI coding agents on ReX-MLE. 1 Introduction Recent advances in large language models (LLMs) have enabled autonomous coding agents capable of solving standard machine learning engineering (MLE) and software engineering tasks 25 ; 4 ; 39 . These frameworks indicate the potential for agents to operate as autonomous AI scientists in the future 10 . However, despite their success on general benchmarks, current agents break down when faced with complex, domain-specific scientific challenges 41 . Medical imaging represents a particularly demanding domain that exposes these limitations. Unlike standard computer vision benchmarks, medical imaging tasks involve high-dimensional and heterogeneous modalities, including 3D CT and MRI volumes, multi-parametric scans, and gigapixel pathology slices, which require specialized preprocessing, normalization, and augmentation strategies 32 . These processes often require the specialized insight of someone with substantial hands‑on experience. Training these models often requires multi-stage pipelines, long training cycles, and careful hyperparameter tuning, demanding meticulous workflow management 8 . Despite rapid progress in agent capabilities, existing evaluation frameworks remain poorly aligned with real-world scientific requirements. Current coding benchmarks 20 ; 4 ; 34 ; 28 tend to emphasize code generation or simplified ML pipeline construction, but overlook competition-grade metrics, assessment of actual model outputs under realistic training constraints, adherence to domain-expert preprocessing and validation standards, and comparison against documented winning strategies. Lacking these elements, prior benchmarks fail to surface the behaviors that prevent current agents from succeeding on real medical imaging tasks, such as early training termination, mismanagement of computational budgets, flawed preprocessing pipelines, and invalid evaluation procedures. To address these gaps, we construct ReX-MLE , a benchmark comprised of 20 diverse challenge tasks derived from 10 high‑impact medical imaging competitions. ReX-MLE spans 8 imaging modalities and multiple task types, including segmentation, detection, classification, image quality assessment, and generative enhancement. Within this environment, agents must independently handle the full scientific workflow, like preprocessing, model design, training, and evaluation, and are required to generate full submission‑ready predictions (e.g., NIfTI volumes, masks, and JSON detection files) rather than single textual responses. Our main contributions are: (i) We introduce ReX-MLE, a comprehensive benchmark for evaluating autonomous AI agents on domain-specific scientific challenges, derived from high-impact medical imaging competitions spanning segmentation, detection, classification, and generation. (ii) We establish a rigorous evaluation protocol that assesses agents’ abilities to generate full, submission-ready prediction files under strict computational and time constraints, mirroring the requirements of real-world scientific discovery. (iii) We conduct an extensive evaluation of leading autonomous systems, including ML-Master and AIDE, utilizing state-of-the-art foundation models (GPT-5, Gemini 3 Pro, and Claude 4.5 Sonnet). As shown in Figure 1 , our results reveal a significant performance disparity, with even the most advanced agents consistently ranking in the lowest percentiles relative to human experts, highlighting critical gaps in domain-specific engineering and process validity. 2 ReX-MLE 2.1 Benchmark Design To systematically evaluate AI agent capabilities on medical imaging tasks, we construct ReX-MLE , a curated benchmark of medical imaging competitions adapted from Grand Challenge 1 1 1 https://grand-challenge.org/challenges/ . Our benchmark is designed to cover diverse modalities, task types, and clinical applications while maintaining rigorous standards for data quality and evaluation protocols, enabling systematic comparison of agent performance against expert-built solutions. Challenge Selection Criteria. We select Grand Challenge tasks that meet standard requirements for data quality and reproducibility. Eligible challenges required substantial community participation (typically over 200 registrants or extensive submissions), open licensing that permits research use, and publicly released evaluation metrics and scripts. We included challenges with complete competition materials, including task descriptions and sample submissions. We ensured broad modality coverage and task diversity, spanning classification, detection, segmentation, regression, and image enhancement. Table 1: Overview of the 20 ReX-MLE challenges across 10 competitions. Challenge Modality Task Type USenhance 13 Ultrasound Image Generation LDCTIQA2023 24 CT Image Quality Assessment PANTHER-T1 2 MRI Tumor Segmentation PANTHER-T2 2 MRI Tumor Segmentation SEG.A 22 CTA Segmentation DENTEX 14 X-ray Detection TopCoW-CTA-Seg 38 CTA Segmentation TopCoW-CTA-Det 38 CTA Detection TopCoW-CTA-Cls 38 CTA Classification TopCoW-MRA-Seg 38 MRA Segmentation TopCoW-MRA-Det 38 MRA Detection TopCoW-MRA-Cls 38 MRA Classification PUMA-Track1-TissueSeg 31 Pathology Segmentation PUMA-Track1-NucleiDet 31 Pathology Nuclei Detection PUMA-Track2-TissueSeg 31 Pathology Segmentation PUMA-Track2-NucleiDet 31 Pathology Nuclei Detection ISLES’22 16 MRI Segmentation CellSeg 26 Microscopy Segmentation TopBrain-CTA-Seg 38 CTA Segmentation TopBrain-MRA-Seg 38 MRA Segmentation Benchmark Composition. ReX-MLE comprises 20 challenges across 10 major medical imaging competitions, covering diverse modalities, task types, and anatomical regions. Table 1 summarizes the benchmark composition. The dataset distribution requires agents to operate across highly heterogeneous data formats and clinical contexts, covering neurovascular imaging (TopCoW, SEG.A, ISLES’22, TopBrain), oncology (PANTHER, PUMA), microscopy (CellSeg), dentistry (DENTEX), and image quality enhancement and reconstruction (LDCT-IQA, USenhance). This diversity enforces generalization across anatomical structures and clinical objectives, providing a realistic testbed for autonomous medical model development. Challenge Adaptation. For each challenge, we prepare standardized materials following the MLE-bench framework ( 4 ) . Provided materials include a concise competition description, automated dataset-preparation scripts, sample submissions, and Python implementations of official evaluation metrics for fully local validation. Each task also includes a YAML metadata file defining challenge identifiers, task types, data paths, and grading parameters. All preparation and evaluation code is publicly released to support reproducibility and future extensions. Figure 2: Overview of ReX-MLE Task Categories. This figure illustrates the four distinct task types included in the benchmark: Segmentation, Detection, Classification, and Image Generation. 2.2 Baseline Agents AIDE ( 30 ) frames machine learning engineering as a code-space optimization problem and performs greedy tree search with iterative refinement. The agent proposes candidate solutions, executes them, and uses automated feedback to diagnose errors and incrementally correct code. ML-Master ( 25 ) integrates large-scale exploration and reflective reasoning through a Monte Carlo Tree Search (MCTS) framework. By maintaining multiple solution trajectories and adaptively balancing exploration of new strategies with exploitation of promising ones, ML-Master efficiently navigates complex search spaces. R \& D-Agent ( 39 ) employs a dual-agent design that separates conceptual exploration from implementation refinement. A Researcher module proposes strategic directions, while a Developer module resolves execution errors and iteratively improves the code, enabling diverse and resilient multi-trace