MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research
Hui Chen, Miao Xiong Yujie Lu Wei Han Ailin Deng, Yufei He Jiaying Wu Yibo Li Yue Liu Bryan Hooi · 2025
Read MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research by Hui Chen, Miao Xiong Yujie Lu Wei Han Ailin Deng, Yufei He Jiaying Wu Yibo Li Yue Liu Bryan Hooi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19955.txt.
Introduction
Hui Chen and co-authors wrote a 2025 paper on testing whether AI agents can do open-ended machine learning research. The catalog creator list runs the names together, but the head gives ten authors. Agents now draft ideas, run experiments, and write papers, yet no shared test measured how well they do each part.
Their benchmark holds two hundred one research tasks drawn from workshops at three top conferences. An automated judging system, using language models with careful rubrics, grades the work. A sample agent walks through four stages, from idea to finished paper. The team tested six leading models and a coding agent. The models wrote coherent papers, but the coding agent often failed to finish sound experiments.
Catalog record
| CREATOR | Hui Chen, Miao Xiong Yujie Lu Wei Han Ailin Deng, Yufei He Jiaying Wu Yibo Li Yue Liu Bryan Hooi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 113 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-19955
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research Hui Chen, Miao Xiong Yujie Lu Wei Han Ailin Deng, Yufei He Jiaying Wu Yibo Li Yue Liu Bryan Hooi Abstract Recent advancements in AI agents have demonstrated their growing potential to drive and support scientific discovery. In this work, we introduce MLR-Bench , a comprehensive benchmark for evaluating AI agents on open-ended machine learning research. MLR-Bench includes three key components: (1) 201 research tasks sourced from NeurIPS, ICLR, and ICML workshops covering diverse ML topics; (2) MLR-Judge , an automated evaluation framework combining LLM-based reviewers with carefully designed review rubrics to assess research quality; and (3) MLR-Agent , a modular agent scaffold capable of completing research tasks through four stages: idea generation, proposal formulation, experimentation, and paper writing. Our framework supports both stepwise assessment across these distinct research stages, and end-to-end evaluation of the final research paper. We then use MLR-Bench to evaluate six frontier LLMs and an advanced coding agent, finding that while LLMs are effective at generating coherent ideas and well-structured papers, current coding agents frequently (e.g., in 80% of the cases) produce fabricated or invalidated experimental results—posing a major barrier to scientific reliability. We validate MLR-Judge through human evaluation, showing high agreement with expert reviewers, supporting its potential as a scalable tool for research evaluation. We open-source MLR-Bench to help the community benchmark, diagnose, and improve AI research agents toward trustworthy and transparent scientific discovery. 1 Introduction A hallmark of human intelligence is the ability to generate new knowledge through research and scientific discovery. From basic research to applied technological development, scientific progress has shaped human civilization. Reproducing this ability—to autonomously generate, test, and validate new knowledge—has long been viewed as one of the grand challenges for artificial intelligence. Recent advances in large language models (LLMs) bring us closer to this vision. LLM-powered agents have demonstrated impressive capabilities in generating research ideas ( chainofideas2024 ; baek-etal-2025-researchagent ; si2025can ; li2024learning ) , conducting experiments and analyses ( huang2024mlagentbench ; wang2024openhands ; chan2025mlebench ; yang2024sweagent ; jimenez2024swebench ; yang2024swebenchmultimodal ; jing2025dsbench ) , drafting scientific articles ( radensky2024let ; wang2024autosurvey ; xi2025omnithinkexpandingknowledgeboundaries ; agarwal2025litllms ) and automated review ( weng2025cycleresearcher ; jin-etal-2024-agentreview ; thakkar2025llmfeedbackenhancereview ; darcy2024margmultiagentreviewgeneration ; ye2024yetrevealingrisksutilizing ) . These advances move us beyond automating isolated tasks, opening up the possibility of automating the entire scientific process, from idea conception to experimentation to dissemination. However, realizing this vision raises an important question: how do we rigorously evaluate the quality of research produced by AI agents? While recent works have shown promising results ( starace2025paperbenchevaluatingaisability ; seo2025paper2code ; lu2024ai ; schmidgall2025agent ; Yamada2025ai ) , the community still lacks a comprehensive benchmark to systematically assess AI agents’ ability to conduct open-ended scientific research, making progress toward autonomous scientific discovery difficult to measure and compare fairly between agents. Moreover, there is also a need for empirical analysis to identify key failure modes—such as hallucinated results, lack of novelty, or methodological flaws — to quantify current limitations and inform future progress in research agents. Figure 1: An overview of the framework of MLR-Bench, consisting of both an end-to-end evaluation (left) and a stepwise evaluation (right), each of which uses LLM judges to automatically assess performance over 201 tasks. For end-to-end evaluation, we use the same model as backbone in idea generation, proposal generation and paper writing. For stepwise evaluation, various models are tested and compared within each step. Toward this vision, we introduce MLR-Bench , a comprehensive benchmark designed for evaluating AI agents on open-ended machine learning research tasks . Specifically, we seek to answer the following research questions: • RQ1: How well can AI agents conduct open-ended machine learning research? (§ 3 ) • RQ2 : How effectively can an LLM judge evaluate research, as measured by its agreement with human reviewers? (§ 4 ) • RQ3: What are the key factors that affect the quality of AI-generated research? (§ 5 ) To systematically address these questions, we decompose the research process into four successive stages: (1) idea generation , (2) research proposal formulation , (3) experimental execution and analysis , and (4) paper writing . This decomposition allows us to assess both stepwise progress and the overall end-to-end research capability of AI agents. MLR-Bench consists of three key components: (1) a collection of 201 real-world research tasks sourced from NeurIPS, ICLR, and ICML workshops over the past three years, covering a wide range of machine learning domains, including LLMs, trustworthy AI, ML systems, AI for science; (2) MLR-Judge , an automated evaluation pipeline consisting of LLM-based judge and structured review rubrics; and (3) MLR-Agent , a modular research agent scaffold capable of automatically completing these tasks by following the four defined research stages. To explore RQ1, we use MLR-Bench to evaluate six state-of-the-art models—including the most recent o4-mini ( o4mini2025 ) , Gemini-2.5-pro-preview ( gemini2025 ) , and Qwen3-235B-A22B ( qwenteam2025qwen3 ) —as well as the advanced coding agent, Claude Code ( claude2025 ) . Our results show that while these models perform well in generating ideas and writing papers, they often fail to deliver innovative or scientifically reliable research, mostly due to coding agents producing invalidated experimental results. To validate the reliability of MLR-Judge (addressing RQ2), we recruit 10 machine learning experts with top-tier conference review experience to independently review the AI-generated research outputs. We find that the level of agreement between an LLM judge and a human judge, when assessing the same output, is very close to the level of agreement between two human judges, demonstrating its effectiveness as a reliable evaluation tool. Finally, to address RQ3, we analyze the justifications provided by both MLR-Judge and human reviewers, identifying a recurring failure mode where agents tend to generate fabricated or unverified results after execution failures—revealing a gap between fluent output generation and true scientific rigor. Our contributions include: • MLR-Bench : To our knowledge, the most comprehensive evaluation benchmark for AI research agents to date, featuring 201 open-ended ML research tasks, a human-aligned MLR-Judge for automated research quality assessment, and a modular MLR-Agent supporting both stepwise and end-to-end research execution. • Comprehensive Evaluation and Insights : We apply MLR-Bench to evaluate 6 frontier models, providing comparative insights into their strengths and limitations in scientific discovery. Our analysis highlights critical failure modes, including the widespread issue of fabricated experimental results, pointing to key challenges for future research. 2 MLR-Bench In this section, we describe the overall flow of MLR-Bench. Fig. 1 presents an overview of the framework of MLR-Bench, which consists of both an end-to-end evaluation pipeline to evaluate AI agents’ ability to complete end-to-end research, and a stepwise evaluation pipeline, which separately evaluates AI agents’ abilities on (1) Idea Generation, (2) Proposal Generation, (3) Experimentation, and (4) Paper Writing. Tasks. To measure the performance of AI agents across various machine learning topics, we collect 201 tasks from ICLR, ICML, and NeurIPS workshops over the past three years. These tasks are categorized into 9 core ML topics: LLMs and Vision Language Models (VLMs), AI for Science, ML Theory, Trustworthy AI, Computer Vision, ML Systems, Multimodality, Reinforcement Learning (RL), and other emerging topics. Fig. 2 illustrates the distribution of our tasks across these ML topics. To curate our