MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI
Bohan Lyu Yucheng Yang Siqiao Huang Jiaru Zhang Qixin Xu Xinghan Li, Xinyang Han Yicheng Zhang Huaqing Zhang Runhan Huang Kaicheng Yang, Zitao Chen Wentao Guo Junlin Yang Xinyue Ai Wenhao Chai Yadi Cao, Ziran Yang Kun Wang Dapeng Jiang Huan-ang Gao Shange Tang, Chengshuai Shi Simon S. Du Max Simchowitz Jiantao Jiao Dawn Song Chi Jin · 2026
Read MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI by Bohan Lyu Yucheng Yang Siqiao Huang Jiaru Zhang Qixin Xu Xinghan Li, Xinyang Han Yicheng Zhang Huaqing Zhang Runhan Huang Kaicheng Yang, Zitao Chen Wentao Guo Junlin Yang Xinyue Ai Wenhao Chai Yadi Cao, Ziran Yang Kun Wang Dapeng Jiang Huan-ang Gao Shange Tang, Chengshuai Shi Simon S. Du Max Simchowitz Jiantao Jiao Dawn Song Chi Jin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08678.txt.
Introduction
Bohan Lyu and many coauthors introduce MLS-Bench, a test of whether AI agents can invent better machine learning methods, not merely apply known ones. Each of its one hundred forty tasks across twelve domains asks an agent to improve one targeted part of a learning system, then prove the gain holds across settings and scales. A shared community platform keeps results comparable over time.
Current agents fall well short of reliably beating human designed methods, and routine tuning proves far easier than true invention. Extra search, computing, or context alone does not close the gap, because planning and validating claims need scientific insight. AI research gains a benchmark aimed squarely at discovery itself.
Catalog record
| CREATOR | Bohan Lyu Yucheng Yang Siqiao Huang Jiaru Zhang Qixin Xu Xinghan Li, Xinyang Han Yicheng Zhang Huaqing Zhang Runhan Huang Kaicheng Yang, Zitao Chen Wentao Guo Junlin Yang Xinyue Ai Wenhao Chai Yadi Cao, Ziran Yang Kun Wang Dapeng Jiang Huan-ang Gao Shange Tang, Chengshuai Shi Simon S. Du Max Simchowitz Jiantao Jiao Dawn Song Chi Jin |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.08678
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI Bohan Lyu Yucheng Yang Siqiao Huang Jiaru Zhang Qixin Xu Xinghan Li, Xinyang Han Yicheng Zhang Huaqing Zhang Runhan Huang Kaicheng Yang, Zitao Chen Wentao Guo Junlin Yang Xinyue Ai Wenhao Chai Yadi Cao, Ziran Yang Kun Wang Dapeng Jiang Huan-ang Gao Shange Tang, Chengshuai Shi Simon S. Du Max Simchowitz Jiantao Jiao Dawn Song Chi Jin Abstract Modern AI progress has been driven by ML methods that are generalizable across settings and scalable to larger regimes. As large language models demonstrate advanced capabilities in reasoning, coding, and engineering tasks, it is increasingly important to understand whether they can discover such methods rather than only apply existing ones. We introduce MLS-Bench, a benchmark for evaluating whether AI systems can invent generalizable and scalable ML methods. MLS-Bench contains 140 tasks across 12 domains, each requiring an agent to improve one targeted component of an ML system or algorithm and demonstrate that the improvement generalizes across controlled settings and scales. We find that current agents remain far from reliably surpassing human-designed methods, and that engineering-style tuning is easier for them than genuine method invention. We further study the effects of test-time scaling, adaptive compute allocation, and context provision on agents’ discovery performance, together with case studies of their behavior. Our analyses suggest that the bottleneck is not only in proposing new methods, but also in the scientific insight needed to plan, validate, and scale claims about them. More search, compute, or context alone does not remove this bottleneck. We build and maintain a community platform for cumulative and comparable iteration, and release the data and code at https://mls-bench.com . 1 Introduction “We want AI agents that can discover like we can, not which contain what we have discovered.” — Richard S. Sutton, The Bitter Lesson Large language models (LLMs) have evolved from chatbots [ 9 , 70 , 7 , 96 , 98 , 95 , 20 ] into agents that pursue long-horizon objectives [ 83 , 116 , 120 , 64 , 71 , 113 , 30 , 97 , 2 , 22 ] , including software engineering [ 41 , 67 , 13 , 52 ] , deep research [ 86 , 84 , 93 , 108 ] , and mathematical theorem proving [ 37 , 53 , 14 , 54 , 103 ] . Attention has recently shifted to more frontier problems including open optimization tasks such as circle packing [ 69 , 118 , 104 , 61 , 87 ] and machine learning engineering, where agents compete on Kaggle-style tasks [ 36 , 11 , 78 , 73 , 35 , 115 ] . However, even these more advanced settings still do not resemble how human computer scientists discover new general methods. This mismatch comes from the target of evaluation. Most agent benchmarks reward engineering : improving one fixed instance through data processing, tuning, debugging, and model selection. ML science asks for a method-level idea, such as a new architecture, objective, component, or optimizer, that can be validated beyond the setting that produced it [ 89 , 101 , 48 , 32 , 100 , 33 , 66 , 85 , 79 , 119 , 24 , 47 , 42 , 56 ] . The question is whether agents can create such methods, not just improve one leaderboard. Existing benchmarks do not yet isolate this capability. ML-engineering benchmarks mix method choice with implementation and tuning [ 36 , 11 , 78 , 73 , 35 , 115 ] ; end-to-end research benchmarks make attribution hard [ 12 , 90 ] ; and recent narrow discovery benchmarks remain tied to single components or subfields [ 102 , 80 , 18 , 72 ] . Figure 1 : MLS-Bench overview. Left: comparison of Frontier-CS, MLE-Bench, and MLS-Bench tasks. Right: 20 representative MLS-Bench tasks from 140 tasks across 12 domains. We introduce MLS-Bench ( ML S cience), a benchmark containing 140 tasks across 12 ML domains for evaluating whether AI systems can produce genuine, transferable ML method improvements. As shown in Figure 1 , each task asks an agent to improve a targeted component under controlled edit scopes, reproduced strong human baselines, and multiple evaluation settings. This design makes the submitted artifact attributable to the intended method rather than to evaluator changes, training-protocol hacks, or scale increases. We also curate MLS-Bench-Lite , a 30 -task challenging subset covering all 12 areas for rapid iteration and broader model tracking. Our evaluation exposes a large method-discovery gap. Even with strong baselines in context and multiple opportunities to iterate, current frontier agents remain far from reliably matching human-designed methods inside the same scaffold. They are noticeably better at engineering-style tuning than at producing a new method that survives controlled validation, which makes MLS-Bench a demanding target for future foundation models and self-evolving frameworks. We further study the influence of stronger inference-time support, more context, or greater freedom to experiment. These analyses show that current agents’ limitation goes beyond proposing methods: they can search, tune, and recombine familiar ingredients, but struggle with the scientific judgment needed to form hypotheses, design experiments, allocate limited trials, and turn feedback into evidence for scalable claims. Human expert assessment also finds that genuinely new mechanisms are rare and often weakly justified. We maintain MLS-Bench as a community benchmark with a growing leaderboard to guide the development of future foundation models and agent harnesses toward bootstrapping AI development. 2 Related Work Automated scientific discovery. Computational methods have contributed to scientific discovery across diverse domains [ 99 , 49 , 4 , 37 , 81 , 43 , 107 , 92 , 21 , 63 ] , including computer science itself, spanning algorithms and systems [ 65 , 17 , 62 ] , and especially machine learning—including model architectures [ 121 , 55 ] , training procedures [ 1 , 26 , 38 ] , and data and loss design [ 23 , 28 ] . More recently, LLMs have accelerated automated discovery across a broad spectrum: serving as collaborative scientific partners [ 29 , 84 ] , optimizing specific algorithms and computational components [ 69 , 81 , 72 ] , and driving fully autonomous research [ 58 , 112 , 93 ] . A growing set of benchmarks evaluates these emerging capabilities [ 60 , 19 , 57 , 75 , 8 ] . Self-evolving agents and evaluation. The paradigm of LLMs has evolved from single-turn question answering [ 9 ] toward agents that iterate over extended horizons [ 116 , 83 ] . Self-evolving systems iteratively refine solutions through evolutionary search [ 69 , 81 , 16 , 88 ] , open-ended self-improving loops [ 50 , 5 , 74 , 46 ] , and test-time training [ 118 , 76 , 104 , 91 , 122 ] . However, these systems have been demonstrated primarily on specific optimization problems, such as circle packing, contest-style algorithm search, kernel optimization, and activation-function search [ 69 , 118 , 104 , 102 ] . Such settings are narrow in domain and do not capture whether a discovery is scalable and generalizable. Benchmarking LLM agents for Coding and