AutoSOTA: An End-to-End Automated Research System for State-of-the-Art AI Model Discovery

Yu Li Chenyang Shao Xinyang Liu Ruotong Zhao Peijie Liu, Hongyuan Su Zhibin Chen Qinglong Yang Anjie Xu Yi Fang Qingbin Zeng, Tianxing Li Jingbo Xu Fengli Xu Yong Li Tie-Yan Liu · 2026

Read AutoSOTA: An End-to-End Automated Research System for State-of-the-Art AI Model Discovery by Yu Li Chenyang Shao Xinyang Liu Ruotong Zhao Peijie Liu, Hongyuan Su Zhibin Chen Qinglong Yang Anjie Xu Yi Fang Qingbin Zeng, Tianxing Li Jingbo Xu Fengli Xu Yong Li Tie-Yan Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-05550.txt.

Introduction

Li and fifteen colleagues automate the full research loop with AutoSOTA, an eight-agent system that reproduces and improves state-of-the-art models from top papers. Agents ground papers to code, repair environments, track long experiments, and vet gains against spurious wins. Across papers from eight top conferences it discovers 105 new state-of-the-art models, averaging about five hours per paper.

Catalog record

CREATORYu Li Chenyang Shao Xinyang Liu Ruotong Zhao Peijie Liu, Hongyuan Su Zhibin Chen Qinglong Yang Anjie Xu Yi Fang Qingbin Zeng, Tianxing Li Jingbo Xu Fengli Xu Yong Li Tie-Yan Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE111043
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.05550

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AutoSOTA: An End-to-End Automated Research System for State-of-the-Art AI Model Discovery Yu Li Chenyang Shao Xinyang Liu Ruotong Zhao Peijie Liu, Hongyuan Su Zhibin Chen Qinglong Yang Anjie Xu Yi Fang Qingbin Zeng, Tianxing Li Jingbo Xu Fengli Xu Yong Li Tie-Yan Liu Abstract Artificial intelligence research increasingly depends on prolonged cycles of reproduction, debugging, and iterative refinement to achieve State-Of-The-Art (SOTA) performance, creating a growing need for systems that can accelerate the full pipeline of empirical model optimization. In this work, we introduce AutoSOTA, an end-to-end automated research system that advances the latest SOTA models published in top-tier AI papers to reproducible and empirically improved new SOTA models. We formulate this problem through three tightly coupled stages: resource preparation and goal setting; experiment evaluation; and reflection and ideation. To tackle this problem, AutoSOTA adopts a multi-agent architecture with eight specialized agents that collaboratively ground papers to code and dependencies, initialize and repair execution environments, track long-horizon experiments, generate and schedule optimization ideas, and supervise validity to avoid spurious gains. We evaluate AutoSOTA on the recent research papers collected from eight top-tier AI conferences under filters for methodological relevance, code availability, repository readiness, and execution cost. Across these papers, AutoSOTA achieves strong end-to-end performance in both automated replication and subsequent optimization. Specifically, it successfully discovers 105 new SOTA models that surpass the original reported methods, averaging approximately five hours per paper. Case studies spanning LLMs, NLP, computer vision, time series, and optimization further show that the system can move beyond routine hyperparameter tuning to identify architectural innovation, algorithmic redesigns, and workflow-level improvements. These results suggest that end-to-end research automation can serve not only as a performance optimizer, but also as a new form of research infrastructure that reduces repetitive experimental burden and helps redirect human attention toward higher-level scientific creativity. 1 Introduction Artificial intelligence research is increasingly organized around the pursuit of State-Of-The-Art (SOTA) performance. In many areas, the value of a new method is judged not only by conceptual novelty, but by whether it can surpass a highly competitive empirical frontier. Yet, achieving such SOTA breakthroughs is almost never the product of a standalone theoretical insight. More often, it requires a prolonged cycle of implementation, replication, debugging, evaluation, and reflection, in which months of human effort are spent translating promising insights into reproducible performance gains. This tension has become especially pronounced in modern AI, where research artifacts are distributed across papers, repositories, datasets, checkpoints, and undocumented engineering assumptions. As a result, a substantial portion of scientific attention is absorbed by repetitive experimental iteration, which might otherwise be directed toward more original, disruptive exploration. To alleviate this burden, our previous work introduced OmniScientist [ 17 ] , a comprehensive research agent that pushes the paradigm toward fully autonomous, end-to-end scientific discovery. Building upon this overarching framework, an important subsequent challenge is whether we can instantiate this capability into targeted automated systems that reliably accelerate the highly competitive pipeline of SOTA model optimization. Recent progress suggests that this goal is increasingly plausible, but existing efforts remain incomplete. Traditional automated machine learning (AutoML) [ 2 , 4 , 5 , 7 ] systems have demonstrated the value of hyperparameter tuning and architecture search, while newer LLM-based systems have shown impressive capabilities in code generation, debugging, and restricted forms of algorithm discovery. At the same time, emerging autonomous AI scientist systems have begun to explore broader research workflows. However, most existing approaches still operate on relatively narrow problem formulations. For instance, even the latest iterative frameworks like AutoResearch [ 6 ] strictly confine the agent to improving a single bounded code module within a sanitized sandbox, entirely sidestepping the complexities of environment setup. They typically do not solve the full end-to-end problem of starting from an unstructured research paper, recovering the corresponding executable environment, reproducing the reported baseline, and then performing valid, open-ended methodological reflection under realistic constraints. This gap motivates the need for an end-to-end automated AI research system that can bridge unstructured literature, experiment environments setup, and high-level reflection in a single closed-loop framework. In this work, we formulate this challenge as an end-to-end discovery problem that maps a top-tier AI paper to an improved executable repository whose empirical performance surpasses the original method. To make this open-ended task tractable, we decompose it into three tightly coupled sub-problems. The first is resource preparation and goal setting, which requires grounding a paper into a concrete experimental task by locating repositories, datasets, base models, and the exact target metric to be exceeded. The second is experiment evaluation, which requires transforming incomplete and noisy academic code into a runnable baseline that faithfully approximates the reported results despite missing scripts, broken dependencies, and long-horizon debugging challenges. The third is reflection and ideation, which elevates the system from a mere replication engine to an autonomous scientific discoverer. At this stage, the system is tasked with uncovering genuine algorithmic improvements—exploring structural innovations, novel learning objectives, and optimized implementation strategies—while intelligently balancing open-ended exploration against the exorbitant computational cost of real-world trials. Together, these three sub-problems define the core scientific and engineering bottlenecks that any end-to-end automated research system must solve. To address these challenges, we introduce AutoSOTA, a multi-agent framework designed to mimic the division of labor in human AI research. AutoSOTA decomposes the workflow across eight specialized agents organized into the three stages above. In the preparation stage, AgentResource grounds research papers to repositories and external dependencies, while AgentObjective constructs a tree-structured evaluation rubric that translates the paper’s macro goal into dense, verifiable feedback. In the replication stage, AgentInit builds the executable environment, AgentMonitor tracks long-horizon execution and prevents deadlock, and AgentFix performs timely repair of runtime failures. In the reflection stage, AgentIdeator constructs a constrained hypothesis library, AgentScheduler manages iterative optimization across code states and compute resources, and AgentSupervisor enforces a red-line system that prevents invalid metric gains caused by evaluation leakage, protocol violations, or other shortcuts. Through this modular design, AutoSOTA converts fragmented academic artifacts into an execution-ready optimization loop and supports stable, scalable, and scientifically comparable SOTA-oriented research automation. We evaluate AutoSOTA on the research papers drawn from eight top-tier AI conferences, after filtering for methodological relevance, public code availability, repository readiness, and tractable execution cost. Across these curated research papers, AutoSOTA demonstrates strong end-to-end performance in both automated replication and subsequent optimization, successfully discovering new SOTA models for 105 papers. Remarkably, the system yields an average performance improvement of nearly 10% over the original main methods, requiring only about five hours of execution time per paper. We further presents case studies spanning research subfields of LLM, NLP, computer vision, time series, and optimization, illustrating that the framework is capable not only of hyperparameter tuning but also of more substantive architectural, algorithmic, and workflow-level interventions. These results suggest that end-to-end research automation is no longer limited to narrow coding tasks, but can support broad empirical advancement across heterogeneous AI research areas. More broadly, the significance of AutoSOTA lies beyond raw benchmark improvement. As described in the project vision, the system is intended not merely as an optimizer, but as a new research infrastructure for AI science itself: one that can absorb the tedious, high-frequency cycle of experimental iteration and thereby amplify human creativity.

Read the complete text