Auto Researching, not hyperparameter tuning: Convergence Analysis of 10,000 LLM-Guided ML Experiments

Xiaoyi Li · 2026

Read Auto Researching, not hyperparameter tuning: Convergence Analysis of 10,000 LLM-Guided ML Experiments by Xiaoyi Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-15916.txt.

Introduction

Li measures what LLM agents really search when left to design experiments. Two agents ran 10,469 experiments on dashcam collision detection. The design space held 108,000 cells across 27 days of runs. ANOVA splits show architecture choices drive 94 percent of score spread. Hyperparameter tweaks inside one architecture drive just 6 percent. A second dataset repeats the pattern at 75 percent. The winning build pairs V-JEPA 2 features with Zipformer encoders. No human proposed that pairing first, the author notes.

Catalog record

CREATORXiaoyi Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47119
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.15916

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Auto Researching, not hyperparameter tuning: Convergence Analysis of 10,000 LLM-Guided ML Experiments Xiaoyi Li Abstract When LLM agents autonomously design ML experiments, do they perform genuine architecture search—or do they default to hyperparameter tuning within a narrow region of the design space? We answer this question by analyzing 10,469 experiments executed by two LLM agents (Claude Opus and Gemini 2.5 Pro) across a combinatorial configuration space of 108,000 discrete cells for dashcam collision detection over 27 days. Through ANOVA decomposition, we find that architectural choices explain 94% of performance variance ( F = 1324 F=1324 , η 2 = 0.94 \eta^{2}=0.94 ), while hyperparameter variation within a fixed architecture explains only 6%. Cross-task validation on a second collision dataset confirms this finding (75% architecture-explained variance) with a different winning backbone, confirming genuine architecture discovery. The agents’ key contribution is discovering that V-JEPA 2 video features with Zipformer temporal encoders achieve 0.9245 AP—a configuration no human proposed—and concentrating search on productive architectural regions: at N = 50 N=50 , LLM-guided search reaches AP = 0.985 =0.985 versus 0.965 0.965 for from-scratch random search. Post-bugfix convergence follows a power law ( c = 0.11 c=0.11 , R 2 = 0.93 R^{2}=0.93 ); the low exponent reflects the cost of broad exploration, not inefficiency, since the LLM discovers qualitatively better regions than random or Bayesian baselines. We characterize multi-agent search dynamics via entropy cycles and Jensen–Shannon specialization, providing the first large-scale empirical framework for LLM-guided combinatorial ML experiment design. 1 Introduction When LLM agents are given the ability to propose, execute, and learn from machine learning experiments, do they perform genuine architecture search—or do they merely adjust hyperparameters within a fixed architecture? This question is central to understanding the value of LLM-based research agents: if they merely optimize numeric knobs within a fixed architecture, they offer little beyond classical HPO tools. Classical approaches—hyperparameter optimization (HPO) and neural architecture search (NAS)—handle this by defining a fixed, structured search space and applying Bayesian optimization ( Snoek et al. 2012 ; Bergstra et al. 2011 ) or gradient-based methods ( Liu et al. 2019 ) . But real ML research operates in a far richer space: researchers simultaneously choose backbones, temporal encoders, loss formulations, augmentation strategies, and training schedules, guided by intuition, literature knowledge, and iterative reasoning about failure modes. Can LLM agents, with their capacity for natural language reasoning over experimental results, serve as effective policies for this combinatorial search? Prior work on automated ML operates within narrow, predefined search spaces. HPO tools ( Bergstra et al. 2011 ; Akiba et al. 2019 ; Falkner et al. 2018 ) optimize numeric hyperparameters given a fixed architecture. NAS methods ( Liu et al. 2019 ; Pham et al. 2018 ) search cell topologies but do not jointly optimize loss functions, data augmentation, or training schedules. Recent LLM-based research agents ( Lu et al. 2024 ; Huang et al. 2024 ; Jiang et al. 2025 ) generate and execute code but lack formal analysis of their search dynamics. No existing framework formalizes the full autonomous research loop—where an agent observes a history of experiment outcomes, reasons about what to try next across categorical and continuous dimensions simultaneously, and iteratively refines its search strategy—as a mathematically characterizable optimization process. Without formal analysis, we cannot answer basic questions: How does LLM-guided search compare to random or Bayesian search in sample efficiency? What is the convergence rate? When do diminishing returns set in? We address these questions by formalizing autonomous ML research as optimization over a structured combinatorial configuration space \mathcal{C} . Each experiment is a sample from this space, and the LLM agent is a contextual policy \pi(c_{t}\mid H_{t-1}) that maps the history of observations to the next configuration to evaluate. This formulation enables us to borrow tools from bandit theory ( Slivkins 2019 ) , information-theoretic analysis, and empirical process theory to rigorously characterize search dynamics. We instantiate this framework with a concrete system and a large-scale case study: 10,469 experiments for dashcam collision detection on the Nexar dataset ( Nexar 2024 ) , executed autonomously over 27 days on 16 H100 GPUs. Our contributions are: 1. Empirical framework for LLM-guided experiment search (Section 3 ). We formalize autonomous ML research as optimization over a structured combinatorial configuration space \mathcal{C}=\mathcal{C}_{\mathrm{arch}}\times\mathcal{C}_{\mathrm{loss}}\times\mathcal{C}_{\mathrm{train}}\times\mathcal{C}_{\mathrm{data}} with |\mathcal{C}_{\mathrm{discrete}}|=108{,}000 cells. We define the LLM agent as a contextual search policy \pi(c_{t}\mid H_{t-1}) and introduce information-theoretic metrics (entropy, JSD, innovation rate) that enable quantitative comparison of search dynamics across policies. 2. Convergence analysis with baseline comparisons (Section 5 ). Restricting to post-bugfix experiments to isolate search dynamics from infrastructure confounds, we demonstrate that LLM-guided search follows a power-law convergence model \mathrm{AP}^{*}(N)\approx a-bN^{-c} with exponent c=0.11 ( R^{2}=0.93 ), and compare convergence dynamics against random search and TPE baselines. 3. Multi-agent dynamics characterization (Section 5 ). We quantify exploration-exploitation tradeoffs using configuration-space entropy H(t) , agent specialization via Jensen–Shannon divergence D_{\mathrm{JSD}}(\pi_{\mathrm{Claude}},\pi_{\mathrm{Gemini}}) , and innovation rate decay, revealing emergent specialization dynamics and exploration-exploitation cycling. 4. Open dataset of 10,469 experiments . We release the full experiment dataset (configurations, metrics, and agent logs) as a benchmark for studying autonomous research dynamics. 2 System Overview We implement the search loop using Orze, 1 1 1 Orze is open-sourced at https://github.com/warlockee/orze ; see also https://orze.ai an open-source orchestration system for autonomous ML research. This section briefly describes the system as an instantiation of the formal framework in Section 3 ; all implementation details are deferred to Appendix B . Figure 1 illustrates the overall architecture. Figure 1: System overview. Two LLM agents observe the shared leaderboard and propose configurations c_{t}\in\mathcal{C} . The orchestrator deduplicates proposals, schedules execution on a GPU cluster, and updates the history H_{t} . Self-healing handles runtime failures via LLM-assisted diagnosis. Search loop. At step t , the system executes: 1. The agent observes history H_{t-1}=\{(c_{1},y_{1}),\ldots,(c_{t-1},y_{t-1})\} , encoded as a leaderboard and recent configurations. 2. The agent selects c_{t}\sim\pi(c\mid H_{t-1}) by generating a structured YAML configuration. 3. The system evaluates y_{t}=f(c_{t})+\epsilon by training the specified model on GPU and measuring validation AP. 4. The history is updated: H_{t}=H_{t-1}\cup\{(c_{t},y_{t})\} . Multi-agent coordination. Two heterogeneous LLM agents—Claude Opus (819 research cycles) and Gemini 2.5 Pro (864 research cycles)—operate in parallel on a shared filesystem. Each agent generates 3–5 base configurations per cycle; an automated hyperparameter sweep module then expands promising bases into variants (e.g., learning rate grids, loss parameter sweeps), yielding \sim 25 candidate configurations per cycle on average. A deduplication mechanism filters near-identical proposals, and a priority queue schedules experiments across 16 H100 GPUs on two nodes. Over 27 days of continuous operation, the system generated 42,042 total candidate configurations (including auto-sweep expansions), of which 10,469 were executed to completion (25.0% execution rate; the remainder were deduplicated or deprioritized). Of the completed experiments, 1,337 have fully parseable configuration records suitable for the ablation and convergence analyses in Section 5 ; the remainder use legacy configuration formats from early campaign iterations. Research agent as policy. Each LLM research agent receives a context string encoding H_{t-1} —including the current leaderboard (top- k configurations with AP scores), the distribution of recently tried configurations, and failure logs from crashed experiments—and produces a structured configuration c_{t}\in\mathcal{C} . This is a stochastic policy \pi:\mathcal{H}\to\Delta(\mathcal{C}) where

Read the complete text