Can LLMs Beat Classical Hyperparameter Optimization Algorithms? A Study on autoresearch

Fabio Ferreira, Lucca Wobbe, Arjun Krishnakumar, Frank Hutter, & Arber Zela, [0.5em] ELLIS Institute Tübingen · 2026

Read Can LLMs Beat Classical Hyperparameter Optimization Algorithms? A Study on autoresearch by Fabio Ferreira, Lucca Wobbe, Arjun Krishnakumar, Frank Hutter, & Arber Zela, [0.5em] ELLIS Institute Tübingen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-24647.txt.

Introduction

Ferreira and co-authors ask whether language models can beat classical hyperparameter optimisation algorithms. They use the autoresearch testbed, where an agent tunes a small language model's training under a fixed compute budget. Classical methods such as CMA-ES and TPE consistently beat pure language model agents. Letting the model edit source code directly narrows the gap but does not close it. Their hybrid, Centaur, shares the optimiser's internal state with the model and wins overall; even a 0.8B model suffices in the hybrid role.

Catalog record

CREATORFabio Ferreira, Lucca Wobbe, Arjun Krishnakumar, Frank Hutter, & Arber Zela, [0.5em] ELLIS Institute Tübingen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32290
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.24647

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Can LLMs Beat Classical Hyperparameter Optimization Algorithms? A Study on autoresearch Fabio Ferreira, Lucca Wobbe, Arjun Krishnakumar, Frank Hutter, & Arber Zela, [0.5em] ELLIS Institute Tübingen Abstract The autoresearch repository enables an LLM agent to search for optimal hyperparameter configurations on an unconstrained search space by editing the training code directly. Given a fixed compute budget and constraints, we use autoresearch as a testbed to compare classical hyperparameter optimization (HPO) algorithms against LLM-based methods on tuning the hyperparameters of a small language model. Within a fixed hyperparameter search space, classical HPO methods such as CMA-ES and TPE consistently outperform LLM-based agents. However, an LLM agent that directly edits training source code in an unconstrained search space narrows the gap to classical methods substantially despite using only a self-hosted open-weight 27B model. Methods that avoid out-of-memory failures outperform those with higher search diversity, suggesting that reliability matters more than exploration breadth. While small and mid-sized LLMs struggle to track optimization state across trials, classical methods lack domain knowledge. To bridge this gap, we introduce Centaur, a hybrid that shares CMA-ES’s internal state, including mean vector, step-size, and covariance matrix, with an LLM 1 1 1 Named after the mythological half-human, half-horse hybrid: our method merges LLM reasoning with classical optimization. . Centaur achieves the best result in our experiments, with its 0.8B variant outperforming the 27B variant, suggesting that a cheap LLM suffices when paired with a strong classical optimizer. The 0.8B model is insufficient for unconstrained code editing but sufficient for hybrid optimization, while scaling to 27B provides no advantage for fixed search space methods. Preliminary experiments with the frontier model Gemini 3.1 Pro Preview do not close the gap to classical methods. Code is available at https://github.com/ferreirafabio/autoresearch-automl . 1 Introduction autoresearch ( 14 ) demonstrated that an LLM agent can iteratively edit training code to improve a small language model ( {\sim} 50M parameters), reaching a val_bpb of {\sim} 0.978 in less than 100 trials on a single H100. Building on that, 45 showed that TPE, a classical AutoML method, with expert-designed hyperparameters can outperform Karpathy’s agent within a similar budget. Hyperparameter optimization (HPO) is a central component of AutoML ( 9 ; 5 ) . These results raise two questions: (i) how do other classical HPO methods perform on this task? and (ii) can LLM-based HPO methods outperform classical ones? To answer these questions, we benchmark 9 HPO methods, spanning 4 classical, 4 LLM-based, and 1 hybrid, on Karpathy’s autoresearch task. All methods operate under the same 24-hour GPU training budget with 3 seeds. To reduce human priors, we automatically extract 14 hyperparameters from the training script; while the ranges require some domain knowledge, the HP selection itself is automated, removing manual search space curation. Figure 1 compares all 9 methods using 27B LLM variants against cumulative training wall-time, showing that classical methods find better configurations than LLM-based agents within the fixed search space. The exception is Karpathy Agent (Code), which directly edits training source code and is competitive with classical methods, even though the latter find a similar performing hyperparameter configuration {\sim}4{\times} faster. All LLM methods use a self-hosted open-weight model (Qwen3.5-27B); preliminary experiments with the frontier model Gemini 3.1 Pro Preview ( Section A.2 ) show competitive but not superior performance. Beyond this comparison, we propose Centaur, a hybrid method that combines CMA-ES with an LLM by sharing the optimizer’s full internal state, including the mean vector \bm{\mu} , step-size \sigma , and covariance matrix \mathbf{C} . The hypothesis is that CMA-ES and LLMs have complementary strengths: CMA-ES learns the optimization landscape but lacks domain knowledge, while the LLM brings transformer training intuitions but, at the small and mid-sized scales we test (0.8B and 27B), struggles to track optimization state across trials reliably, e.g., LLM methods show OOM rates comparable to random search despite observing full trial history. This motivates pairing the LLM with a classical optimizer whose state can be shared explicitly. We chose CMA-ES because its internal state is particularly interpretable for LLM communication (see Section 4 ). In summary, we make the following contributions: • We benchmark 9 HPO methods on autoresearch ( 14 ) , supporting both fixed-HP and agentic code-editing optimization, under identical 24-hour budgets with 3 seeds. • We show that classical HPO outperforms LLM agents within a fixed search space. An LLM agent that directly edits code is more competitive but still falls short of the best classical methods. • We introduce Centaur , a hybrid that shares CMA-ES’s full internal state with the LLM and achieves the best result in our experiments. • We analyze search diversity, OOM rates, and model scaling across all methods. Preliminary experiments with the frontier model Gemini 3.1 Pro Preview suggest that simply scaling the LLM optimizer does not close the gap to classical methods. Figure 1: Best Validation Bits-Per-Byte (mean \pm std across 3 seeds) of HPO algorithms against cumulative training time. All methods receive the same 24-hour GPU training budget; LLM inference overhead is excluded. All LLM-based methods use Qwen3.5-27B as the LLM optimizer. Classical methods such as CMA-ES and TPE converge faster and to better final values than LLM-based methods. Centaur, our CMA-ES and LLM hybrid, achieves the best result in our experiments. 2 Related Work Classical HPO spans a wide range of approaches ( 5 ; 9 ) , from random search ( 4 ) and Bayesian optimization with Gaussian process surrogates ( 38 ) to sequential model-based approaches with random forests such as SMAC ( 12 ) , tree-structured Parzen estimators such as TPE ( 3 ) , and evolution strategies such as CMA-ES ( 11 ) . We focus on single-task HPO methods without multi-fidelity or transfer to isolate each optimizer’s ability to learn the landscape from scratch. Methods such as Hyperband ( 19 ) , BOHB ( 8 ) , transfer HPO ( 41 ; 2 ) , and zero-shot approaches ( 23 ) could in principle be applied to this benchmark but are not in the scope of this study. In line with the increasing interest in open-ended agentic discovery ( 43 ; 22 ; 18 ; 20 ; 40 ) , recent work explores LLMs as components in HPO pipelines. While in principle, methods such as AlphaEvolve ( 22 ) or ShinkaEvolve ( 18 ) can be tasked to optimize the objective in autoresearch, we focus only on methods tailored specifically for HPO. LLAMBO ( 42 ) uses an LLM as a surrogate model inside Bayesian optimization, replacing the Gaussian process with LLM-based performance predictions. SLLMBO ( 21 ) integrates an LLM with TPE into a joint sampler. 44 prompt LLMs directly for HP suggestions. LLaMA-ES ( 16 ) uses LLMs to tune CMA-ES’s own hyperparameters. 36 partition the search space into subregions via a bandit mechanism and use an LLM to propose candidates within each region. The authors further extend that to multi-objective optimization ( 37 ) . However, none of these methods share the classical optimizer’s full internal state with the LLM, which is the key idea behind Centaur. Our work differs from prior work in four ways. First, we use autoresearch ( 14 ) as a real-world benchmark that naturally accommodates

Read the complete text