LLMs as In-Context Meta-Learners for Model and Hyperparameter Selection

Youssef Attia El Hili, Albert Thomas, Malik Tiomoko, Abdelhakim Benechehab, Corentin Léger, Corinne Ancourt, Balázs Kégl · 2025

Read LLMs as In-Context Meta-Learners for Model and Hyperparameter Selection by Youssef Attia El Hili, Albert Thomas, Malik Tiomoko, Abdelhakim Benechehab, Corentin Léger, Corinne Ancourt, Balázs Kégl (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-26510.txt.

Introduction

Youssef Attia El Hili, with six co-authors, asks whether a large language model can pick the right learning model for a new data set. Choosing a model family and its settings usually takes expert feel or a costly automatic search. The team turns each data set into plain metadata, a summary a language model can read, and asks it to recommend both.

El Hili and the team test two styles of asking. One uses only the model's built-in knowledge. The other adds examples of past tasks with the models that did well on them. Across made-up and real benchmarks, the language model picked competitive models and settings without any search. The example-fed style did better, which the authors read as evidence of in-context meta-learning, learning how to learn from the prompt itself.

Catalog record

CREATORYoussef Attia El Hili, Albert Thomas, Malik Tiomoko, Abdelhakim Benechehab, Corentin Léger, Corinne Ancourt, Balázs Kégl
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE72 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-26510

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LLMs as In-Context Meta-Learners for Model and Hyperparameter Selection Youssef Attia El Hili, Albert Thomas, Malik Tiomoko, Abdelhakim Benechehab, Corentin Léger, Corinne Ancourt, Balázs Kégl Abstract Model and hyperparameter selection are critical but challenging in machine learning, typically requiring expert intuition or expensive automated search. We investigate whether large language models (LLMs) can act as in-context meta-learners for this task. By converting each dataset into interpretable metadata, we prompt an LLM to recommend both model families and hyperparameters. We study two prompting strategies: (1) a zero-shot mode relying solely on pretrained knowledge, and (2) a meta-informed mode augmented with examples of models and their performance on past tasks. Across synthetic and real-world benchmarks, we show that LLMs can exploit dataset metadata to recommend competitive models and hyperparameters without search, and that improvements from meta-informed prompting demonstrate their capacity for in-context meta-learning. These results highlight a promising new role for LLMs as lightweight, general-purpose assistants for model selection and hyperparameter optimization. Code is available at this link . Keywords Large Language Models \cdot In-Context Learning \cdot Hyperparameter Optimization \cdot Meta-Learning 1 Introduction The performance of machine learning (ML) models hinges on the selection of appropriate algorithms and their hyperparameters. This joint optimization task is commonly referred to as the Combined Algorithm Selection and Hyperparameter optimization (CASH) problem ( Thornton et al. 2013 ; Bergstra and Bengio 2012 ; Snoek et al. 2012 ) . Traditionally, practitioners have relied on manual tuning, grid search, or Bayesian optimization techniques ( Mockus et al. 1978 ; Shahriari et al. 2016 ) to navigate this complex search space. However, these approaches are computationally expensive and demand substantial domain expertise. This creates barriers to entry and limits the scalability of ML applications across diverse domains. Large language models (LLMs) have recently shown strong capabilities in reasoning, knowledge synthesis, and problem-solving across domains ( Wei et al. 2022 ) . As they scale, they exhibit emergent behaviors that enable adaptation to new tasks by reusing prior experience in context ( Brown et al. 2020 ; Dong et al. 2024 ) . These behaviors have been interpreted as a form of in-context meta-learning, with transformers proposed as general-purpose meta-learners ( Kirsch et al. 2024 ) and LLMs studied explicitly in this role ( Coda-Forno et al. 2023 ) . Much of this prior work has focused on demonstrating the phenomenon itself, often in synthetic or language-oriented tasks. By contrast, model and hyperparameter selection provides a practical and consequential setting in machine learning where generalization across tasks directly impacts performance and efficiency. If LLMs can transfer knowledge in this context, they may offer a new paradigm for addressing the CASH problem and extend our understanding of their adaptability beyond controlled demonstrations. Figure 1: Overview of the method. Each task is represented by metadata, and the LLM outputs model and hyperparameter configurations. The dotted arrow indicates the inclusion of prior-task metadata-configuration pairs in the meta-informed setting. This research introduces two prompting strategies for leveraging LLMs in model and hyperparameter selection. The Zero-Shot strategy relies solely on high-level task metadata, requiring no prior examples. The Meta-Informed strategy augments this by incorporating pairs of task metadata and well-performing model configurations from previous tasks, enabling more informed recommendations (Figure 1 ). Unlike prior work ( Zheng et al. 2023 ; Zhang et al. 2024 ) , our approach operates without iterative validation feedback. It also enables cross-task generalization in the meta-informed case. Importantly, we prompt the LLM to propose complete configurations consisting of both model families and associated hyperparameters, which can then be directly evaluated or integrated into downstream pipelines. We evaluate both prompting strategies on tabular regression and classification tasks. Results show that LLMs, when properly prompted, can make surprisingly effective recommendations even in zero-shot settings where conventional methods often require extensive experimentation. The meta-informed strategy further improves performance by leveraging prior knowledge, often approaching or matching the quality of expert-guided selections. Taken together, these findings highlight the potential of LLMs as meta-learners in automated machine learning: they can reason about datasets, models, and hyperparameters with minimal tuning, offering a scalable and accessible alternative to traditional search-based or expert-driven workflows. This also complements concurrent applications of LLMs to other stages of the AutoML pipeline such as feature engineering with CAAFE ( Hollmann et al. 2023 ) . The remainder of this paper is structured as follows. Section 2 reviews related work in hyperparameter optimization, meta-learning, and LLM-based methods. Section 3 introduces our formal problem setup and frames CASH as a meta-learning task. Section 4 presents a controlled synthetic experiment that motivates our approach by showing how LLM prompting can capture useful hyperparameter patterns in a simple setting. Section 5 then describes our methodology and evaluates LLM-based prompting strategies on a diverse suite of benchmark datasets. Section 6 discusses broader implications, limitations, and future directions. Finally, Section 7 summarizes our contributions. 2 Related Work Hyperparameter Optimization. Early work on hyperparameter optimization (HPO) relied on simple search strategies such as grid search and random search ( Bergstra and Bengio 2012 ) . More sophisticated model-based methods, such as Bayesian optimization (BO), iteratively fit surrogate models to past evaluations and propose promising configurations ( Bergstra et al. 2011 ; Snoek et al. 2012 ) . Subsequent advances introduced multi-fidelity and bandit-based approaches, including Successive Halving ( Jamieson and Talwalkar 2016 ) and Hyperband ( Li et al. 2017 ) , which exploit early stopping to allocate resources efficiently. Later extensions sought to transfer knowledge across related tasks or account for computational budgets, for example through multi-task Bayesian optimization and compute-aware methods ( Swersky et al. 2013 ; Golovin et al. 2017 ) . These methods significantly improved efficiency but still treat each optimization task largely in isolation. Meta-Learning for HPO. To overcome this limitation, meta-learning approaches aim to accelerate HPO by leveraging prior experience across tasks. Transfer Neural Processes (TNP) ( Wei et al. 2021 ) , for example, incorporate meta-knowledge such as surrogate models and historical trial data to improve sample efficiency. Meta-Bayesian optimization methods extend this idea by learning priors over surrogate models from related tasks, enabling faster convergence on new optimization problems ( Feurer et al. 2015 ; Perrone et al. 2018 ) . Other approaches, such as ALFA ( Baik et al. 2020 ) , adapt hyperparameters dynamically during training using a meta-learner, while SHSR ( Borboudakis et al. 2023 ) prunes unpromising regions of the search space using past AutoML runs. PriorBand ( Mallik et al. 2023 ) further accelerates HPO by combining expert beliefs with low-fidelity proxy tasks to guide search in deep learning pipelines. These methods illustrate the value of meta-knowledge, but they still assume a fixed model class. The CASH Problem. In practice, algorithms and hyperparameters must be optimized jointly, formalized as the CASH problem ( Thornton et al. 2013 ) . A common approach is to treat model choice as a categorical hyperparameter, as in Auto-WEKA ( Thornton et al. 2013 ) and Auto-sklearn ( Feurer et al. 2015 ) , but the resulting search space is large and expensive to explore. Bandit-based formulations address this by casting algorithm selection as arms with HPO inside each arm, e.g., MaxUCB ( Balef et al. 2025 ) ,

Read the complete text