LLM-Meta-SR: In-Context Learning for Evolving Selection Operators in Symbolic Regression

Hengzhe Zhang, Qi Chen, Bing Xue, Wolfgang Banzhaf, Mengjie Zhang · 2025

Read LLM-Meta-SR: In-Context Learning for Evolving Selection Operators in Symbolic Regression by Hengzhe Zhang, Qi Chen, Bing Xue, Wolfgang Banzhaf, Mengjie Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18602.txt.

Introduction

Hengzhe Zhang and co-authors use in 2025 large language models to design the selection step inside evolutionary search for symbolic regression, where algorithms hunt for formulas that fit data. Past attempts to evolve algorithms with language models lacked meaning aware guidance and produced bloated code. Their frame fixes both faults, with a paired selection method that respects meaning and a control on code growth.

Domain knowledge is folded into the prompt so the model makes apt operators. On standard test sets, the model made operators that beat nine expert designed rivals. The evolved operator also improved a leading symbolic regression system. Results suggest language models can serve as meta designers of search parts.

Catalog record

CREATORHengzhe Zhang, Qi Chen, Bing Xue, Wolfgang Banzhaf, Mengjie Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE129 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-18602

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LLM-Meta-SR: In-Context Learning for Evolving Selection Operators in Symbolic Regression Hengzhe Zhang, Qi Chen, Bing Xue, Wolfgang Banzhaf, Mengjie Zhang Abstract Large language models (LLMs) have revolutionized algorithm development, yet their application in symbolic regression, where algorithms automatically discover symbolic expressions from data, remains limited. In this paper, we propose a meta-learning framework that enables LLMs to automatically design selection operators for evolutionary symbolic regression algorithms. We first identify two key limitations in existing LLM-based algorithm evolution techniques: lack of semantic guidance and code bloat. The absence of semantic awareness can lead to ineffective exchange of useful code components, while bloat results in unnecessarily complex components; both can hinder evolutionary learning progress or reduce the interpretability of the designed algorithm. To address these issues, we enhance the LLM-based evolution framework for meta-symbolic regression with two key innovations: a complementary, semantics-aware selection operator and bloat control. Additionally, we embed domain knowledge into the prompt, enabling the LLM to generate more effective and contextually relevant selection operators. Our experimental results on symbolic regression benchmarks show that LLMs can devise selection operators that outperform nine expert-designed baselines, achieving state-of-the-art performance. Moreover, the evolved operator can further improve a state-of-the-art symbolic regression algorithm, achieving the best performance among 28 symbolic regression and other machine learning algorithms across 116 regression datasets. This demonstrates that LLMs can exceed expert-level algorithm design for symbolic regression. I Introduction Symbolic regression (SR) is the task of discovering mathematical expressions that accurately model a given dataset. It has the potential to produce interpretable models when appropriate complexity control mechanisms are employed, and its ability to uncover underlying relationships makes it widely applicable across various domains. Formally, given a dataset \mathcal{D}=\{\mathbf{x}^{(i)},y^{(i)}\}_{i=1}^{N} , where \mathbf{x}^{(i)}\in\mathbb{R}^{d} are input features and y^{(i)}\in\mathbb{R}^{m} are target values, the goal is to find a symbolic expression f^{*}\in\mathcal{F} from a space of candidate expressions \mathcal{F} that minimizes a loss function \mathcal{L} : f^{*}=\arg\min_{f\in\mathcal{F}}\mathcal{L}(f(\mathbf{x}),y). (1) Many SR algorithms have been proposed, including evolutionary [ 11 ] , neural [ 23 ] , and Monte Carlo tree search-based [ 60 ] approaches. Among them, genetic programming (GP) remains one of the most widely used SR paradigms due to its flexible representation and global, gradient-free search mechanism. In recent years, there has been growing interest in hybrid methods that combine the flexibility of GP with the learning capabilities of neural networks [ 18 , 63 ] . In both evolutionary [ 11 ] and neural-evolutionary [ 18 , 63 ] SR algorithms, the system follows an iterative learning process in which solutions are progressively refined by selecting promising candidates. In this process, the selection operator, which identifies candidate solutions worth refining, plays a key role in system effectiveness. However, current selection operators are typically manually designed by experts, requiring significant effort and trial-and-error, which can hinder progress. Ideally, an automated algorithm design framework capable of generating effective selection operators for SR would substantially enhance research and development efficiency in this area. Other evolutionary components—such as mutation and crossover operators—are tightly coupled to the underlying representation, whether tree-based, graph-based, or grammar-based, which limits transferability across SR systems. In contrast, the selection operator is largely representation-agnostic. Thus, this paper focuses on automated selection design without entangling the framework with representation-specific design choices. Large language models (LLMs) have recently emerged as powerful tools for automated heuristic design in various optimization tasks [ 42 , 30 , 58 ] . Recent studies have shown that LLMs can effectively replace traditional crossover and mutation operators to generate candidate solutions in SR [ 42 , 17 , 63 ] . However, selection operators are still manually designed by experts, creating a gap in automated algorithm design. This motivates the exploration of automated selection operator design using LLMs, which is the focus of this work. We adopt a meta-learning framework based on LLM-driven algorithm evolution to automatically design a core algorithmic component—specifically, the selection operator—in evolutionary SR, aiming to achieve strong performance across a wide range of regression tasks. In this meta-learning framework, a population of algorithms is evolved, where each individual in the population represents a complete selection operator, implemented as a piece of code that defines a selection strategy, rather than a solution to a specific SR problem. The population thus maintains a collection of candidate algorithms, and the evolutionary process aims to discover improved selection operators through crossover and mutation operations guided by LLMs. While LLM-guided algorithm search frameworks such as FunSearch [ 42 ] , EoH [ 30 ] , and ReEvo [ 58 ] have laid the groundwork for algorithm evolution, two challenges remain underexplored in LLM-driven algorithm design. The first challenge is the underutilization of semantic information when generating algorithms. Here, we define semantic information as the performance of an algorithm on each task instance. Typically, only the average performance across all training instances is provided [ 20 ] , which overlooks fine-grained behavioral differences. As illustrated in Fig 2 , consider a scenario with multiple algorithms in the meta-learning population: some perform well on the first dataset but poorly on the second, while others show the opposite pattern. Despite their contrasting behaviors, these algorithms may exhibit similar average performance. In LLM-driven algorithm evolution, we present multiple algorithms to the LLM so it can infer useful patterns and generate improved variants. However, algorithms that behave similarly at the instance level offer little benefit for this purpose as they succeed and fail on the same datasets and therefore provide redundant information. In contrast, presenting complementary algorithms enables the LLM to integrate distinct strengths, increasing the likelihood of generating an algorithm that performs well on both datasets. Therefore, incorporating semantic information is crucial for effective algorithm evolution. Fig. 1 : Illustration of fine-grained semantic differences between algorithms during meta-evolution. P1 and P2 are parent selection operator algorithms, O is an offspring algorithm, and D is a dataset. Fig. 2 : Average code length of solutions in the population over generations without bloat control techniques, and the score of the best solution. The second challenge is complexity bias , illustrated in Fig 2 . LLMs tend to generate overly long or intricate code during optimization, similar to code bloat in GP [ 2 ] . This results in the accumulation of redundant or non-functional logic, which impairs interpretability, wastes a large number of tokens, and slows optimization. As shown in Fig 2 , such complexity can lead to performance stagnation, ultimately limiting the effectiveness of evolution. In this paper, we propose an LLM-driven method for meta SR that automatically designs selection operators for evolutionary SR algorithms 1 1 1 Source Code: https://anonymous.4open.science/r/LLM-Meta-SR/ . The main contributions are as follows: • We propose an LLM-driven meta SR framework to automatically design selection operators using in-context learning. The LLM learns from design and evaluation history and automatically discovers generalizable operators that consistently outperform human-designed counterparts across a wide range of unseen datasets. • We identify the issue of bloat in LLM-based code generation and introduce a bloat control strategy, which improves both the interpretability of the evolved code and the effectiveness of the evolutionary process. • We propose semantic feedback and complementary selection mechanisms to fully leverage semantic information during LLM-driven generation, explicitly guiding algorithm evolution, integrating

Read the complete text