Symbolic Regression with Multimodal Large Language Models and Kolmogorov–Arnold Networks

2025

Read Symbolic Regression with Multimodal Large Language Models and Kolmogorov–Arnold Networks (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-07956.txt.

Introduction

Thomas Harvey, Fabian Ruehle, Kit Fraser-Taliente, and James Halverson present in 2025 a new way to find formulas that fit data, using a vision-capable language model. Symbolic regression hunts for a short equation behind a set of numbers. Most methods must be handed a menu of allowed building blocks first.

Here the model simply looks at a plot of a one-variable function and proposes a formula shape. Standard fitting tools then tune its free numbers, and a genetic process breeds the best guesses. Clever prompting can steer the guesses without any fixed menu. For many-variable problems, the team first trains a special network whose edges each hold a one-variable curve. Reading those curves gives the full formula, which a language model then tidies. Their slogan from the work is that one variable is all you need.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-07956

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Symbolic Regression with Multimodal Large Language Models and Kolmogorov–Arnold Networks Abstract We present a novel approach to symbolic regression using vision-capable large language models (LLMs) and the ideas behind Google DeepMind’s Funsearch . The LLM is given a plot of a univariate function and tasked with proposing an ansatz for that function. The free parameters of the ansatz are fitted using standard numerical optimisers, and a collection of such ansätze make up the population of a genetic algorithm. Unlike other symbolic regression techniques, our method does not require the specification of a set of functions to be used in regression, but with appropriate prompt engineering, we can arbitrarily condition the generative step. By using Kolmogorov–Arnold Networks (KANs), we demonstrate that “univariate is all you need” for symbolic regression, and extend this method to multivariate functions by learning the univariate function on each edge of a trained KAN. The combined expression is then simplified by further processing with a language model. Thomas R. Harvey a, 1 1 1 [email protected] , Fabian Ruehle b,c,a, 2 2 2 [email protected] , Kit Fraser-Taliente d, 3 3 3 [email protected] , James Halverson b,a, 4 4 4 [email protected] {}^{\text{a}} NSF AI Institute for Fundamental Interactions, MIT, Cambridge, MA 02139, USA {}^{\text{b}} Department of Physics, Northeastern University, Boston, MA 02115, USA {}^{\text{c}} Department of Mathematics, Northeastern University, Boston, MA 02115, USA {}^{\text{d}} Rudolf Peierls Centre for Theoretical Physics, University of Oxford, Oxford OX1 2JD, UK 1 Introduction Symbolic regression is a long-standing and inherently challenging problem in the fields of machine learning and applied mathematics. The task involves searching for mathematical expressions that best describe a given dataset, yet the space of possible functions is unyielding. To navigate this expansive search space effectively, modern symbolic regression algorithms often rely on explicit or implicit complexity measures to mitigate expression “bloat” and encourage simpler expressions [ 1 , 2 , 3 ] . Humans exhibit an impressive intuitive ability to infer plausible functional forms from visual representations of data, typically guided by a natural simplicity bias. When presented with a graph of a function, expert human observers tend to prefer and propose simpler ansätze that often outperform those generated by automated symbolic regression methods. This observation naturally raises the question: can large multimodal language models (LLMs) emulate this human-like intuition? In this work, we show that they can—at least in the context of univariate functions—especially when integrated with DeepMind’s FunSearch [ 4 ] . To that end, we introduce LLM-LEx (Large Language Models Learning Expressions), a package available at our GitHub repository [ 5 ] , which leverages commercially available LLMs for guiding symbolic regression. By invoking the Kolmogorov–Arnold representation theorem, which tells us that multivariate functions can be represented as sums and compositions of univariate functions, we moreover argue that focussing on univariate cases is sufficient for a broad class of problems. We train Kolmogorov–Arnold Networks (KANs) on the data for which we aim to find a symbolic expression. Then, we employ LLM-LEx to propose symbolic expressions for the edge functions within these networks. The combined expression is then simplified using an additional step of generative modelling using LLMs. We refer to this combined method as KAN-LEx, and it is likewise publicly available at our repository. We emphasise that we do not expect beyond state-of-the-art performance from this method of symbolic regression. For one, many traditional and freely available packages have been highly optimised through years of development. Our aim is simply to demonstrate that this new approach to symbolic regression is viable and surprisingly successful given the simplicity of its implementation. Our initial implementation comprised approximately 100 lines of code. The combination of our method with KANs allows application to multivariate functions, although in principle any symbolic regression method can be used for this step. Unless stated otherwise, we use gpt-4o via OpenRouter throuhout. 2 Symbolic Regression with LLMs In this section, we begin with a simple example of using a vision-capable language model for symbolic regression with a one-shot prompt. We then extend this approach by incorporating a genetic algorithm. We compare to traditional techniques and subsequently benchmark against a set of randomly generated functions. We also consider the use of open-source language models as an alternative to proprietary ones. We then consider the effects of noisy data, before concluding with a discussion on prompt engineering and learning special functions. 2.1 A simple example Figure 1 : An example function to learn Consider the graph in Figure 1 , generated by interpolating 500 evenly spaced points between -1.2 and 1.2 . It is not too difficult for a human to make a reasonable ansatz for the underlying function. In particular, the oscillatory behaviour and intersection at zero suggest the function may take the form y(x)=g(x)\sin(ax) , where g(x) is a smooth function and a is a constant. The way the function flattens out toward zero then suggests the presence of exponential decay. Putting these together, a suitable, and correct, ansatz for such a function would be y(x)=ce^{-bx^{2}}\sin(ax), (1) which after fitting to the data yields a=2 , b=10 and c=1 . Despite the apparent ease with which a human can make an educated guess for the function, developing software that replicates this kind of intuition is highly challenging. This, in essence, captures the core difficulty of symbolic regression. For example, when we input the 100 data points into a traditional symbolic regression algorithm—specifically, Mathematica’s FindFormula 1 1 1 As it is proprietary, very little public information is available about how FindFormula operates. However, it is believed to involve a heuristic search over symbolic expressions, guided by both the complexity and accuracy of the functions. Such methodology is typical of other approaches to symbolic regression. function [ 6 ] —we find \displaystyle f(x)= \displaystyle-13.7072x^{19.}+104.411x^{17.}-344.585x^{15.}+647.044x^{13.}-763.814x^{11.} (2) \displaystyle+591.433x^{9.}-303.867x^{7.}+101.756x^{5.}-20.6594x^{3.}+1.98844x. While ( 2 ) gives a good fit to the data, for the given range, the resulting expression is difficult to interpret and offers little additional insight into the underlying structure of the problem: we might just as well have fitted the data with a high-degree polynomial from the outset. We turn instead to a multimodal language model (in this example, gpt-4o ): we will see language models display some of the intuitive symbolic regression ability exhibited by humans, most likely acquired from their extensive pretraining data. We present the LLM with the image in Figure 1 , accompanied by the following prompt: “An initial ansatz for this function is curve = lambda x, params: params[0]*x + params[1]. Give an improved ansatz for the image. params can be any length” The response from the LLM will, sometimes 2 2 2 As the response from the LLM is probabilistic, it does not always return exactly the correct function. We address this in the next subsection by incorporating a genetic algorithm, where this actually becomes a strength. The probabilistic nature of the LLM’s response can be thought of as mutation; when given access to LLM sampling parameters, we gain some parametric control over this mutation. , contain the Python lambda function: ⬇ curve = lambda x , params : np . sin ( params [0]* x )* np . exp (- params [1]* x **2) Remarkably, this is exactly the functional form we

Read the complete text