Towards Scaling Laws for Symbolic Regression

David Otte, Jörg K.H. Franke, Frank Hutter · 2025

Read Towards Scaling Laws for Symbolic Regression by David Otte, Jörg K.H. Franke, Frank Hutter (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-26064.txt.

Introduction

David Otte, Jorg K. H. Franke, Arber Zela, Fabio Ferreira, and Frank Hutter run the first systematic study of scaling laws for symbolic regression. Symbolic regression searches for a short math formula that explains data, which can yield insight and models that people can read. Deep learning versions now rival genetic programming methods, but how performance grows with scale was largely unknown.

Otte and the team train a transformer pipeline at five model sizes across three orders of magnitude of compute, on carefully generated data. Both validation loss and the share of problems solved follow clear power law trends with compute. They also map how best batch size and learning rate grow with model size, giving guidance for training larger systems.

Catalog record

CREATORDavid Otte, Jörg K.H. Franke, Frank Hutter
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE20 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-26064

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards Scaling Laws for Symbolic Regression David Otte, Jörg K.H. Franke, Frank Hutter Abstract Symbolic regression (SR) aims to discover the underlying mathematical expressions that explain observed data. This holds promise for both gaining scientific insight and for producing inherently interpretable and generalizable models for tabular data. In this work we focus on the basics of SR. Deep learning-based SR has recently become competitive with genetic programming approaches, but the role of scale has remained largely unexplored. Inspired by scaling laws in language modeling, we present the first systematic investigation of scaling in SR, using a scalable end-to-end transformer pipeline and carefully generated training data. Across five different model sizes and spanning three orders of magnitude in compute, we find that both validation loss and solved rate follow clear power-law trends with compute. We further identify compute-optimal hyperparameter scaling: optimal batch size and learning rate grow with model size, and a token-to-parameter ratio of ≈ \approx 15 is optimal in our regime, with a slight upward trend as compute increases. These results demonstrate that SR performance is largely predictable from compute and offer important insights for training the next generation of SR models. \workshoptitle MATH-AI 1 Introduction Symbolic regression seeks to uncover the underlying mathematical expressions that describe the relationship between a set of observed variables. In recent years, pre-trained transformer models for symbolic regression have become increasingly popular and have started to achieve performance comparable to classic genetic programming methods [ 1 , 2 , 3 , 4 ] . We observe that prior work has focused on tweaking the training process while holding the scale mostly constant. To date, we are not aware of any symbolic regression models trained with more than \approx 100 million parameters. Motivated by the impact of scaling laws in language modeling [ 5 , 6 ] , we ask: Do similar scaling laws exist for symbolic regression, and can they have a comparable impact on future model design? To answer this question, we see the need for a more controlled and scalable training setup. Specifically, we propose (1) a more systematic synthetic data generation approach that allows for stricter control over the expressions seen during training, and (2) targeted architectural improvements motivated by recent tabular foundation models. To keep the scope manageable yet insightful, our expressions contain only integer constants and at most two variables, while still covering a broad operator mix commonly seen in symbolic regression. Based on this setup, we conduct the first systematic scaling study in symbolic regression, spanning five model sizes and three orders of magnitude in compute. Our goal is not to beat existing baselines, but to establish that symbolic regression with transformers follows predictable scaling laws, which can serve as a design principle for future models. Our key contributions are: (1) We demonstrate power-law scaling of solved rate and loss with compute over three orders of magnitude. (2) We identify systematic trends in optimal learning rate, batch size, and token-to-parameter ratio. (3) We introduce a scalable end-to-end pipeline that allows a clean and efficient scaling analysis. 2 Related Work Biggio et al. 2021 were the first to suggest pre-training transformers on millions of synthetic symbolic regression tasks. They use a set-encoder and a standard decoder to autoregressively predict function skeletons, whose constants are then refined via BFGS. Subsequent work has explored improvements on this theme, for example sharpening skeleton separation with a contrastive term [ 7 ] or framing the problem as multimodal translation [ 3 ] . Other approaches, such as E2E [ 2 ] and SymFormer [ 8 ] propose end-to-end architectures to directly output full expressions including constants, simplifying the pipeline and achieving comparable performance with genetic programming approaches. Our study builds on this end - to - end paradigm, coming closest to E2E, but shifts the focus from loss engineering to the role of scale. 3 Methodology In this section, we describe our approach for data generation and model training. We adopt many ideas from Kamienny et al. 2022 , but apply targeted improvements that allow for more efficient training and a better scaling analysis. Figure 1 : Overview of our two-step data generation. In the first step, we recursively generate a set of base expressions by applying a set of binary operators (BOP) and a set of unary operators (UOP). In the second step, we sample expression-dataset pairs from our base expressions. Data generation Fig. 1 shows our two-step approach for data generation. While previous work usually adopts the expression sampling mechanism proposed by Lample and Charton 2019 , we generate a complete set of expressions recursively, starting with just the variables and then iteratively applying unary and binary operators. This allows us to get more control over the expressions the model sees during training, especially in terms of the number of duplicates, and avoids biasing the model towards certain, more common expressions. Furthermore, we filter out equivalent expressions and unify their representation, allowing us to feed the model with cleaner training data. We obtain our set of base expressions E by taking all expressions up to a fixed threshold. For each of these expressions, we try to sample k expression-dataset pairs. Therefore, we first insert random constants and then sample a random dataset from a Gaussian mixture following Kamienny et al. 2022 . More details about our data generation can be found in App. A . Tokenization Similar to existing approaches, we decide to use base 10 floating-point notation to encode the tabular dataset. Thus, we split each numeric value into a mantissa, including the sign, and an exponent. Under the assumption that different representations of expressions are equally difficult to learn for the model, we represent our target expressions as LaTeX strings for better readability. In the target expression, we tokenize all constants digit-wise. Figure 2 : Overview of our model architecture. Architecture Given the great scaling properties of transformers [ 10 , 5 ] , we mainly adopt the standard encoder-decoder architecture used by previous work [ 1 , 2 ] and only apply targeted improvements. Fig. 2 shows an overview of our architecture. While previous work decided to merge each input point into a single embedding, we choose a different embedding strategy and aim to get a single embedding for each cell in our dataset. Therefore, we up-project our mantissas and exponents to the embedding dimension and simply sum them together. This embedding strategy allows us to take inspiration from recent tabular foundation models and adopt their encoder architecture, which has proven to be very powerful for handling tabular data [ 11 , 12 ] . The main innovation here is that in each layer, we now apply both row- and column-wise attention across different variables and different data points. Our decoder follows the standard sequence-to-sequence transformer architecture [ 13 ] and only cross-attends to the updated embeddings of the target cells. We train with cross-entropy loss between the predicted tokens and the tokens of the true expression. Experiments To analyze scaling, we follow the strategies of scaling law papers in language modeling [ 5 , 6 , 14 ] . To approximate training compute, we adapt the FLOP estimate of Kaplan et al. 2020 to encoder-decoder models:

Read the complete text