Data-Efficient Symbolic Regression via Foundation Model Distillation
Wangyang Ying, Jinghan Zhang, Haoyue Bai, Nanxu Gong, Xinyuan Wang, Kunpeng Liu, Chandan K. Reddy, Yanjie Fu · 2025
Read Data-Efficient Symbolic Regression via Foundation Model Distillation by Wangyang Ying, Jinghan Zhang, Haoyue Bai, Nanxu Gong, Xinyuan Wang, Kunpeng Liu, Chandan K. Reddy, Yanjie Fu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-19487.txt.
Introduction
Wangyang Ying and co-authors present in 2025 a way to teach equation-finding models from very little data. Symbolic regression hunts for plain formulas behind measurements, prized for science because the result can be read and checked. Big foundation models for the task stumble on small, specialized data sets, sometimes doing worse than scratch.
Their framework, EQUATE, fine-tunes such a model by distillation, aligning symbol and number views and steering the search in a continuous embedding space by fitness and simplicity. Across three public benchmarks it beat leading baselines in accuracy and robustness while keeping formulas short and inference fast.
Catalog record
| CREATOR | Wangyang Ying, Jinghan Zhang, Haoyue Bai, Nanxu Gong, Xinyuan Wang, Kunpeng Liu, Chandan K. Reddy, Yanjie Fu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-19487
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Data-Efficient Symbolic Regression via Foundation Model Distillation Wangyang Ying, Jinghan Zhang, Haoyue Bai, Nanxu Gong, Xinyuan Wang, Kunpeng Liu, Chandan K. Reddy, Yanjie Fu Abstract . Discovering interpretable mathematical equations from observed data (a.k.a. equation discovery or symbolic regression) is a cornerstone of scientific discovery, enabling transparent modeling of physical, biological, and economic systems. While foundation models pre-trained on large-scale equation datasets offer a promising starting point, they often suffer from negative transfer and poor generalization when applied to small, domain-specific datasets. In this paper, we introduce EQUATE (Equation Generation via QUality-Aligned Transfer Embeddings), a data-efficient fine-tuning framework that adapts foundation models for symbolic equation discovery in low-data regimes via distillation. EQUATE combines symbolic-numeric alignment with evaluator-guided embedding optimization, enabling a principled embedding-search-generation paradigm. Our approach reformulates discrete equation search as a continuous optimization task in a shared embedding space, guided by data-equation fitness and simplicity. Experiments across three standard public benchmarks (Feynman, Strogatz, and black-box datasets) demonstrate that EQUATE consistently outperforms state-of-the-art baselines in both accuracy and robustness, while preserving low complexity and fast inference. These results highlight EQUATE as a practical and generalizable solution for data-efficient symbolic regression in foundation model distillation settings. 1. Introduction Many scientific and engineering applications require uncovering the underlying mathematical equations that describe observed data. This symbolic regression task aims to discover interpretable model equations that capture relationships within data without relying on predefined functional forms. Such equations support transparent modeling, inverse engineering, and low-cost discovery of underlying physical, biological, or economic laws ( Wang et al. 2019 ; Angelis et al. 2023 ; Cranmer et al. 2020 ; Can and Heavey 2011 ; Claveria et al. 2017 ; La Cava et al. 2023 ) . Prior literature on model equation discovery is two-fold: 1) Genetic Programming (GP) based methods ( Burlacu et al. 2020 ; Gustafson et al. 2005 ; Haider et al. 2023 ; Augusto and Barbosa 2000 ; Zhong et al. 2018 ) ; 2) deep learning based methods ( Petersen et al. 2019 ; Kim et al. 2020 ; Zhang et al. 2023 ; d’Ascoli et al. 2022 ) . Unfortunately, GP methods suffer from high search complexity and thus are inefficient and slow; deep learning methods cannot apply to single small-scale task-specific datasets (i.e., perform poorly when large-scale data are not available). Emerging foundation model techniques can learn generalizable knowledge and robust representations from massive datasets and become prevalent in model equation discovery. For example, a Transformer-based foundation equation discovery model was proposed in ( Biggio et al. 2021 ) via large-scale pre-training on generic datasets, followed by other foundation models ( Kamienny et al. 2022 ; Shojaee et al. 2023 ; Holt et al. 2023 ; Valipour et al. 2021 ) . The underlying idea of these foundation models is to encode a dataset into an embedding vector, then decode the embedding vector to the corresponding equation. This motivates our study of data-efficient symbolic regression via foundation model distillation: How can we effectively distill general symbolic knowledge from foundation models into compact, task-specific equations using only small datasets? Figure 1. FKT transfers generic knowledge to specific domains through fine-tuning and knowledge distillation. We formalize this as the FKT problem: developing a dedicated F inetuning framework for foundation K nowledge T ransfer in small-data symbolic regression tasks. As illustrated in Figure 1 , this framework should preserve useful foundation model priors, tailor them to task-specific data, and avoid the cost of training from scratch (as shown in Figure 1 ). There are two main challenges in solving FKT: 1) representation alignment of symbolic equation, numeric data, and data-equation fitness. Equation spaces (symbolic, discrete), data spaces (numeric, continuous), and data-equation fitness (pairwise, scored) are structurally incompatible. How can we learn an aligned embedding space that enables seamless transfer between these modalities? 2) optimization steering for structured and meaningful task-specific search of equations. The generation objectives of existing equation discovery foundation models are typically borrowed from textual generation pre-training, such as token-level cross-entropy loss. However, classic textual generation objectives tend to generate equations with high token-level similarity. This optimization direction fundamentally deviates from the desired directions of equation discovery: high equation discovery fitness and equation simplicity. The optimization challenge aims to answer: How can we develop an optimization feedback mechanism to allow for a more structured and meaningful search of best-fitting equations under small tasking data? Our insight: integrating numeric-symbolic alignment and evaluator-guided optimization for transfer fine-tuning on small tasking data. Our AI task is to develop a foundation knowledge transfer fine-tuning framework for model equation discovery from small tasking data. We found that an embedding-search-generation framework opens three new perspectives to augment equation discovery fine-tuning with small domain tasking data: 1) The embedding step empowers us to redesign the encoder’s neural structure (e.g., attentive fusion with pre-trained transformer encoder) to compress not just numeric data patterns but also symbolic equation structures to a symbol-numeric aligned embedding space of data-equation pairs, while still exploiting foundation model knowledge. Such an alignment strategy exhibits better representation expressiveness and truthfulness than the methods that only compress numerical data or directly fine-tune existing foundation equation discovery models. 2) The embedding step creates a continuous space to represent data-equation fitness, thus reformulating discrete equation search into differentiable optimization. The gradient-based equation search step empowers us to redesign a specialized domain loss (i.e., trade-off between data-equation fitness and equation simplicity) for precision fine-tuning with small-tasking data. Interestingly, by adding and learning an evaluator to predict data-equation fitness and equation simplicity during fine-tuning, we find that the learned evaluator can provide task-specific gradient guidance (feedback) to steer and tailor equation search to domain data, and avoid misalignment when transferring foundation general knowledge to specific tasks. 3) In our task setting, we are only given a single small task-specific dataset to be distilled into an equation. We demonstrate that the data-equation fitness gap between the foundation model-generated equation and the best-fitting equation can serve as fine-tuning supervision signals. Summary of technical solution. We propose Equation Generation via Quality-Aligned Transfer Embeddings ( EQUATE ), which includes four steps: 1) Training Set Preparation: We leverage a pre-trained foundation model to generate imperfect yet relevant candidate equations on the task-specific dataset. These equations are applied to sampled data subsets to create training instances labeled by their data-equation fitness scores. 2) Embedding Space Construction: We design a neural architecture comprising a partially frozen transformer encoder for numerical data, an LSTM encoder for symbolic equations, an attention-based fusion module, an evaluator, and a partially frozen decoder. This setup learns a shared symbolic-numeric embedding space that captures both structural and semantic fitness. 3) Fitness-Guided Search: We perform gradient-based optimization in the embedding space, using evaluator feedback to guide the search toward embeddings corresponding to high-fidelity equations. 4) Equation Generation: The optimized embedding is decoded via the autoregressive decoder to produce the final equation. The decoder is fine-tuned to align with domain-specific data while retaining prior knowledge from the foundation model. Our contributions. This paper introduces EQUATE, a fine-tuning framework that adapts foundation models for symbolic regression in small, domain-specific settings. Our key contributions are: 1) Fine-Tuning Framework for Symbolic Regression: We propose a novel fine-tuning framework that adapts pretrained foundation models for symbolic equation