Sparse Interpretable Deep Learning with LIES Networks for Symbolic Regression

Mansooreh Montazerin, Majd Al Aawar, Antonio Ortega, Ajitesh Srivastava · 2025

Read Sparse Interpretable Deep Learning with LIES Networks for Symbolic Regression by Mansooreh Montazerin, Majd Al Aawar, Antonio Ortega, Ajitesh Srivastava (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-08267.txt.

Introduction

Mansooreh Montazerin and co-authors recover plain math formulas from data using a small, readable neural network. Their 2025 paper deals with symbolic regression, which seeks formulas rather than black box predictions. Past methods search huge populations or generate formulas token by token, which scales poorly. Their network has a fixed design with four interpretable building blocks: log, identity, exponential, and sine units.

Training uses oversampling and a custom loss that rewards sparsity and steadies gradients. After training, pruning trims the network down to a compact formula. On standard benchmarks, the method beat all baselines in both accuracy and sparseness. Tests removing each design piece showed each one matters. The authors argue readable formulas give insight that fitted curves cannot. Such formulas can be checked, reused, and built upon by hand.

Catalog record

CREATORMansooreh Montazerin, Majd Al Aawar, Antonio Ortega, Ajitesh Srivastava
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-08267

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Sparse Interpretable Deep Learning with LIES Networks for Symbolic Regression Mansooreh Montazerin, Majd Al Aawar, Antonio Ortega, Ajitesh Srivastava Abstract Symbolic regression (SR) aims to discover closed-form mathematical expressions that accurately describe data, offering interpretability and analytical insight beyond standard black-box models. Existing SR methods often rely on population-based search or autoregressive modeling, which struggle with scalability and symbolic consistency. We introduce LIES (Logarithm, Identity, Exponential, Sine), a fixed neural network architecture with interpretable primitive activations that are optimized to model symbolic expressions. We develop a framework to extract compact formulae from LIES networks by training with an appropriate oversampling strategy and a tailored loss function to promote sparsity and to prevent gradient instability. After training, it applies additional pruning strategies to further simplify the learned expressions into compact formulae. Our experiments on SR benchmarks show that the LIES framework consistently produces sparse and accurate symbolic formulae outperforming all baselines. We also demonstrate the importance of each design component through ablation studies. I Introduction Uncovering the underlying mathematical laws that govern complex systems is a fundamental goal in science and engineering. Symbolic Regression (SR) serves as a powerful approach to this task by discovering abstract mathematical expressions that best describe relationships within observed data and provide insights into the mechanism of the underlying processes [ 1 ] . Unlike conventional regression techniques that fit parameters within predefined models, SR searches for both the structure and parameters of equations, offering a flexible and interpretable way to model intricate phenomena [ 2 ] . This ability makes SR highly valuable across disciplines, enabling the derivation of governing equations in fields such as fluid mechanics, molecular interactions, astrophysics, and materials science [ 3 , 4 , 5 , 6 ] . A key advantage of SR is its interpretability. Resulting models are expressed in symbolic form, making them understandable and generalizable beyond the training data [ 7 ] . These models can be derived from experimental measurements, simulations, or real-world observations, allowing researchers to gain insights into the underlying mechanisms of physical systems [ 8 ] . However, discovering meaningful symbolic representations is inherently challenging due to the vast combinatorial search space of potential equations. Finding a balance between accuracy, simplicity, and generalization is crucial, as overly complex models risk overfitting while overly simplistic ones may fail to capture essential system dynamics [ 9 ] . SR spans a wide range of methods from traditional approaches like Genetic Programming (GP) to advanced Deep Learning (DL) methods like Transformers, Graph Neural Networks (GNNs), and deep generative models [ 10 , 11 , 12 , 13 , 14 , 7 , 15 ] . Evolutionary algorithms like GP [ 12 ] explore the space of mathematical expressions by iteratively refining candidate solutions. These equations are constructed using fundamental components, known as primitives, which can include constants and elementary functions like addition, multiplication, and trigonometric operations. However, these methods use a predefined set of operations and construct candidate expressions through a sequential generation and evaluation process, which often leads to overly complex and less interpretable equations. Alternatively, recent DL methods [ 13 , 7 , 15 ] aim to generate candidate formulae by treating expressions as sequences, structured graphs, or samples from learned latent spaces, enabling better search for candidate expressions compared to traditional GP-based methods. These approaches offer better scalability and adaptability but face several limitations: they often require large amounts of training data, struggle with generating syntactically valid or semantically meaningful expressions, and offer limited interpretability during training, as symbolic forms are only revealed after decoding. To address these challenges, we propose a new framework that uses a neural network architecture while preserving the interpretability of traditional GP. Specifically, it uses stacking layers of a small set of operators as activations in such a way that any candidate formula can be represented by sparsification and pruning the network. Therefore, it reduces the problem of learning the best candidate to the problem of learning compact and sparse neural networks. Our model, LIES, is a feedforward architecture with a fixed set of activation functions — bounded logarithm (L) and exponential (E), sine (S), and identity (I) — carefully chosen to capture operations commonly found in natural and scientific laws, such as multiplication, division, exponentiation, and periodic behavior. Unlike GP-based methods that generate expressions through iterative search, LIES trains a fixed network whose structure and activation functions are explicitly aligned with symbolic operations. This design enables the model to recover interpretable expressions via structured pruning, while retaining the data efficiency and expressiveness of neural networks. In contrast to black-box DL models, LIES incorporates symbolic structure directly into the architecture, requiring less training data and producing more compact, meaningful formulae. The overall framework progressively sparsifies the trained network to distill a final symbolic expression that is both concise and interpretable. Specifically, our main contributions are as follows: • We propose a novel architecture that incorporates a specific set of activation functions (L, I, E, and S) reflecting operations common in natural laws to achieve interpretable symbolic structures. The model is trained with a modified loss function that helps avoid unstable gradients due to logarithm and exponential functions. • We introduce a pipeline that promotes sparsity and interpretability through a sparsity-aware loss function, combined with model pruning and symbolic simplification. • We evaluate our method on 61 formulae from the AI Feynman dataset [ 8 ] , showing strong symbolic and numerical performance, and provide an ablation study to isolate the contribution of each component in the pipeline. II Related Works Symbolic regression (SR) is a foundational technique for discovering underlying mathematical relationships from data. It aims to automatically generate symbolic expressions or closed-form equations that best explain the observed input–output behavior. Genetic programming (GP) plays a central role in SR, evolving populations of candidate expressions through iterative processes such as mutation, crossover, and selection. Pioneering work by John Koza [ 16 ] laid the foundation for GP-based SR, evolving expressions through mimicry of biological selection. Subsequent works aimed to improve the efficiency of GP and its applicability to a wider range of data types. For example, Gustafson et al. [ 17 ] improved GP by preventing crossover between individuals with identical fitness, which often produced redundant offspring. This simple constraint led to better performance, especially on more complex SR tasks. PySR [ 12 ] represents a more recent and performant GP-based SR system. It introduces a multi-population evolutionary algorithm with an evolve–simplify–optimize loop, enabling the discovery of concise and interpretable expressions, especially for scientific datasets. Nonetheless, GP-based methods in general remain computationally expensive, sensitive to hyperparameter settings, and prone to generating overly complex or redundant expressions due to their heuristic nature. In contrast, deep learning (DL)-based SR approaches leverage the representation learning capabilities of neural networks to model symbolic mappings more flexibly. Petersen et al. [ 18 ] introduced a deep reinforcement learning approach to SR, where a Recurrent Neural Network generates expressions sequentially. Biggio et al. [ 19 ] proposed a seq2seq model mapping input-output pairs to a symbolic expression built from a vocabulary. SR can also be deemed as surrogate modeling, offering interpretable and tractable approximations for complex, expensive, or black-box functions. Gaussian process surrogates [

Read the complete text