Neuro-Symbolic Activation Discovery: Transferring Mathematical Structures from Physics to Ecology for Parameter-Efficient Neural Networks

Anas Hajbi · 2026

Read Neuro-Symbolic Activation Discovery: Transferring Mathematical Structures from Physics to Ecology for Parameter-Efficient Neural Networks by Anas Hajbi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-10740.txt.

Introduction

Anas Hajbi proposes Neuro-Symbolic Activation Discovery, a way to give neural networks activation functions drawn from science itself. Standard networks use generic activations like ReLU, GELU, or SiLU. Those shapes ignore the math structure inside scientific data. Hajbi's framework mines that structure with genetic programming, which evolves formulas that fit the data.

The found formulas are readable, and they are injected into the network as custom activations. The title frames a transfer from physics to ecology: structure learned in one field may serve another. If it works, networks could need fewer parameters for the same skill. The approach blends symbolic math with deep learning in a concrete, testable way.

Catalog record

CREATORAnas Hajbi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-10740

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Neuro-Symbolic Activation Discovery: Transferring Mathematical Structures from Physics to Ecology for Parameter-Efficient Neural Networks Anas Hajbi Abstract Modern neural networks rely on generic activation functions (ReLU, GELU, SiLU) that ignore the mathematical structure inherent in scientific data. We propose Neuro-Symbolic Activation Discovery , a framework that uses Genetic Programming to extract interpretable mathematical formulas from data and inject them as custom activation functions. Our key contribution is the discovery of a Geometric Transfer phenomenon: activation functions learned from particle physics data successfully generalize to ecological classification, outperforming standard activations (ReLU, GELU, SiLU) in both accuracy and parameter efficiency. On the Forest Cover dataset, our Hybrid Transfer model achieves 82.4% accuracy with only 5,825 parameters, compared to 83.4% accuracy requiring 31,801 parameters for a conventional heavy network—a 5.5 × \times parameter reduction with only 1% accuracy loss. We introduce a Parameter Efficiency Score ( E param = AUC / log 10 ⁡ ( Params ) E_{\text{param}}=\text{AUC}/\log_{10}(\text{Params}) ) and demonstrate that lightweight hybrid architectures consistently achieve 18–21% higher efficiency than over-parameterized baselines. Crucially, we establish boundary conditions: while Physics → \to Ecology transfer succeeds (both involve continuous Euclidean measurements), Physics → \to Text transfer fails (discrete word frequencies require different mathematical structures). Our work opens pathways toward domain-specific activation libraries for efficient scientific machine learning. Keywords: Neuro-Symbolic AI, Activation Functions, Genetic Programming, Transfer Learning, Scientific Machine Learning, Parameter Efficiency, Green AI 1 Introduction Deep learning has achieved remarkable success across scientific domains, from particle physics [ 1 ] to ecology [ 2 ] . However, a fundamental inefficiency persists: neural networks use the same activation functions regardless of domain. ReLU, GELU, and SiLU are general-purpose nonlinearities designed for broad applicability, not for encoding the specific mathematical structures present in scientific data. This universality comes at a cost. To model complex continuous dynamics using piecewise-linear (ReLU) or smooth-but-generic (GELU) functions, networks require substantial over-parameterization. A physics simulation network must learn to approximate trigonometric relationships from scratch, despite such relationships being fundamental to the domain. We propose a radically different approach: let the data reveal its own mathematics . Using Genetic Programming (GP), we evolve symbolic expressions on small data subsets (10%) and compile them into differentiable activation functions. This creates Hybrid architectures that combine the learnability of gradient descent with the inductive bias of domain-specific symbolic mathematics. 1.1 Key Contributions 1. Neuro-Symbolic Activation Discovery: A pipeline that uses GP to discover interpretable activation functions from scientific data, compiled into PyTorch modules for end-to-end training. 2. Geometric Transfer Phenomenon: We demonstrate that activation functions discovered on particle physics data transfer successfully to ecological classification, providing evidence for shared mathematical structure across continuous scientific domains. 3. Parameter Efficiency Analysis: We introduce an efficiency metric and show that Hybrid architectures achieve 18–21% higher efficiency than conventional heavy networks, with 5–6 \times fewer parameters. 4. Boundary Conditions: We establish when transfer succeeds (continuous \to continuous) versus fails (continuous \to discrete), providing practical guidelines for deployment. 1.2 Motivating Observation Consider two seemingly unrelated classification tasks: • HIGGS: Classifying particle collision events based on kinematic variables (momenta, angles, energies). • Forest Cover: Classifying forest types based on cartographic variables (elevation, slope, aspect, soil type). Both involve continuous measurements in Euclidean space. Particle kinematics follow conservation laws with trigonometric structure; terrain metrics follow smooth geographic gradients. We hypothesize that a mathematical formula capturing “wave-like” or “periodic” patterns in physics data might also capture “gradient-like” patterns in ecological data. Our experiments confirm this hypothesis: a formula discovered on HIGGS transfers to Forest Cover, achieving higher accuracy than GELU or SiLU with identical parameter counts. 2 Related Work 2.1 Activation Function Design The Rectified Linear Unit (ReLU) [ 4 ] revolutionized deep learning through its simplicity and gradient-friendly properties. Subsequent work explored smooth alternatives: GELU [ 5 ] incorporates stochastic regularization and is standard in Transformers; SiLU/Swish [ 6 ] was discovered via reinforcement learning and excels in vision models; Mish [ 7 ] provides smooth, non-monotonic behavior. Learnable activations include PReLU [ 8 ] (learned negative slope), APL [ 9 ] (piecewise linear), and PAU [ 10 ] (rational functions via gradient descent). However, these approaches: (1) search within predefined function families, and (2) produce non-interpretable parameterizations. Our method differs fundamentally: we discover closed-form symbolic expressions that are human-readable and transferable across domains. 2.2 Symbolic Regression and Equation Discovery Genetic Programming has been applied to scientific discovery since Schmidt & Lipson’s seminal work [ 11 ] . Modern tools like PySR [ 12 ] distill trained neural networks into symbolic expressions. AI Feynman [ 13 ] combines neural networks with symbolic regression to rediscover physics equations. These approaches apply symbolic regression post-hoc for interpretability. We invert this paradigm: GP runs before training to discover activation functions that encode domain-appropriate inductive biases. 2.3 Physics-Informed Neural Networks PINNs [ 14 ] embed physical knowledge via loss function penalties (e.g., PDE residuals). Our approach is complementary: we embed domain structure through the architecture itself , specifically the activation function, requiring no modification to standard training. 3 Methodology Our framework consists of two phases: Discovery (finding the formula) and Injection (using it as an activation). 3.1 Phase 1: Symbolic Discovery via Genetic Programming Given a dataset \mathcal{D}=\{(\mathbf{x}_{i},y_{i})\}_{i=1}^{N} with \mathbf{x}_{i}\in\mathbb{R}^{d} and y_{i}\in\{0,1\} , we extract a discovery subset \mathcal{D}_{\text{disc}}\subset\mathcal{D} containing 10% of samples (stratified by class). 3.1.1 GP Configuration We employ gplearn [ 15 ] with: • Population: 500 individuals • Generations: 5 • Parsimony Coefficient: 0.01 (penalizes complexity) • Function Set: \mathcal{F}=\{\texttt{add},\texttt{sub},\texttt{mul},\sin,\cos,|\cdot|\} Safety Constraint: We include multiplication ( mul ) but rely on the parsimony coefficient to discourage unbounded growth. Protected operations (e.g., \sqrt{|x|+\epsilon} ) prevent numerical instabilities. Convergence Note: While typical GP applications use 50–100 generations, we found that 5 generations with a population of 500 individuals was sufficient for convergence in our setting. This rapid convergence is attributable to: (1) the restricted function set (6 operators), which limits the search space; (2) the parsimony coefficient, which penalizes overly complex expressions; and (3) the binary classification objective, which provides a clear fitness signal. Preliminary experiments with 10 and 20 generations yielded similar formulas, confirming that the search had converged. 3.1.2 Formula Generalization GP discovers expressions with specific feature indices (e.g., mul(cos(X25), X12) ). For use as an activation function, we generalize by replacing all feature references with the layer input x : f_{\text{discovered}}(X_{25},X_{12})\;\rightarrow\;\sigma(x)=\texttt{mul}(\cos(x),x) (1) This generalized form is applied element-wise to hidden layer outputs. Abstraction Note: The original GP expression describes interactions between specific physical features (e.g., X_{12} might represent transverse momentum, X_{3} might represent pseudorapidity). By replacing all feature indices with the scalar x , we abstract this domain-specific interaction into a general-purpose self-interaction term f(x) . This transformation sacrifices the precise physical semantics but retains the functional form —the oscillatory, multiplicative structure—which we hypothesize encodes useful inductive bias for continuous data. 3.2 Phase 2: Hybrid Architecture We compile discovered formulas into PyTorch modules using dynamic code generation: Algorithm 1 AutoSymbolicLayer Compilation 0: Formula string f , operator dictionary \mathcal{O} 1: Replace feature indices: X\d+ \to x 2: Replace operators: sin( \to ops["sin"]( 3: Compile: exec(f"def forward(x, ops): return {f}") 4: return Callable PyTorch module 3.2.1 Network Architectures To isolate the effect of activation functions, we compare two architectures: Table 1 : Architecture

Read the complete text