When Good Equations Get Bad Scores: Improving Symbolic Regression Through Better Parameter Optimization
Boxiao Wang, Kai Li, Zhiwei Chen, Yang Huang, Runxiang Wang, Ziwen Zhang, Yifan Zhang, Jian Cheng · 2026
Read When Good Equations Get Bad Scores: Improving Symbolic Regression Through Better Parameter Optimization by Boxiao Wang, Kai Li, Zhiwei Chen, Yang Huang, Runxiang Wang, Ziwen Zhang, Yifan Zhang, Jian Cheng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-23272.txt.
Introduction
Wang and seven coauthors propose SAGE-Fit, a symbolic regression fitting tool for the Good Structure, Bad Score problem. Most methods use a bi-level setup: an outer loop searching for form, and an inner loop tuning smooth settings. Curved ops render the inner loop highly non-convex, and fast local solvers often yield poor local minima and low scores for correct forms. SAGE-Fit uses dual native priors of symbolic forms. Tests show SAGE-Fit improves scoring truth and various systems.
Catalog record
| CREATOR | Boxiao Wang, Kai Li, Zhiwei Chen, Yang Huang, Runxiang Wang, Ziwen Zhang, Yifan Zhang, Jian Cheng |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60418 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.23272
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
When Good Equations Get Bad Scores: Improving Symbolic Regression Through Better Parameter Optimization Boxiao Wang, Kai Li, Zhiwei Chen, Yang Huang, Runxiang Wang, Ziwen Zhang, Yifan Zhang, Jian Cheng Abstract Symbolic Regression (SR) plays a central role in scientific knowledge discovery by distilling mathematical equations from observational data. Most existing SR methods function within a bi-level optimization framework: an outer loop that searches for the discrete equation structure, and an inner loop that optimizes the continuous parameters of that structure. Crucially, parameter-fitting quality directly determines a structure’s score and thus the outer-loop search. However, nonlinear operators make the inner loop highly non-convex, and budget-driven reliance on fast local solvers (e.g., BFGS) often yields poor local minima and underestimated scores for correct structures. This “Good Structure, Bad Score” phenomenon becomes a key bottleneck, degrading efficiency and misguiding the search away from the true equation. To resolve this, we propose SAGE-Fit ( S tructure- A ware and Semantics- G uided E valuator for Symbolic Regression), an SR-native fitting framework that exploits the dual native priors of symbolic expressions. By capitalizing on the structural and semantic priors unique to SR, we design tailored modules for each property, thereby effectively mitigating this optimization bottleneck. Extensive experiments demonstrate that our approach, as a plug-and-play module, significantly enhances evaluation fidelity and universally improves the performance of various SR systems. 1 Introduction Symbolic Regression (SR) differs from traditional regression by simultaneously searching for optimal equation structures and the associated parameters, with the goal of discovering interpretable physical laws from data ( 34 ; 2 ) . While methodologies have evolved from genetic programming ( 22 ; 34 ; 41 ) to reinforcement learning ( 31 ; 24 ; 14 ; 25 ) and large language models ( 37 ; 30 ; 15 ; 12 ) , the underlying paradigm remains a bi-level optimization framework . This framework consists of an outer loop that proposes discrete equation skeletons (e.g., expression trees or token sequences) and an inner loop that optimizes the continuous constants (parameters) within these structures. Ideally, this decomposition allows the outer loop to focus on structural exploration while the inner loop handles numerical fitting. However, we identify a critical asymmetry in current SR research: while the outer loop has benefited from sophisticated architectural innovations, the inner loop largely relies on generic local optimizers such as BFGS or Nelder-Mead ( 39 ; 9 ) . This reliance assumes that these fast solvers can reliably assess the quality of a structure. In reality, nested nonlinear operators (e.g., exponentials and trigonometric functions) create a highly non-convex loss landscape with numerous local minima—a “needle-in-a-basin” topology that generic optimizers struggle to navigate. Consequently, structurally correct equations often receive poor fitness scores because the local optimizer fails to locate the global parameters, a phenomenon we term “Good Structure, Bad Score.” The repercussions of this inner-loop failure are twofold. First, it degrades evaluation fidelity : the search algorithm is misled to discard promising candidates, which prevents convergence to the true physical law. Second, it wastes computational resources on repeated, ineffective optimization attempts. As demonstrated in our experiments, even when the outer loop successfully identifies the exact ground-truth skeleton, standard optimizers frequently fail to fit the parameters, rendering the discovery process futile. Thus, the bottleneck of modern SR has shifted from the generation of valid structures to the accurate evaluation of these structures. To understand when and why this happens, we conduct a landscape-level case analysis that characterizes optimization failures in SR from two complementary viewpoints: correct-structure landscapes and landscapes of partially correct structures. Instead of relying on black-box continuous solvers that ignore equation structures, we argue that a highly faithful evaluator must inherently respect the distinct nature of symbolic expressions. To this end, we propose SAGE-Fit ( S tructure- A ware and Semantics- G uided E valuator for Symbolic Regression), a unified parameter fitting framework that explicitly exploits the dual native priors of symbolic equations. Specifically, to target the structural prior, we leverage the hierarchical parameter distribution within the expression tree to design a Tree-Directed Variable Projection that deterministically decouples conditionally linear parameters; concurrently, by utilizing the explicitly known analytic equation structure upon the reduced non-linear manifold, we formulate an SR-specialized Projected Gauss–Newton method to achieve highly efficient local convergence. To target the semantic prior, we address the critical risk of basin collapse inherent to random parameter-space initialization by introducing Semantics-Guided Initialization . By evaluating the actual input-output behavioral mappings of candidates rather than relying on deceptive distances in the parameter space, we design a Function-Space Farthest-Point Sampling (FS-FPS) mechanism to guarantee that the starting points are maximally diverse in their true semantic behavior. Finally, we integrate these unified modules as a plug-and-play evaluator into representative SR frameworks spanning genetic programming, reinforcement learning, and LLM-based candidate generation. Across these diverse settings, upgrading the inner-loop optimizer consistently improves both symbolic accuracy and numerical fidelity, thereby demonstrating that enhancing inner-loop optimization alone can substantially strengthen mainstream SR systems. 2 Related Work With the increasing adoption of two-part SR paradigms, the structure discovery module determines the operators and compositional form of the expression, while parameter fitting module refines the numerical constants via continuous optimization algorithms under the fixed structure. This decomposed strategy substantially reduces the complexity of the overall search space, decouples structure discovery from parameter estimation and improves the reliability and stability of candidate structure evaluation. Since this work focuses on the bi-level formulation, we review prior studies from two complementary perspectives: (i) outer-loop structure search, and (ii) inner-loop parameter optimization. 2.1 Structure Search Early SR methods primarily relied on deterministic modeling and sparse regression, such as FFX ( 29 ; 40 ; 19 ) and SINDy ( 3 ; 4 ; 39 ) . These methods exhibit high efficiency and stability when the hypothesis space is constrained to a predefined library of basis functions; however, their expressive power is largely limited by the design of the function library. To expand the searchable function space, genetic programming (GP)-based SR methods have been extensively studied ( 22 ; 34 ; 41 ) . These methods typically represent expressions as trees and perform global search over the symbolic composition space via evolutionary operators such as mutation and crossover. Building upon this paradigm, methods such as PySR ( 7 ) further adopt multi-objective formulations for candidate selection, leading to improved search efficiency and expression quality. With the advancement of neural network methodologies, SR has increasingly been combined with neuro-symbolic and search-driven approaches. For instance, Equation Learner (EQL) replaces standard neural network activation functions with interpretable mathematical operators ( 28 ; 33 ; 8 ) , enabling the learning of symbolically meaningful functional connections. Other methods, including DSR ( 31 ) , DSO ( 24 ; 14 ) , and DSN ( 25 ) , formulate SR as a sequential decision-making problem and employ reinforcement learning to explore the expression space. In addition, several studies integrate Monte Carlo Tree Search (MCTS) with deep generative models to enhance global exploration and the stability of symbolic expression search ( 18 ; 16 ) . More recently, SR has witnessed growing interest in large language model (LLM)-driven approaches ( 37 ; 30 ; 15 ; 12 ) . Benefiting from strong capabilities