LLM-SRBench : A New Benchmark for Scientific Equation Discovery with Large Language Models
Parshin Shojaee ∗, Ngoc-Hieu Nguyen ∗, Kazem Meidani, Amir Barati Farimani, Khoa D Doan, Chandan K Reddy · 2025
Read LLM-SRBench : A New Benchmark for Scientific Equation Discovery with Large Language Models by Parshin Shojaee ∗, Ngoc-Hieu Nguyen ∗, Kazem Meidani, Amir Barati Farimani, Khoa D Doan, Chandan K Reddy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-10415.txt.
Introduction
Parshin Shojaee and five co-workers wrote a 2025 paper building a benchmark for testing whether language models can truly discover scientific equations. Old benchmarks reuse famous formulas that models may simply recall. Their set holds over two hundred problems across four fields. One branch rewrites known models in uncommon forms to foil recall.
The other branch invents fresh problems that demand data-driven reasoning. Top current systems hit only about a third symbolic accuracy. Both open and closed models were run through the set. The authors offer the benchmark as a sterner yardstick for discovery claims.
Catalog record
| CREATOR | Parshin Shojaee ∗, Ngoc-Hieu Nguyen ∗, Kazem Meidani, Amir Barati Farimani, Khoa D Doan, Chandan K Reddy |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-10415
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LLM-SRBench : A New Benchmark for Scientific Equation Discovery with Large Language Models Parshin Shojaee ∗, Ngoc-Hieu Nguyen ∗, Kazem Meidani, Amir Barati Farimani, Khoa D Doan, Chandan K Reddy Abstract Scientific equation discovery has long been a cornerstone of scientific progress, enabling the derivation of laws governing natural phenomena. Recently, Large Language Models (LLMs) have gained interest for this task due to their potential to leverage embedded scientific knowledge for hypothesis generation. However, it is difficult to assess the true discovery capabilities of these methods because existing benchmarks often use well-known equations. This makes them vulnerable to memorization by LLMs and results in inflated performance metrics that do not reflect genuine discovery. In this paper, we introduce LLM-SRBench, a comprehensive benchmark with 239 239 challenging problems across four scientific domains specifically designed to evaluate LLM-based scientific equation discovery methods while preventing trivial memorization. Our benchmark comprises two main categories: LSR-Transform, which transforms common physical models into less common mathematical representations to test reasoning beyond memorized forms, and LSR-Synth, which introduces synthetic, discovery-driven problems requiring data-driven reasoning. Through extensive evaluation of several state-of-the-art methods, using both open and closed LLMs, we find that the best-performing system so far achieves only 31.5 % 31.5\% symbolic accuracy. These findings highlight the challenges of scientific equation discovery, positioning LLM-SRBench as a valuable resource for future research. † † affiliationnotice: Equal contribution Website: https://github.com/deep-symbolic-mathematics/llm-srbench Figure 1 : Error analysis comparing simple LLM sampling (Llama-3.1-8B) on 100 Feynman problems versus LLM-SRBench datasets (LSR-Transform and LSR-Synth). The sharp drops in numeric error curves and considerably lower symbolic error for Feynman problems suggest memorization rather than gradual discovery. 1 Introduction Equation discovery, the process of uncovering symbolic mathematical expressions from observational data, has been a cornerstone of scientific advancement. This task, also known as symbolic regression (SR), goes beyond mere data-driven predictive modeling by seeking interpretable mathematical relations that reveal the underlying mechanisms of natural phenomena. When scientists derive mathematical equations from empirical data, they gain more than just predictive power – they obtain insights into fundamental physical principles, enable extrapolation beyond observed data, and facilitate knowledge transfer across scientific domains ( Langley, 1981 ; Schmidt & Lipson, 2009 ) . Figure 2 : Overview of the LLM-based Scientific Equation Discovery. The benchmark tasks ( left ) combine scientific context with numerical data. The discovery process ( middle ) iteratively leverages LLM’s scientific knowledge and data-driven reasoning to generate hypotheses for underlying equations. Discovered hypotheses, represented as equation strings, trees, or programs, are then evaluated ( right ) using multiple metrics including data fidelity, symbolic accuracy, and computational efficiency. Standard approaches to equation discovery have primarily relied on genetic programming (GP) and evolutionary algorithms ( Cranmer, 2023 ; La Cava et al., 2021 ) , which represent mathematical expressions as trees and navigate the vast space of possible equations through evolutionary search techniques. However, these methods face two fundamental challenges. First, the NP-hard nature of equation discovery ( Virgolin & Pissis, 2022 ) makes their random mutation and crossover operations computationally prohibitive across vast search spaces. Second, unlike human scientists who leverage their domain knowledge and expertise to guide hypothesis formation, these approaches are mostly purely data-driven, and isolated from existing scientific knowledge. These limitations have motivated researchers to develop methods that incorporate scientific domain knowledge into the equation discovery process. Large Language Models (LLMs) have recently emerged as a promising solution to these challenges, offering a new paradigm for scientific equation discovery. LLMs, trained on vast corpora of scientific literature, possess extensive embedded scientific knowledge. This has sparked significant interest in leveraging LLMs for scientific equation discovery, with several recent works demonstrating their potential ( Shojaee et al., 2024b ; Ma et al., 2024 ; Grayeli et al., 2024 ; Merler et al., 2024 ; Du et al., 2024 ; Reddy & Shojaee, 2024 ; Zhang et al., 2024 ) . These LLM-based approaches have shown to enhance the equation hypothesis generation process by incorporating scientific priors, guiding the exploration of equation search spaces more efficiently, and providing interpretable reasoning for the search process. Despite the promising potential of LLM-based equation discovery methods, their rigorous and robust evaluation still remains an open challenge. The current scientific equation discovery benchmarks are primarily represented by SRBench ( La Cava et al., 2021 ) and SRSD ( Matsubara et al., 2022 ) . SRBench incorporates two key data groups for this purpose: the Feynman physics equations ( Udrescu & Tegmark, 2020 ) , and Strogatz dynamical systems ( La Cava et al., 2016 ; Strogatz, 2018 ) . A notable extension to this framework is SRSD ( Matsubara et al., 2022 ) , which enhances the Feynman benchmark by incorporating physically meaningful sampling ranges for data points. However, these benchmarks exhibit significant limitations for the evaluation of LLM-based methods. Their problems are mostly based on known physics equations from textbooks, which makes them often subject to memorization by LLMs. As noted by ( Shojaee et al., 2024b ) , LLMs frequently succeed on these common equation discovery benchmarks through simple recitation based on variable names and problem descriptions, rather than the actual process of data-driven discovery and reasoning. Our analysis (shown in Fig. 1 ) also confirms this finding - the sudden drop in the numeric error curve within the first few iterations and significantly lower symbolic error on Feynman problems indicate memorized solutions rather than a meaningful search towards discovery. To mitigate this issue, ( Shojaee et al., 2024b ; Ma et al., 2024 ) have introduced a handful of five custom-crafted problems designed to prevent memorization by manually modifying known physical models. While these efforts represent a step forward, the small scale and limited diversity of these problem sets are insufficient to provide a comprehensive evaluation framework for emerging LLM-based methods in scientific equation discovery. A more robust and systematic benchmark is needed to enable standardized evaluation and foster the development of innovative methods in this emerging field. In this paper, we introduce LLM-SRBench , a new benchmark designed to rigorously evaluate the capabilities of LLM-based scientific equation discovery methods. LLM-SRBench addresses the limitations of existing benchmarks by constructing problem sets that avoid trivial recitation while leveraging the scientific priors embedded in LLMs, simulating conditions akin to scientific discovery. The benchmark is structured around two main categories of problems, each targeting distinct aspects of equation discovery. The first category focuses on transforming common scientific problems, such as those from the Feynman equations, into different mathematical representations of the same underlying physical problem. By symbolically altering input-output mappings and generating less common mathematical forms for the same problem, we challenge LLM-based equation discovery to go beyond memorization of the common forms. This approach is motivated by recent findings on the fragility of LLMs’ reasoning capabilities to unfamiliar representations of otherwise familiar problems ( Mirzadeh et al., 2024 ; Xie et al., 2024 ; Wu et al., 2023 ) . The second category extends the approach introduced by ( Shojaee et al., 2024b ) , which combines known terms in the underlying equation with synthetic, novel terms to create problems that go beyond memorization and demand data-driven reasoning. We expand this idea into a comprehensive set of benchmark problems