DiscoverPhysics: Benchmarking LLMs for Out-of-the-Box Scientific Thinking
Matt L. Wiemann, Lindsay M. Smith, Peter Melchior, Siddharth Mishra-Sharma, Andrew Gordon Wilson, Pavel Izmailov, Carolina Cuesta-Lázaro · 2026
Read DiscoverPhysics: Benchmarking LLMs for Out-of-the-Box Scientific Thinking by Matt L. Wiemann, Lindsay M. Smith, Peter Melchior, Siddharth Mishra-Sharma, Andrew Gordon Wilson, Pavel Izmailov, Carolina Cuesta-Lázaro (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26087.txt.
Introduction
Wiemann and six colleagues test whether language models can discover physics rather than recall it. Frontier models ace standard physics tests, but those scores may reflect memorised science. DiscoverPhysics builds 22 mock worlds whose laws break ours on purpose, with odd gravity, mixed species, hidden matter-like particles and shifting interactions. An agent runs rounds of tests, watches raw motion data, and finally submits a plain-language account plus Python code for the inferred law. Success demands sharp tests and steady refinement of guesses.
Catalog record
| CREATOR | Matt L. Wiemann, Lindsay M. Smith, Peter Melchior, Siddharth Mishra-Sharma, Andrew Gordon Wilson, Pavel Izmailov, Carolina Cuesta-Lázaro |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 101675 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.26087
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DiscoverPhysics: Benchmarking LLMs for Out-of-the-Box Scientific Thinking Matt L. Wiemann, Lindsay M. Smith, Peter Melchior, Siddharth Mishra-Sharma, Andrew Gordon Wilson, Pavel Izmailov, Carolina Cuesta-Lázaro Abstract Frontier LLMs now perform strongly across a wide range of physics evaluations, but it is hard to disentangle genuine reasoning from recall of established science. We introduce DiscoverPhysics , an interactive benchmark that asks a LLM agent to discover the laws of motion of a simulated world whose physics deliberately deviates from our own. We construct 22 worlds governed by, among others, screened and fractional-power gravity, multi-species couplings, hidden dark-matter-like particles, non-coordinate-free physics, and time-varying interactions. Each world is generated on demand by an N-body simulator, for which the agent proposes several rounds of experiments, observes raw trajectory data, and ultimately submits both a natural-language explanation of the world’s physics and a Python implementation of the inferred law. Because solving a world requires the agent to design informative experiments and revise its hypotheses, the benchmark probes long-horizon reasoning over an experimental history. We evaluate submissions along two complementary axes: trajectory MSE on held-out particles and an LLM-judged explanation score following an expert-written rubric assessing conceptual understanding of each world. Across eleven frontier models, we find that the strongest agents pass only half of the worlds and consistently fail on those where latent structure must be uncovered. Open-source models lag substantially behind commercial models, both in their ability to design informative experiments and in extracting conclusions from the data. We further find that good predictive accuracy does not guarantee high explanation quality and that conceptual understanding depends on hypothesis refinement through well-chosen experiments. DiscoverPhysics Leaderboard Figure 1: Schematic of the DiscoverPhysics benchmarking pipeline. Our benchmark does not store data products, instead data is generated on demand by a N-body simulator from the equations specified by the world definition. We then generate an initial set of particle trajectories and feed this, as well as the agent prompt about task and experiment instructions, to an LLM. The LLM agent is tasked with discovering the physical laws of the world it is operating in and can propose experiments to be run by the N-body simulator or perform parameter fitting on proposed laws. The agent may experiment for up to n rounds, after which a final law and text explanation must be submitted, both of which are then scored to determine if the agent passed. 1 Introduction Large language models (LLMs) already excel at a wide range of specialized tasks like competitive math, coding and graduate-level scientific knowledge. But these represent only a fraction of the skills needed for scientific discovery in the physical sciences, which also requires generating hypotheses, designing experiments, updating beliefs based on noisy observations, and sometimes giving up preconceived notions entirely. Existing evaluations cover slices of the discovery skill space. Knowledge benchmarks measure what a model has learned; reasoning benchmarks measure inference over given inputs; symbolic regression benchmarks measure the recovery of known equation families from fixed datasets. Recent interactive physics benchmarks come closer, but they typically present canonical laws with shifted parameters and a known set of relevant variables, leaving the conceptual part of scientific discovery largely unprobed. We introduce DiscoverPhysics , an extensible benchmark of simulated worlds whose physics is deliberately non-canonical: short-range exponentially screened forces, fractional Laplacians, hidden particle species, non coordinate-free physics, time-varying couplings, and extra dimensions are some of the examples. Agents interact with each world by placing test particles, observing their trajectories under noise, and iteratively refining their hypothesis about the underlying law. Because the laws governing these worlds are not standard, the agent must identify which features of the observed dynamics are relevant and build mechanistic models that, ideally, predict yet unseen data in this world. The worlds are curated rather than procedurally generated, but they deliberately span a diverse set of non-standard force laws, designed to probe whether LLM agents can discover unusual equations of motion through experimentation and altering their initial guesses. Our benchmark also seeks to define evaluation metrics for scientific discovery. Predictive accuracy, often the single performance metric for ML tasks, alone does not capture what makes a scientific theory valuable. It also needs to provide a conceptual explanation of a phenomenon that generalizes beyond the observations that motivated it [ 6 ] . The history of the natural sciences is full of examples where empirical models fit the data long before the underlying principle was identified. Alchemy attempted to explain the transformation of substances before modern notions of chemistry and elements existed. In physics, Lorentz transformations were derived empirically to explain electromagnetic phenomena before Einstein recognized they follow from a single principle: the invariance of the speed of light. Similarly, energy conservation was long observed in mechanical systems long before Noether’s theorem revealed why: conservation laws arise from continuous symmetries. These insights, not just the specific form of the equations, are what allow strong theories to generalize far beyond the initial experiments. DiscoverPhysics therefore evaluates agents along two axes. Trajectory MSE measures how well the agent’s discovered law predicts the dynamics on held-out test particles. An explanation score, computed by an LLM judge using a human-written rubric, measures whether the agent has correctly identified the conceptual features of the world that explain those dynamics. Our contributions are: • We introduce DiscoverPhysics , an extensible benchmark of simulated worlds with non-canonical physics that probes scientific discovery through active experimentation, including the design of informative experiments and the iterative refinement of hypotheses. • We define a dual evaluation framework that assesses predictive accuracy (trajectory MSE) and explanatory adequacy (LLM-judged explanation score), capturing whether agents fit the data, identify the underlying principles, or both. • We benchmark 11 frontier and open-source LLMs and find that the strongest agents pass only about half of the worlds, consistently failing on those that require uncovering latent structure such as hidden particle species or extra dimensions. • We show that predictive accuracy and conceptual understanding can decouple: the model achieving the lowest trajectory MSE is not the one with the highest explanation score, and benefits less from additional experimental rounds, which we read as a tendency to fit data without revising its conceptual picture. • We show that open-source models substantially lag frontier models, both in designing informative experiments and in extracting conclusions from the resulting data, with their performance largely unchanged between guided and randomized experimentation. • We release the simulator, public world definitions, and evaluation framework to enable evaluation of future LLMs and community extensions to new worlds. 1 1 1 For access to full public and private benchmarking suite request access via HuggingFace: DiscoverPhysics 2 Related work Scientific discovery. LLMs are employed to accelerate scientific research, from probabilistic frameworks for model discovery [ 18 ] to coding agents that evolve algorithms for open but targeted scientific problems [ 12 ] . Systems like The AI Scientist [ 9 ] take a most ambitious framing, automating the full research lifecycle including ideation, experimentation, and writeup. These works aim to establish that LLMs can meaningfully participate in scientific workflows, but their evaluation typically targets paper-level quality judged by LLMs rather than recovery of a ground-truth mechanism, leaving open the question of how well models perform discovery against an objective standard.