Physics Is All You Need? A Case Study in Physicist-Supervised AI Development of Scientific Software
Nhat-Minh Nguyen · 2026
Read Physics Is All You Need? A Case Study in Physicist-Supervised AI Development of Scientific Software by Nhat-Minh Nguyen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30353.txt.
Introduction
Nhat-Minh Nguyen presents a study of a physicist's AI coding agent over 57 sessions to build CLAX-PT, a perturbation theory module in JAX. The agent resolved ten of 15 events against oracle tests, and two more with the physicist's knowledge. The three it could not, evading detection, mistook symptom reduction for root-cause resolution. It spent 33 sessions tuning coefficients within an architecture that could not match the physics. Three supervision practices proved critical: testing, shared changelogs, and a rule against unphysical patches. Supervision design, not capability, determined trustworthiness.
Catalog record
| CREATOR | Nhat-Minh Nguyen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41477 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.30353
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Physics Is All You Need? A Case Study in Physicist-Supervised AI Development of Scientific Software Nhat-Minh Nguyen Abstract Are AI agents tools, co-authors, or researchers? We present a quantified case study ( N = 1 N=1 ): a physicist supervising an AI coding agent (Claude Code, Sonnet and Opus models) over 12 work days and 57 sessions to build a differentiable one-loop perturbation theory (a next-to-leading-order calculation for predicting galaxy clustering) module in JAX, clax-pt ( ∼ 2,100 {\sim}2{,}100 lines, validated to ≲ 1 % {\lesssim}1\% accuracy against the established C reference class-pt ). We documented 15 supervision events during the v0.1.0 development window and classified each by intervention level (Appendix A ). The agent resolved ten autonomously (convention errors, algorithm transcription, numerical coefficients) by iterating against oracle test suites. Two more were accelerated by the physicist spotting magnitude discrepancies invisible to shape-based comparisons. The three it could not—all evaded oracle detection—share a common property: the agent treated symptom reduction as equivalent to root-cause resolution. It spent 33 of the 57 sessions adjusting coefficients within a code architecture that could not represent the target physics, and could not re-evaluate its choice of class-pt branch even when the physicist explicitly prompted reconsideration; only an injected physics concept (anisotropic BAO damping) triggered the redesign. Separately, the agent committed a calibrated scalar correction that passed all oracle tests, but the value corresponded to no quantity in the reference theory and would produce wrong predictions at any other cosmology. The fudge factor was caught and replaced within the same session. Three supervision practices, developed iteratively, proved critical for catching what oracle tests missed: testing at diverse parameter points beyond the fiducial calibration; shared changelogs that surfaced stalled exploration across sessions; and an explicit rule against unphysical numerical patches. In this case study, the design of these supervision protocols, not model capability, was the primary factor in whether the agent’s output was trustworthy. Closing the gap we observed would require agents that can propose architectural alternatives rather than optimize within a given structure, and distinguish predictive adequacy from explanatory correctness. The agent in this case study did not exhibit these capabilities, and they are not obviously addressed by scaling alone. 1 Introduction Most empirical evidence on AI coding agents in science comes from one of two regimes: standardized coding benchmarks and large-scale software projects where automated testing fully captures correctness ( 2 ) , or fully autonomous multi-agent systems that generate manuscripts without sustained human oversight ( 14 ) . Neither captures the reality of scientific software development, where correctness is defined by agreement with physical law rather than by compilation or test passage, and where physicists work with AI agents to produce and validate a scientific software. We present a detailed case study of exactly this scenario. Over 12 work days and 57 sessions, a physicist supervised an AI coding agent (Claude Code, Sonnet and Opus models) to build clax-pt , a differentiable one-loop perturbation theory module (computing next-to-leading-order predictions for galaxy clustering) in jax . The module computes nine output power spectra validated to {\lesssim}1\% accuracy against the established reference code class-pt ( 5 ) . The resulting implementation comprises approximately 2,100 lines of code. The contribution of this paper is not the code itself (described in a companion paper) but the empirical supervision record: what the human did, what the agent did, and where the boundary between their roles became load-bearing. Our central claim is that the supervision protocol—not model capability—was the primary factor determining whether the agent’s output was trustworthy scientific software. The agent autonomously resolved 10 of the 15 documented issues by iterating against oracle test suites. Two more were accelerated by the physicist’s domain knowledge input. The three it could not resolve were invisible to oracle testing and required human physics judgment to diagnose. These three bugs consumed a disproportionate fraction of the project: 33 of 57 sessions were spent within a fundamentally wrong code architecture that the agent could not recognize as wrong because it passed local consistency checks. The physicist’s interventions—an architectural redesign and the rejection of a physically unmotivated numerical patch—were the decisive acts that produced correct code. Both share a common cognitive structure. The physicist assessed not whether the code produced right numbers, but whether it produced them for the right reasons . 2 The Project 2.1 What Was Built We compute predictions for the gravitational clustering of galaxies, the three-dimensional distribution mapped by galaxy redshift surveys. The calculation involves loop integrals (analogous to next-to-leading-order corrections in quantum field theory) that are expensive to evaluate and sensitive to subtle physical effects: bulk motions of matter and oscillation features imprinted by sound waves in the early universe. Getting these effects wrong by even 1% can bias the cosmological parameters (the parameters of the standard model of cosmology) inferred from the galaxy power spectrum. One-loop perturbation theory (the next-to-leading-order correction beyond the linear approximation) extends the linear matter power spectrum to mildly nonlinear scales by computing correction terms from second- and third-order density fields, producing predictions for galaxy clustering that are essential for extracting cosmological parameters from spectroscopic surveys ( 8 ; 6 ) . clax-pt implements this calculation in jax : it takes a linear power spectrum and cosmological parameters as input, evaluates the tree-level and one-loop terms (the latter via FFTLog decomposition ( 13 ; 7 ) ), applies infrared (IR) resummation ( 12 ; 15 ) , and adds ultraviolet (UV) counterterms that absorb sensitivity to small-scale physics outside the perturbative regime ( 1 ; 3 ; 4 ) . IR resummation corrects for large-scale bulk flows which, if unaccounted for, artificially smear the baryon acoustic oscillation (BAO) feature, the standard-ruler imprint of primordial sound waves. The code returns nine validated output spectra ( Table 1 ). The implementation is approximately 2,100 lines of jax code, end-to-end differentiable, and validated to {\lesssim}1\% accuracy against class-pt ( 5 ) . Technical details are described in a companion paper. Table 1: Accuracy of clax-pt vs. class-pt at Planck 2018 fiducial cosmology, z=0.38 , k<0.3\;h/ Mpc. Hexadecapole uses |\Delta|/\max(|\mathrm{ref}|) due to zero crossings. Spectrum Max error Mean error P_{mm},P_{gg},P_{gm} (real-space) 0.31% 0.04% Monopole ( \ell=0 ) 0.59% 0.40% Quadrupole ( \ell=2 ) 0.89% 0.50% Hexadecapole ( \ell=4 ) 1.43% 0.37% 2.2 The Supervision Protocol Before any code was written, the physicist established a supervision protocol adapted from the methodology of Anthropic’s C-compiler agent project ( 2 ) , which used 16 parallel agents over 2,000 sessions to build a compiler that successfully builds the Linux kernel. Four infrastructure elements structured every session: 1. class-pt as oracle. Every function was tested against reference data generated by the established C implementation. Tests were written before code—the agent knew what the correct output looked like before attempting to produce it. 2. CHANGELOG as shared memory. Because each agent session starts with no memory of previous sessions, a structured log recorded what was attempted, what failed, and what succeeded. This prevented later sessions from re-exploring dead ends (e.g., a discrete sine transform (DST) grid bug that was solved on day 3 was never re-investigated). 3. A --fast flag for context-window hygiene. Tests printed at most 10 lines on success and 20