A Judge Agent Closes the Reliability Gap
in AI-Generated Scientific Simulation · 2026
Read A Judge Agent Closes the Reliability Gap by in AI-Generated Scientific Simulation (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-25780.txt.
Introduction
Yang shows that a judge agent can make AI-written scientific simulations far more reliable. Language models often write simulation code that fails in silence on hard problems. The judge applies classic checks: well-posedness, convergence, and error bounds. Across 134 test cases in 12 domains, silent failures fell from 42 percent to 1.5 percent. In a blind trial, 72 tasks from 12 outside scientists reached an 89 percent success rate with the judge. The rate without it was 53 percent. Yang also defines a spec format that makes problems machine-readable.
Catalog record
| CREATOR | in AI-Generated Scientific Simulation |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 110904 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.25780
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Judge Agent Closes the Reliability Gap in AI-Generated Scientific Simulation Abstract Abstract. Large language models can generate scientific simulation code, but the generated code silently fails on most non-textbook problems. We show that classical mathematical validation—well-posedness, convergence, and error certification—can be fully automated by a Judge Agent, reducing the silent-failure rate from 42% to 1.5% across 134 test cases spanning 12 scientific domains. The headline result comes from a prospective benchmark: 72 blinded tasks submitted by 12 independent scientists yield an 89% success rate (95% CI: [80%, 95%]) with automated error bounds, versus 53% without the Judge. On clinical CT (the only powered experiment, n=200 ), the pipeline reaches 99% of expert quality. The residual 1.5% concentrates at bifurcation points where certifiability breaks down. We formalize this boundary through the simulability class \mathfrak{S} and introduce spec.md , a structured specification format that makes any scientific computation problem machine-readable and solver-independent. Code, data, and all 72 benchmark tasks are publicly archived. Classical mathematical validation—well-posedness, convergence, error bounds—can be fully automated. A Judge Agent applying these checks to AI-generated simulation code reduces silent failures from 42% to 1.5% across 134 test cases (12 domains). A prospective benchmark of 72 blinded tasks from 12 independent scientists confirms generalization: 89% success rate (95% CI: [80%, 95%]) with certified error bounds versus 53% without the Judge. The residual 1.5% concentrates at bifurcation points, formalizable as the boundary of the simulability class \mathfrak{S} . Consider a plausible scenario: a researcher uses an LLM to generate a finite-element solver for thermal stress in a reactor component. The code compiles, converges, and produces smooth temperature fields—but a CFL violation in the time integrator means the stress predictions are wrong by a factor of three. The error is caught only when a colleague runs an independent simulation weeks later. Such silent failures are not hypothetical: in the SciCode benchmark [ 2 ] , frontier LLMs solve fewer than half of research-level computational problems correctly. The wrong answers look as plausible as the right ones. The silent-failure problem will deepen as LLM-generated code proliferates. Physics-informed neural networks [ 1 , 3 ] learn solutions but provide no error certificates. Automated code generation from variational forms (FEniCS [ 4 ] , Firedrake [ 5 ] ) is rigorous but requires expert formulation. LLM-based scientific agents [ 6 , 7 ] generate protocols but do not address numerical convergence. The reproducibility crisis [ 8 , 44 ] needs a validation layer, not more code generators. The mathematical tools for catching these failures already exist—Lax–Richtmyer convergence theory [ 9 ] , Hadamard well-posedness [ 10 ] , CFL stability conditions [ 27 ] —but applying them requires numerical analysis expertise that the typical end-user lacks. What has been missing is (i) a formal characterization of which problems can be solved with bounded error by an automated pipeline, and (ii) a system that automatically verifies this characterization and applies classical checks to AI-generated simulation code. We address both gaps with three interlocking contributions that form a hierarchy, not a list: The foundation : the simulability class \mathfrak{S} (Definition 1 ), which formalizes “what can be solved reliably by any computational system” through four implementation-independent conditions—finite specifiability (S1), Hadamard stability (S2), approximability (S3), and certifiability (S4). Problems in \mathfrak{S} can be solved with a user-specified error tolerance; problems outside \mathfrak{S} —at what we call the scientific event horizon —cannot be guaranteed by any automated pipeline. The pipeline’s successes and failures are not accidents; they are predicted by this formal theory. The mechanism : a Judge Agent ( \mathcal{A}_{J} ) that automates verification of S2–S4 on AI-generated code. Within a three-agent pipeline (Fig. 1 ), a Plan Agent ( \mathcal{A}_{P} ) translates natural language into a structured specification, \mathcal{A}_{J} validates \mathfrak{S} -membership via 5 pre-execution gates, and an Execute Agent ( \mathcal{A}_{E} ) generates and runs the solver. After execution, \mathcal{A}_{J} certifies the result (S4). Removing the Judge increases the silent-failure rate from 1.5% to 42% across 134 test cases—the headline empirical result. The interface : spec.md , a structured specification format that realizes S1. Every problem in \mathfrak{S} can be expressed as a valid spec.md file encoding the six-tuple (\Omega,\mathcal{E},\mathcal{B},\mathcal{I},\mathcal{O},\varepsilon) —domain, equations, boundary conditions, initial conditions, observables, and tolerance. The format is solver-independent: any compliant solver can consume a spec.md file and any Judge can verify it. The 72 prospective tasks are archived in this format, enabling independent replication without access to our specific pipeline. The underlying mathematics is entirely classical; the contribution is empirical evidence that automated \mathfrak{S} -verification substantially reduces silent failures across 12 scientific domains, with a median setup-time efficiency gain of \rho\approx 480\times relative to expert workflows. What this paper does not claim. This is not about building a better PDE solver: on any single well-defined problem, COMSOL [ 11 ] or FEniCS will produce a more accurate solution (Extended Data Table 5 ). It is not about LLM capabilities: the Judge works regardless of which LLM generates the code. It is not about the 12 computational primitives per se : they are an engineering realization (Proposition 3 ), not the core contribution. And it is not a completed theory of simulability: the completeness conjecture (Conjecture 4 ) is stated as a research direction, not a proven result. The error bound is conditional on a correct spec.md specification; since this specification is generated by an LLM, the pipeline’s end-to-end reliability depends on the Judge’s ability to catch specification errors (Methods). The pipeline retains a 1.5% qualitative failure rate at the boundary of \mathfrak{S} (“Failure analysis,” below). NL problem \mathcal{A}_{P} Plan spec.md \mathcal{A}_{J} \in\mathfrak{S} ? \mathcal{A}_{E} Execute \mathcal{A}_{J} Audit Bounded-error solution \hat{x} \in\mathfrak{S} Pass \notin\mathfrak{S} / Redesign Flag Pre-gates: well-posedness, stability, Lipschitz, cost Post-audit: conservation, monotonicity, symmetry, bounds, residuals, archetype Figure 1 : Three-agent pipeline grounded in the simulability class \mathfrak{S} . The Plan Agent translates natural language into a spec.md file (realizing S1). The Judge Agent verifies S2–S4 via 5 pre-execution gates and audits the solution via post-execution quality checks (realizing S4). Rejection triggers redesign (up to 3 rounds) or a certificate that the problem lies outside \mathfrak{S} . (a) spec.md structure #Specification: {Title} header ##Domain S1.1: \Omega — geometry, dimension, grid ##Equations S1.2: \mathcal{E} — PDEs/operators + parameters ##Boundary Conditions S1.3: \mathcal{B} — Dirichlet, Neumann, constraints ##Initial Conditions S1.4: \mathcal{I} — starting state or N/A ##Observables S1.5: \mathcal{O} — output quantities + units ##Tolerance S1.6: \varepsilon — numerical thresholds + metric ##Primitives Required optional — links to 12-primitive basis ##Task Instance Variations optional — public/dev/hidden splits 6 mandatory sections (S1 six-tuple) Syntax: YAML-compatible key: value — simple field key: | — multi-line block #comment — inline annotation - item — list entry [a, b] — array (b) Example: CT reconstruction #Specification: CT Reconstruction (LoDoPaB) ##Domain domain: 2D image grid geometry: 128 x 128 pixels pixel_size: 0.7 mm #from LoDoPaB-CT dimension: 2 ##Equations equations: | forward: y = Radon(x) + eta inverse: min_x 0.5*||Radon(x)-y|| ˆ 2 + lam_TV*TV(x) + lam_L2*||x|| ˆ 2 constraint: x >= 0 parameters: n_angles: 128 #projection angles n_detectors: 183 #per angle geometry: parallel_beam noise_model: Poisson lambda_TV: 1.0e-3 #Morozov image_size: [128, 128] ##Boundary Conditions boundary: | non_negativity: x(i,j) >= 0 support: x = 0 outside body contour