Computational references are not experiments: pre-registered validation of machine-learned sodium-cathode voltages
Krishna Teja Vepa · 2026
Read Computational references are not experiments: pre-registered validation of machine-learned sodium-cathode voltages by Krishna Teja Vepa (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-23725.txt.
Introduction
Vepa reports a pre registered validation failure for a machine learning screen of sodium battery cathodes, and retires the screen, in a 2026 paper on materials informatics. Learning based screens are trained and judged against computed voltages from simulation databases. Those references carry systematic errors, so a model can ace every benchmark while inheriting a scale that disagrees with experiment.
On an audited set of known sodium ion cathodes, the raw held out error averages well over half a volt, and the conservative metric is worse. The error grows with voltage, so no simple calibration shift can rescue it. Where model, database, and experiment can all be compared, the database reference itself sits far below measurement, dominating the error. A prior art search finds most of the targeted substitution space already published, and a follow up audit retires absolute voltage claims from the author's own ledger too.
Catalog record
| CREATOR | Krishna Teja Vepa |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.23725
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Computational references are not experiments: pre-registered validation of machine-learned sodium-cathode voltages Krishna Teja Vepa Abstract Machine-learning screens for battery materials are trained and judged almost entirely against computed reference voltages, and those references carry their own systematic errors. We report a case in which this matters quantitatively: our own screening stack (a graph-network voltage screen, a prior-art triage layer, and a local PBE+ U U bench) fails pre-registered validation against experiment-anchored literature values. Verdict thresholds, failure modes, and the primary metric were committed before analysis. On an operator-audited set of known Na-ion cathodes ( n = 6 n=6 after one documented exclusion; verdict unchanged at n = 7 n=7 ), the raw held-out mean absolute error was 0.67 V 0.67\text{\,}\mathrm{V} , the pre-registered conservative metric, the upper 95% confidence bound of the cross-validated bias-corrected error, was 1.09 V 1.09\text{\,}\mathrm{V} , and the residual was strongly voltage-dependent ( r = − 0.94 r=-0.94 ), so no additive calibration is valid. On the two compounds where prediction, database reference, and experiment could all be compared, the Materials Project PBE+ U U reference sat about 0.54 V 0.54\text{\,}\mathrm{V} below measurement: the reference, not the model, dominated the error. A prior-art screen found at least 70% of the targeted Na substitution space already published. We retire the screen, bound what “verified” means for our DFT ledger, and run a pre-registered calibration audit of it against four benchmark Li couples. That audit returns a standard deviation of 0.31 V 0.31\text{\,}\mathrm{V} , above its 0.30 V 0.30\text{\,}\mathrm{V} bar, so absolute-voltage claims are retired from our own ledger as well; the residual is chemistry-structured (the layered- Co and Fe -phosphate couples share + 0.41 V +$0.41\text{\,}\mathrm{V}$ to within 0.09 V 0.09\text{\,}\mathrm{V} , with the Mn spinel apart), pointing to a per-chemistry offset rather than a single global one. I Introduction Generative models now propose inorganic crystals far faster than any laboratory, or any density-functional-theory (DFT) queue, can check them. The GNoME release alone reported 2.2 million candidate structures and declared roughly 380,000 of them stable 27 , and an autonomous laboratory reported synthesizing dozens of those candidates within weeks 35 . The published record since then has been less kind. Cheetham and Seshadri found that almost none of the examined GNoME compounds were simultaneously credible, new, and useful by the standards a chemist would apply 8 , and a re-examination of the autonomous-synthesis claims concluded that none of them was convincingly demonstrated 25 . The field’s binding constraint has moved from generating candidates to deciding which claims about them deserve trust. That decision rests on reference data more often than the literature acknowledges. Screening models for battery cathodes are typically trained on intercalation voltages computed within GGA or GGA+ U 1 ; 39 , most commonly the Materials Project values 19 , and they are usually evaluated against the same computed quantities. A model can therefore pass every benchmark available to it while inheriting, invisibly, whatever systematic error separates its reference scale from the electrochemical measurements the field actually cares about. Validation against experiment is the only test that can expose this, and it is the test least often run. This paper runs that test on our own system, under rules fixed in advance. The system under test (Sec. II ) is a small closed-loop materials pipeline (generator, graph-network voltage screen, prior-art triage layer, and a local PBE+ U verification bench) built and operated by one person. Before any validation metric was computed, we committed verdict thresholds, a conservative primary metric, and four named failure modes to the repository, and a fifth stop condition was fixed before literature curation began, so that the result could not be renegotiated after the fact (Sec. III ); the commit hashes are the audit trail 31 . The outcome is negative on every axis we pre-registered. The voltage screen is not screening-grade against experiment, and its residual structure forbids the additive recalibration that would otherwise be the obvious patch (Sec. IV ). The computed references themselves sit about half a volt below measurement on the rows where a three-way comparison was possible, which redistributes blame in an uncomfortable direction: part of what the model learned to reproduce was reference error, not electrochemistry (Sec. V ). And the targeted “discovery” space the generator was aimed at was at least 70% already published, before any question of model accuracy arises. The constructive output is the protocol itself, a small quote-anchored experimental reference set with documented provenance, and a pre-registered calibration audit of our own DFT scale, whose decision rule was committed in advance and whose verdict is reported in Sec. VI . We claim no discovered material, no validated screen, and no calibrated scale. The paper’s contribution is the discipline: every quantitative claim below traces to a committed artifact, every failure is reported under the definition fixed before the analysis ran, and the strongest objections we know of are stated and answered in Sec. VII . II System under test The pipeline, called QME, is a solo-operated closed loop that runs on a single Apple M4 desktop. One cycle proceeds: generate candidate crystals, screen them with a property-predicting graph neural network (GNN), rank survivors with a Gaussian-process acquisition function over learned embeddings, screen the ranked list for prior art, verify the top picks with paired DFT calculations, and fine-tune the GNN on the newly verified results. Each stage is described here at the level needed to interpret the validation; computational details are in Sec. VIII . Candidates come from prototype substitution: redox-site substitution into known framework structures retrieved from the Materials Project (MP) 19 , followed by charge-neutrality and electronegativity screening with SMACT 12 . The screen is a Siamese multi-head GNN that ingests a (charged, discharged) structure pair and predicts the average intercalation voltage with a Monte-Carlo-dropout uncertainty. Its training corpus contains 2814 graphs, of which 20 pairs (0.7%) are Na chemistries; the model is, by construction, a Li-dominated predictor asked to extrapolate. A second network, used for stability estimates and for the embedding space, is not ours: it is the universal interatomic-potential backbone MACE-MP-0 6 ; 5 with property heads, and we treat it throughout as borrowed machinery. The acquisition function is a plain upper-confidence-bound rule over a PCA reduction of those embeddings; a diversity-weighted variant exists in the codebase but was never wired into the orchestrator, and with five verified training points the acquisition stage has no demonstrated value of any kind. We make no claim for it. The prior-art layer is the part of the stack built to say no . A candidate is flagged when a literature, structure-database, or family-pattern source names its composition: citations count only if a chemical formula in the retrieved record matches the candidate’s composition exactly, as a de-intercalated framework, or as the same element set, which removes the loose-token false positives that plague relevance-ranked search. The layer is safety-locked: it can downgrade a candidate to known but can never certify novelty, and its machine verdict for absence is bounded by the coverage of the sources it actually searched. In its committed precision audit it flagged 8 of 8 known compounds and false-flagged 0 of 5 fictional control compositions. Verification is paired-state DFT: both