Proof-Carrying Materials: Falsifiable Safety Certificates for Machine-Learned Interatomic Potentials
Abhinaba Basu, Pavan Chakraborty · 2026
Read Proof-Carrying Materials: Falsifiable Safety Certificates for Machine-Learned Interatomic Potentials by Abhinaba Basu, Pavan Chakraborty (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-12183.txt.
Introduction
Basu and Chakraborty wrap machine-learned atom models in safety certificates that tests can falsify. Used alone as a stability filter, one such model missed most DFT-stable materials in their benchmark. Their Proof-Carrying Materials pipeline attacks the gap in three stages. Adversarial search hunts failures across many materials, then envelope refinement adds confidence bounds. Lean 4 formal certification closes the loop for cleared cases. Audits of CHGNet, TensorNet, and MACE exposed blind spots unique to each model, and a risk model learned to predict such failures.
Catalog record
| CREATOR | Abhinaba Basu, Pavan Chakraborty |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62574 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.12183
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Proof-Carrying Materials: Falsifiable Safety Certificates for Machine-Learned Interatomic Potentials Abhinaba Basu, Pavan Chakraborty Abstract Machine-learned interatomic potentials (MLIPs) are deployed for high-throughput materials screening without formal reliability guarantees. We show that a single MLIP used as a stability filter misses 93% of density functional theory (DFT)-stable materials (recall 0.07) on a 25,000-material benchmark. Proof-Carrying Materials (PCM) closes this gap through three stages: adversarial falsification across compositional space, bootstrap envelope refinement with 95% confidence intervals, and Lean 4 formal certification. Auditing CHGNet, TensorNet and MACE reveals architecture-specific blind spots with near-zero pairwise error correlations ( r ≤ 0.13 r\leq 0.13 ; n = 5,000 n=5{,}000 ), confirmed by independent Quantum ESPRESSO validation (20/20 converged; median DFT/CHGNet force ratio 12 × 12\times ). A risk model trained on PCM-discovered features predicts failures on unseen materials (area under the ROC curve [AUC-ROC] = 0.938 ± 0.004 =0.938\pm 0.004 ) and transfers across architectures (cross-MLIP AUC-ROC ≈ 0.70 \approx 0.70 ; feature importance r = 0.877 r=0.877 ). In a thermoelectric screening case study, PCM-audited protocols discover 62 additional stable materials missed by single-MLIP screening—a 25% improvement in discovery yield. Keywords: machine-learned interatomic potentials, adversarial testing, formal verification, Lean 4, materials discovery, uncertainty quantification 1 Introduction Universal MLIPs—CHGNet 1 , MACE 4 , TensorNet 27 , ALIGNN 35 , SevenNet 36 , EquiformerV2 37 —underpin high-throughput materials screening 38 ; 55 , yet they are deployed without formal reliability guarantees 59 ; 60 . Matbench Discovery 2 , MLIP Arena 57 , CHIPS-FF 58 and earlier benchmarks 10 evaluate 45+ models on 257K WBM (Wang–Botti–Marques 3 ) structures, but aggregate accuracy metrics cannot answer the deployment-critical question: on which chemistries is this MLIP unreliable? This creates a quantifiable safety gap. A single-MLIP stability screen achieves precision 0.47 and recall 0.07 across 25,000 WBM materials, missing 93.0% of DFT-stable candidates. The gap is not merely statistical: CHGNet rejects TlBiSe 2 (a topological insulator with over 1,000 citations 26 ) and Cs 2 KTlBr 6 (a lead-free perovskite solar cell candidate with 1.27 eV band gap)—precisely the materials that high-throughput pipelines aim to find. Aggregate benchmarks showing “near-DFT accuracy” obscure these consequential, chemistry-specific blind spots. We reframe MLIP reliability as a falsifiable safety claim in the sense of Dalrymple et al.’s Guaranteed Safe AI 13 : a world model (the MLIP), a safety specification (bounds on acceptable error), and a verifier (machine-checked proofs) 45 . This mirrors proof-carrying code 15 , where untrusted programs ship with proofs of safety. Proof-Carrying Materials (PCM) instantiates this in three stages (Fig. 1 ): 1. Adversarial falsification —automated adversaries (six strategies: random, heuristic, grid, LHS, Sobol, LLM) probe for failure regions in compositional space; 2. Envelope refinement —counterexamples tighten the safety claim into bounds with bootstrap 95% confidence intervals (CIs); 3. Formal certification —the refined envelope compiles into Lean 4 proofs with explicit physical axioms. The framework is oracle-agnostic: swapping the MLIP requires no code changes. Our central findings are threefold. First, MLIP blind spots are architecture-specific : three MLIPs produce near-zero pairwise force correlations ( r=0.13 , 0.10 , -0.01 ; n=5{,}000 ) and fail on largely disjoint chemistries. Second, perturbation-based uncertainty quantification does not predict these compositional failures ( r=0.039 , p=0.58 ), meaning structural UQ and adversarial compositional auditing capture independent failure dimensions. Third, and most importantly, PCM enables prospective validation : compositional features discovered through adversarial auditing predict failures on unseen materials with AUC-ROC 0.938\pm 0.004 and P@20% = 1.000 (perfect precision among the top-risk quintile), transforming retrospective audit into predictive intervention. These failure patterns transfer across architectures: a CHGNet-trained risk model predicts MACE failures (AUC-ROC = 0.697 average cross-transfer) with feature importance correlation r=0.877 . Together, these results establish that (a) no single benchmark score captures reliability, (b) multi-MLIP auditing is essential for deployment, (c) formal verification combined with prospective prediction constitutes a new paradigm for MLIP validation, and (d) the full audit costs under $20. Figure 1 : The PCM pipeline. Stage 1 : automated adversaries (six strategies including LLMs) propose compositional feature vectors; the MLIP oracle evaluates each against the DFT reference. Stage 2 : counterexamples refine the safety envelope with bootstrap CIs. Stage 3 : the envelope compiles into Lean 4 proofs with explicit axioms. 2 Related Work Adversarial MLIP testing. CAGO 11 generates adversarial structures via uncertainty-calibrated geometry optimization on individual materials. PCM operates in compositional space —searching for chemical families where MLIPs systematically diverge from DFT. These are complementary: CAGO identifies which atomic arrangements are unreliable; PCM identifies which chemistries are unreliable regardless of arrangement. In our evaluation, perturbation-based uncertainty is not predictive of compositional failure ( r=0.039 , p=0.58 ). Uncertainty quantification. Conformal prediction 19 provides distribution-free marginal coverage guarantees applied to interatomic potentials 20 . Monte Carlo dropout 29 , deep ensembles 30 and chemical-space uncertainty estimation 31 ; 32 , committee disagreement 63 and misspecification-aware regression 61 ; 62 have been applied to molecular property prediction. Critically, these methods provide marginal coverage (“95% of all materials fall within the prediction interval”) but cannot answer the deployment question which specific compositions are unreliable and why . Conformal prediction intervals widen uniformly across compositional space; they do not identify the compositional drivers (e.g. heavy elements, large unit cells) that concentrate failures, nor do they enable targeted DFT allocation for high-risk chemistries. PCM concentrates adversarial sampling on failure boundaries, producing tighter envelopes where they matter most: in a split conformal baseline on the same 5,000 WBM materials (2,500 calibration, 2,500 test), the feature-space conformal envelope at \alpha=0.05 contains 1,924 materials (precision 6.8%) versus PCM’s adversary-informed bootstrap envelope with 97 materials (precision 7.2%)—a 20\times reduction in envelope size at comparable precision. Active learning 12 ; 50 ; 51 selects informative points for retraining ; PCM selects adversarial points for auditing —the adversarial budget (200 queries) is orders of magnitude smaller than retraining data requirements. Safety cases and formal verification. AMLAS 16 provides structured safety-case methodology for ML components but relies on human argumentation. Seshia et al. 17 identify the specification gap as a central challenge for verified AI. Neural network verification 43 ; 44 focuses on input–output properties of classifiers; PCM verifies domain-specific reliability claims about regression models. Adversarial testing in ML 33 ; 34 targets input perturbations; falsification of cyber-physical systems 7 ; 8 provides compositional testing frameworks. Komendantskaya 64 extends proof-carrying code to neuro-symbolic programs, and Kamran et al. 65 apply it to LLM-generated code completions. PCM instantiates the Guaranteed Safe AI framework 13 for materials: the MLIP is the world model, the envelope is the specification, and Lean 4 proofs are the verifier output—providing machine-checkable evidence that structured safety cases currently lack. 3 Results 3.1 Architecture-specific failure profiles We evaluate three architecturally distinct MLIPs—CHGNet v0.3.0 1 , TensorNet (MatPES-PBE-v2025.1) 27 ; 28 , and MACE-MP medium 4 ; 54 —on 5,000 WBM materials. Structures are synthetic compositional probes—atoms placed at irrational (golden-ratio) fractional coordinates to avoid symmetry artifacts, with DFT-derived cell volumes (Methods)—designed to test whether each MLIP has learned the correct energy landscape for a given chemistry. The blind-spot classification itself—which materials are DFT-stable but MLIP-unstable—derives from the WBM benchmark, where both DFT and MLIPs evaluate fully relaxed equilibrium structures. Force failure threshold: 50 eV Å -1