Comparative study of ensemble-based uncertainty quantification methods for neural network interatomic potentials

Yonatan Kurniawan, Mingjian Wen, Ellad B. Tadmor, Mark K. Transtrum · 2025

Read Comparative study of ensemble-based uncertainty quantification methods for neural network interatomic potentials by Yonatan Kurniawan, Mingjian Wen, Ellad B. Tadmor, Mark K. Transtrum (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-06456.txt.

Introduction

Yonatan Kurniawan and co-authors compare in 2025 how well ensemble methods state their uncertainty in machine learning models of atomic forces. Such models let scientists simulate materials at near first principles accuracy for a fraction of the cost. Yet a model's own error bars are often used as a stand in for true accuracy, above all on new systems where direct checks are too costly.

The team tested bootstrap, dropout, random start, and snapshot ensembles on carbon forms of many shapes. In familiar territory they used held out data; outside it, they checked full energy curves and vibration patterns. Uncertainty bars behaved in surprising ways on new ground, at times flattening or moving against real error. The message is a caution: a confident model is not always a right one, and system level tests are needed before trusting these tools.

Catalog record

CREATORYonatan Kurniawan, Mingjian Wen, Ellad B. Tadmor, Mark K. Transtrum
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-06456

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Comparative study of ensemble-based uncertainty quantification methods for neural network interatomic potentials Yonatan Kurniawan, Mingjian Wen, Ellad B. Tadmor, Mark K. Transtrum Abstract Machine learning interatomic potentials (MLIPs) enable atomistic simulations with near first-principles accuracy at substantially reduced computational cost, making them powerful tools for large-scale materials modeling. The accuracy of MLIPs is typically validated on a held-out dataset of ab initio energies and atomic forces. However, accuracy on these small-scale properties does not guarantee reliability for emergent, system-level behavior—precisely the regime where atomistic simulations are most needed, but for which direct validation is often computationally prohibitive. As a practical heuristic, predictive precision—quantified as inverse uncertainty—is commonly used as a proxy for accuracy, but its reliability remains poorly understood, particularly for system-level predictions. In this work, we systematically assess the relationship between predictive precision and accuracy in both in-distribution (ID) and out-of-distribution (OOD) regimes, focusing on ensemble-based uncertainty quantification methods for neural network potentials, including bootstrap, dropout, random initialization, and snapshot ensembles. We use held-out cross-validation for ID assessment and calculate cold curve energies and phonon dispersion relations for OOD testing. These evaluations are performed across various carbon allotropes as representative test systems. We find that uncertainty estimates can behave counterintuitively in OOD settings, often plateauing or even decreasing as predictive errors grow. These results highlight fundamental limitations of current uncertainty quantification approaches and underscore the need for caution when using predictive precision as a stand-in for accuracy in large-scale, extrapolative applications. I Introduction Machine learning interatomic potentials (MLIPs) have emerged as powerful tools in computational materials science, offering a promising alternative to traditional approaches for simulating atomic-scale systems. First-principles methods, such as density functional theory (DFT), provide high-fidelity predictions of material properties but are computationally prohibitive for large systems or long simulation times. Classical empirical potentials, while computationally efficient, often lack the flexibility and transferability needed to accurately model diverse materials behavior. MLIPs bridge this gap by leveraging machine learning algorithms to learn the underlying potential energy surface (PES) directly from quantum-mechanical reference data. These models enable energy and force evaluations that are orders of magnitude faster than first-principles methods, while maintaining comparable levels of accuracy behler_generalized_2007_edited ; thompson_spectral_2015 ; shapeev_moment_2016 ; lindsey_chimes_2017 ; drautz_atomic_2019 ; wen_uncertainty_2020 ; bartok_machine_2018 ; batzner_e3-equivariant_2022 ; chen_universal_2022 ; deng_chgnet_2023 . Modern MLIPs are data-driven, black-box models designed to replicate the PES of atomic systems by learning directly from quantum-mechanical dataset. Training these models typically involves fitting to small-scale quantities, such as atomic forces, energies, and sometimes stresses, computed for a diverse set of atomic configurations ercolessi_interatomic_1994 . The training configurations are often obtained from molecular dynamics trajectories, random structure sampling, or perturbations around equilibrium geometries. Once trained, MLIPs are deployed to study macroscopic material properties of interest that emerge from simulations over much larger time and length scales, such as elastic properties pasianot_empirical_1991 ; allred_elastic_2004 ; tasnadi_efficient_2021 , thermal conductivity arabha_recent_2021 ; broido_lattice_2005 ; korotaev_accessing_2019 , and defect dynamics bertin_crystal_2023 ; page_atomistic_2024 ; serafin_grain_2025 . The performance of MLIPs is often assessed by evaluating their prediction accuracy, typically quantified using error metrics that compare model predictions against ground truth DFT values, for example. In this framework, lower error indicates higher accuracy. Initial assessments usually focus on small-scale property predictions—energies and forces—evaluated on in-distribution (ID) samples, which consist of atomic configurations similar to those seen during training (e.g., a held-out test set). However, high accuracy on ID samples does not necessarily imply high accuracy on out-of-distribution (OOD) samples, which are often represented by downstream, large-scale material property predictions tavazza_uncertainty_2021 . This discrepancy stems from the mismatch in simulation scales between training and downstream application. Large-scale property simulations frequently involve exploration into high-dimensional regions of configuration space that are poorly represented in the training set. Moreover, such properties may depend sensitively on specific features of the PES that are difficult to sample accurately, such as saddle points corresponding to transition states samanta_sampling_2014 ; asgeirsson_exploring_2020 and high-energy regions in high pressure simulations cammi_studying_2022 ; lindsey_chimes_2025 . Although this motivates the need to assess accuracy on both ID and OOD samples, generating such ground truth data—whether quantum-mechanical or experimental—is often prohibitively expensive or even infeasible. In addition to accuracy, it is also important to assess model precision through estimates of predictive uncertainty, where lower uncertainty indicates higher precision. Precision reflects the consistency of a model’s predictions under various sources of variability, such as those arising from the training data, model architecture, initialization, and optimization. Various uncertainty quantification (UQ) methods have been developed to capture these effects in MLIPs. For example, techniques like mean-variance estimation tan_single-model_2023 , Gaussian mixture models zhu_fast_2023 , and quantile regression bilbrey_uncertainty_2025 aim to estimate aleatoric uncertainty, i.e., irreducible uncertainty from inherent variability in the data. These methods learn such variability directly from the training data, including implicit sources of variability arising from choices of DFT exchange-correlation functionals and numerical tolerances henkel_uncertainty_2021 ; dai_uncertainty_2024 . In contrast, ensemble-based approaches li_uncertainty_2019 ; wen_uncertainty_2020 and Bayesian neural networks tran_methods_2020 target epistemic uncertainty, which arises from limited data coverage, model misspecification, and parameter uncertainty. Ensemble methods, in particular, are popular due to their simplicity, model-agnostic implementation, and practical effectiveness. Additionally, distance-based metrics vita_ltau-ff_2024 ; hu_robust_2022 and Gaussian process models are often employed to detect OOD configurations and estimate model confidence in previously unexplored regions of configuration space. The use of prediction precision as a proxy for accuracy has been suggested as a practical solution to the difficulty of evaluating accuracy, especially in the context of large-scale material property predictions frederiksen_bayesian_2004 . Although estimating prediction precision (i.e., uncertainty) can be computationally intensive—often requiring multiple model evaluations—it remains far less costly than quantum-mechanical or experimental validation. However, precision and accuracy are fundamentally distinct concepts and do not necessarily correlate (see Fig. 1 ). While one might expect accurate predictions to be accompanied by high precision (top-left panel) and inaccurate predictions by low precision (bottom-right panel), this relationship does not always hold. For example, a model may produce accurate material property predictions on average, yet still exhibit a large ensemble spread (top-right panel). Conversely, a prediction can appear highly precise, with low ensemble variance, yet deviate significantly from the true value, resulting in overconfident predictions (bottom-left panel). Recognizing this potential disconnect is crucial for evaluating when uncertainty estimates can be interpreted meaningfully and when they might give a false sense of confidence or caution. Figure 1 : Illustration of precision and accuracy using a dartboard metaphor. Each panel shows a different combination of prediction accuracy (closeness to the true value, indicated by the bullseye) and precision (spread of predictions). This illustration emphasizes that high precision does not necessarily imply high accuracy. In this paper, we investigate when prediction precision can serve as a reliable surrogate for prediction accuracy, particularly in the OOD domain. We conduct a comparative study of several ensemble-based UQ methods for MLIPs, highlighting key considerations when using precision as a proxy for accuracy. To this end, our study involves developing MLIPs, generating multiple MLIP ensembles, and applying them within widely used atomistic simulation software to compute large-scale material properties. These tasks are streamlined by the infrastructure within the Open Knowledgebase of Interatomic Models (OpenKIM) project, which enables seamless integration of MLIPs into simulation workflows Tadmor_Elliott_Sethna_Miller_Becker_2011 ; elliott:tadmor:2011

Read the complete text