INCARBench: A Benchmark for Scientific Configuration in VASP INCAR by Large Language Models
Bin Shao 1,3,4,∗ , Jixiang Li 2 , Xinyue Zhang 1 , Baishun Yang 4 , Zhiyang Liu 1,3,4,∗ , Weichao Wang 1,4,6,7,∗ 1 College of Electronic Information and Optical Engineering, Nankai University, Tianjin 300350, China 2 School of Physics, Nankai University, Tianjin 300071, China 3 Tianjin Key Laboratory of Optoelectronic Sensor and Sensing Network Technology, Nankai University, Tianjin 300350, China 4 Shenzhen Research Institute of Nankai University, Shenzhen 518057, China 5 CIC nanoGUNE BRTA, Tolosa Hiribidea 76, 20018 San Sebastián, Spain 6 College of Energy and Environment Science, Yunnan Normal University, Kunming 650500, China 7 Southwest United Graduate School, Kunming 650092, China ∗ Corresponding authors: [email protected]; [email protected]; [email protected] · 2026
Read INCARBench: A Benchmark for Scientific Configuration in VASP INCAR by Large Language Models by Bin Shao 1,3,4,∗ , Jixiang Li 2 , Xinyue Zhang 1 , Baishun Yang 4 , Zhiyang Liu 1,3,4,∗ , Weichao Wang 1,4,6,7,∗ 1 College of Electronic Information and Optical Engineering, Nankai University, Tianjin 300350, China 2 School of Physics, Nankai University, Tianjin 300071, China 3 Tianjin Key Laboratory of Optoelectronic Sensor and Sensing Network Technology, Nankai University, Tianjin 300350, China 4 Shenzhen Research Institute of Nankai University, Shenzhen 518057, China 5 CIC nanoGUNE BRTA, Tolosa Hiribidea 76, 20018 San Sebastián, Spain 6 College of Energy and Environment Science, Yunnan Normal University, Kunming 650500, China 7 Southwest United Graduate School, Kunming 650092, China ∗ Corresponding authors: [email protected]; [email protected]; [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-23571.txt.
Introduction
Shao and colleagues benchmark whether large language models can correctly configure scientific simulations, in a 2026 paper on computational materials science. The test concerns VASP, a widely used package for first-principles calculations, meaning simulations built directly from basic physics. Its INCAR file sets the physical model and numerical choices for each run. A file can execute fine yet describe the wrong science, so surface correctness is not enough.
The benchmark, INCARBench, covers both writing configurations from scratch and repairing broken ones, across nineteen model configurations. Several frontier models score high on semantic and policy accuracy, yet task-critical correctness stays much lower. Errors cluster where settings couple together, such as magnetism, correlated materials, and methods that add an on-site interaction term. Repair tests also show that fixing wrong settings and leaving valid ones untouched are separate skills, and preservation remains a major weakness.
Catalog record
| CREATOR | Bin Shao 1,3,4,∗ , Jixiang Li 2 , Xinyue Zhang 1 , Baishun Yang 4 , Zhiyang Liu 1,3,4,∗ , Weichao Wang 1,4,6,7,∗ 1 College of Electronic Information and Optical Engineering, Nankai University, Tianjin 300350, China 2 School of Physics, Nankai University, Tianjin 300071, China 3 Tianjin Key Laboratory of Optoelectronic Sensor and Sensing Network Technology, Nankai University, Tianjin 300350, China 4 Shenzhen Research Institute of Nankai University, Shenzhen 518057, China 5 CIC nanoGUNE BRTA, Tolosa Hiribidea 76, 20018 San Sebastián, Spain 6 College of Energy and Environment Science, Yunnan Normal University, Kunming 650500, China 7 Southwest United Graduate School, Kunming 650092, China ∗ Corresponding authors: [email protected]; [email protected]; [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.23571
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
INCARBench: A Benchmark for Scientific Configuration in VASP INCAR by Large Language Models Bin Shao 1,3,4,∗ , Jixiang Li 2 , Xinyue Zhang 1 , Baishun Yang 4 , Zhiyang Liu 1,3,4,∗ , Weichao Wang 1,4,6,7,∗ 1 College of Electronic Information and Optical Engineering, Nankai University, Tianjin 300350, China 2 School of Physics, Nankai University, Tianjin 300071, China 3 Tianjin Key Laboratory of Optoelectronic Sensor and Sensing Network Technology, Nankai University, Tianjin 300350, China 4 Shenzhen Research Institute of Nankai University, Shenzhen 518057, China 5 CIC nanoGUNE BRTA, Tolosa Hiribidea 76, 20018 San Sebastián, Spain 6 College of Energy and Environment Science, Yunnan Normal University, Kunming 650500, China 7 Southwest United Graduate School, Kunming 650092, China ∗ Corresponding authors: [email protected]; [email protected]; [email protected] Abstract Large language models (LLMs) are increasingly being integrated into first-principles computational workflows, yet their ability to configure scientific calculations remains poorly understood. Here, we introduce INCARBench, a benchmark for evaluating LLMs on input configuration for the Vienna Ab initio Simulation Package (VASP) through both configuration generation and repair tasks. Evaluating 19 model configurations reveals substantial capability differences among current frontier models. While several models achieve high semantic and policy accuracy, task-critical correctness remains substantially lower, demonstrating that parameter-level correctness does not necessarily imply scientifically valid configurations. Failure analysis shows that errors concentrate in physically coupled settings involving DFT+ U U , magnetism, and correlated materials, where multiple constraints must be satisfied simultaneously. Repair evaluation further reveals that correcting incorrect settings and preserving already-valid configurations are distinct capabilities, with configuration preservation remaining a major challenge. These findings establish scientific configuration as a measurable capability of large language models and provide a foundation for developing more reliable AI systems for computational materials science. Keywords: large language models; density functional theory; VASP; INCAR configuration; first-principles workflows 1 Introduction Large language models (LLMs) are increasingly being integrated into first-principles computational workflows, assisting with workflow planning, input preparation, execution management, and result interpretation [ 22 , 4 , 7 , 31 , 38 , 15 , 5 ] . A critical step in these workflows is input configuration: the translation of scientific intent into a computational setup. In the Vienna Ab initio Simulation Package (VASP) [ 19 , 20 , 21 ] , this role is performed by the INCAR file [ 37 ] , which specifies the physical model, numerical settings, and workflow behavior of a calculation. Different tasks and physical effects, including geometry optimization, band-structure analysis, magnetism [ 16 ] , DFT+ U [ 13 , 23 ] , and van der Waals interactions [ 14 , 12 ] , require distinct and often coupled parameter settings. Consequently, an executable INCAR file does not necessarily correspond to a scientifically correct calculation (Figure 1 ). Despite rapid progress in reasoning and scientific applications [ 29 , 8 , 1 , 11 , 35 , 26 , 34 , 33 ] , the ability of LLMs to configure first-principles calculations remains largely unexplored. Existing evaluations primarily focus on scientific knowledge, reasoning, property prediction, workflow execution, atomistic modeling capability, or downstream task performance [ 39 , 6 , 32 , 24 , 30 , 31 ] . They rarely evaluate whether a model can translate scientific intent into a computational configuration that is physically meaningful and workflow consistent. Moreover, practical workflows frequently require modifying existing configurations rather than generating them from scratch, making configuration repair an equally important capability [ 15 , 5 ] . To address this gap, we introduce INCARBench, a benchmark for evaluating LLMs on VASP INCAR generation and repair. The benchmark assesses whether models can translate scientific intent into computational configurations and modify existing configurations while preserving valid settings. Configuration quality is evaluated through three complementary dimensions: Semantic Correctness , Policy Correctness , and Task-Critical Correctness (TCC), which respectively assess physical intent encoding, numerical-policy selection, and satisfaction of workflow-defining and physics-defining requirements. Figure 1: Scientific intent encoding in first-principles calculations. Scientific objectives are translated into semantic, policy, and task-critical requirements that jointly determine a VASP INCAR configuration. An executable INCAR file is not necessarily scientifically correct, motivating the evaluation of scientific configuration quality beyond syntax validity and parameter matching. 2 Benchmark Design 2.1 Benchmark Dataset Figure 2 summarizes the design of INCARBench. The generation task contains 48 materials drawn from 16 representative material families and covers a broad range of scientific scenarios commonly encountered in first-principles calculations, including metals, semiconductors, insulators, oxides, correlated materials, layered materials, magnetic materials, and battery materials. The 16 material-family types are listed in the Supplementary Material, Section S1, and the complete benchmark data are available at https://github.com/liuzhiyangnku/incarbench . Each material is paired with four representative VASP workflow types: static self-consistent field (SCF) calculation, geometry relaxation, band-structure non-self-consistent field (NSCF) calculation, and density-of-states (DOS) NSCF calculation. Applying all four workflow types to all 48 materials yields 192 generation cases. The repair task is derived directly from the generation task. Each generation case produces three repair variants: a correct-to-preserve variant, a workflow-mistake variant, and a physics-mistake variant. The correct-to-preserve variants contain no injected errors and evaluate a model’s ability to avoid over-editing valid configurations, a common failure mode observed in configuration repair. The workflow-mistake variants introduce errors in workflow-defining settings, such as NSCF-related controls and charge-density handling. The physics-mistake variants introduce errors in physics-defining settings, including magnetism, DFT+ U , van der Waals (vdW) corrections, smearing strategies, and symmetry-sensitive configurations. This results in 576 repair cases in total. Figure 2: Construction of the INCARBench benchmark. The generation task contains 192 cases constructed from 48 materials and four representative workflow types. The repair task is derived directly from the generation cases. Each generation case produces three repair types, resulting in 576 repair cases in total. Table 1 summarizes the benchmark statistics. To support failure localization, each case is assigned one or more physics-aware challenge types. These challenge categories identify scientific settings that require coordinated configuration choices, including NSCF workflows, DFT+ U , magnetism, van der Waals (vdW) corrections, smearing strategies, and symmetry-sensitive settings. They are used throughout the evaluation to analyze model performance across different scientific regimes and to localize failure modes. Table 1: INCARBench dataset summary. Item Value Materials 48 materials from 16 material families Workflow types Static SCF, Geometry Relaxation, Band-Structure NSCF, DOS NSCF Generation cases 192 Repair cases 576 Repair variants Correct-to-Preserve, Workflow-Mistake, Physics-Mistake Challenge types NSCF Workflow, DFT+U, Magnetism, vdW, Smearing, Symmetry 2.2 Evaluation Framework For each benchmark case, the reference INCAR is derived from Materials Project VASP calculation records [ 18 , 17 , 28 ] and normalized for benchmark evaluation. Because multiple INCAR configurations may correspond to scientifically reasonable calculations, the reference is treated as a standardized evaluation target rather than a unique ground truth. Detailed benchmark construction, normalization rules, parameter classifications, and benchmark definitions are provided in the Supplementary Material, Sections S1–S3. Generated INCAR files are compared against the normalized reference configuration. INCARBench evaluates configuration quality through three complementary dimensions: semantic correctness, policy correctness, and TCC. Semantic correctness measures whether the intended physical meaning of a calculation is correctly encoded. Policy correctness evaluates whether numerical and methodological choices are appropriately configured. TCC evaluates whether all task-critical requirements of a calculation are simultaneously satisfied. Together, these dimensions distinguish parameter-level correctness from