Expert-Grounded Automatic Prompt Engineering for Extracting Lattice Constants of High-Entropy Alloys from Scientific Publications using Large Language Models

Shunshun Liu, Talon R. Booth, Yangfeng Ji, Wesley Reinhart, Prasanna V. Balachandran · 2025

Read Expert-Grounded Automatic Prompt Engineering for Extracting Lattice Constants of High-Entropy Alloys from Scientific Publications using Large Language Models by Shunshun Liu, Talon R. Booth, Yangfeng Ji, Wesley Reinhart, Prasanna V. Balachandran (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-22130.txt.

Introduction

Shunshun Liu and four co-authors show how expert feedback can teach a language model to mine numbers from research papers reliably. Their test task is pulling lattice constants, a basic crystal measurement, for high-entropy alloys out of published articles. Rather than hand-tuning prompts for months, they optimize them automatically through feedback cycles on just seven expert-annotated papers.

Recall for Claude 3.5 Sonnet climbed from 0.27 to above 0.9 under this small budget. The tuned prompt then extracted data covering 1,861 alloy compositions from 2,267 publications, and it transferred well to three newer models. The authors sort model mistakes into contextual hallucination, semantic misreading, and unit conversion errors, arguing that validation protocols remain essential.

Catalog record

CREATORShunshun Liu, Talon R. Booth, Yangfeng Ji, Wesley Reinhart, Prasanna V. Balachandran
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-22130

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Expert-Grounded Automatic Prompt Engineering for Extracting Lattice Constants of High-Entropy Alloys from Scientific Publications using Large Language Models Shunshun Liu, Talon R. Booth, Yangfeng Ji, Wesley Reinhart, Prasanna V. Balachandran Abstract Large language models (LLMs) have shown promise for scientific data extraction from publications, but rely on manual prompt refinement. We present an expert-grounded automatic prompt optimization framework that enhances LLM entity extraction reliability. Using high-entropy alloy lattice constant extraction as a testbed, we optimized prompts for Claude 3.5 Sonnet through feedback cycles on seven expert-annotated publications. Despite a modest optimization budget, recall improved from 0.27 to > > 0.9, demonstrating that a small, expert-curated dataset can yield significant improvements. The approach was applied to extract lattice constants from 2,267 publications, yielding data for 1,861 compositions. The optimized prompt transferred effectively to newer models: Claude 4.5 Sonnet, GPT-5, and Gemini 2.5 Flash. Analysis revealed three categories of LLM mistakes: contextual hallucination, semantic misinterpretation, and unit conversion errors, emphasizing the need for validation protocols. These results establish feedback-guided prompt optimization as a low-cost, transferable methodology for reliable scientific data extraction, providing a scalable pathway for complex LLM-assisted research tasks. keywords Large language models, prompt optimization, high-entropy alloys, lattice constants, scientific data extraction Introduction Materials informatics studies based on experimental data have historically relied on manual data extraction from scientific publications. This process is tedious and time-consuming. Consequently, scaling this approach to extract data from thousands of publications remains impractical. Over the past decade, researchers have focused on automating data extraction from unstructured sources, such as journal publications. Initially, natural language processing (NLP) techniques were developed that relied on expert annotation Kim et al. 2017 ; Venugopal et al. 2019 ; Olivetti et al. 2020 ; Shetty and Ramprasad 2021 ; Wang et al. 2022 , and more recently, there has been a shift toward leveraging large language models (LLMs) for this purpose Zheng et al. 2023 ; Gilligan et al. 2023 ; Sayeed et al. 2024 ; Dagdelen et al. 2024 ; Polak and Morgan 2024 ; Jiang et al. 2025 ; Foppiano et al. 2024 ; Liu et al. 2024 ; Ekuma 2025 ; Biswajeet and Kadkhodaei 2025 ; Liu et al. 2025 ; Turan and Sparks 2025 ; Polak and Morgan 2025 . Our survey of LLM applications in materials science literature (Figure S1 in the Supplementary Information, SI) identified automated knowledge and data extraction as the most prevalent use cases. These studies have demonstrated that LLM-based data extraction from scientific publications can achieve reasonable accuracy, but performance depends critically on prompt quality. A prompt is a structured set of instructions provided to an LLM that specifies the task, defines constraints, and outlines output format and content White et al. 2023 ; Schulhoff et al. 2024 . Effective prompts typically include four elements Zheng et al. 2023 ; Polak and Morgan 2024 ; White et al. 2023 ; Schulhoff et al. 2024 ; Giray 2023 : (1) Instructions that define the task and desired behavior, (2) Context that provides relevant background information, (3) Input Data to be processed, and (4) Output Indicators that specify the desired format and structure of the response. Our literature survey identified several recent publications that provide systematic approaches to prompt engineering for extracting materials science data from scientific publications. We discuss two representative studies in detail below. Zheng et al. Zheng et al. 2023 developed a human-in-the-loop prompt engineering strategy for chemistry-focused applications based on three core principles: ( i ) well-formulated prompts designed to minimize hallucination by avoiding queries that may produce fabricated content, ( ii ) detailed instructions that convey context and the desired response format, and ( iii ) structured output requirements using well-defined templates to facilitate extraction. Their workflow first identified relevant sections in publications (those containing metal-organic framework, MOF, synthesis conditions), confirmed data presence, and then extracted MOF synthesis parameters from those sections. The utilization of GPT-3.5 and GPT-4, coupled with an interactive prompt refinement strategy enhanced by human feedback, demonstrated significant efficacy in achieving robust performance metrics, which the authors discussed in terms of the precision, recall, and F1 score. Polak and Morgan Polak and Morgan 2024 developed a two-stage zero-shot prompt engineering strategy. Stage 1 employed a relevancy prompt to identify sentences containing target data, while Stage 2 used question-and-answer prompts to extract specific information. They demonstrated this approach by constructing two databases: critical cooling rates of metallic glasses and yield strengths of high-entropy alloys (HEAs). Using GPT-3.5 (gpt-3.5-turbo-0301), GPT-4 (gpt-4-0314), and LLaMA2-chat (70B), they also evaluated extraction accuracy through precision and recall metrics. Additional studies by Ha et al. Ha et al. 2025 (nanotoxicity), Zhang et al. Zhang et al. 2024 (magnetic materials), Liu et al. Liu et al. 2024 (metallic glasses), Gupta et al. Gupta et al. 2024 (polymers), and Prasad et al. Prasad et al. 2024 (titanium-based alloys) further demonstrated the value of carefully structured prompts for large-scale data extraction from scientific texts. Current prompt engineering methods, despite advancements, have a significant limitation. They require considerable human involvement for refining prompts and are typically optimized based on individual publications rather than across a collection of publications. To address this limitation, we introduce a prompt engineering strategy that integrates domain expert-constructed data into an automated optimization process, rather than relying on post-hoc, human-in-the-loop refinement. Our workflow is shown in Figure 1 . Building on the TextGrad framework Yuksekgonul et al. 2025 , which enables prompt optimization through backpropagation of LLM-generated “textual gradients”, our workflow leverages a small, expert-grounded dataset carefully curated from multiple scientific publications to automatically and iteratively refine prompts without human-in-the-loop. These optimized prompts can then be used for large-scale automated data extraction from thousands of scientific publications. We demonstrate this methodology using HEAs as a test case, which are an emerging class of materials with significant technological interest Miracle and Senkov 2017 ; George et al. 2020 ; Han et al. 2024 . In our prompts, we use the term “high entropy alloys”, which the LLM interprets as encompassing multi-principal element alloys (MPEAs) and compositionally complex alloys (CCAs), as these terms are used interchangeably in the literature. For consistency, we use HEAs throughout this manuscript, with the understanding that it broadly encompasses MPEAs and/or CCAs. Our objective is to develop an expert-grounded, automatic prompt optimization framework for extracting experimentally measured, room temperature lattice constant data from thousands of HEA publications using LLMs. Current HEA databases, such as the one by Borg et al. Borg et al. 2020 , which includes mechanical properties and phase information, and the database constructed by Polak and Morgan Polak and Morgan 2024 , which contains yield strength data along with annotated source passages, do not provide lattice constant data for experimentally synthesized alloys. It is essential to construct an HEA dataset that includes lattice constant data from the experimental HEA literature to advance alloy development efforts Tandoc et al. 2025 . They are important inputs for predicting solid-solution strengthening effects using mechanistic models Maresca and Curtin 2020 ; Liu et al. 2022 and help estimate the coherency strain relationships between the matrix and precipitate phases in precipitation-hardened systems Li et al. 2023 . However, the manual extraction of lattice constant data from the rapidly expanding HEA scientific

Read the complete text