Text to Band Gap: Pre-trained Language Models as Encoders for Semiconductor Band Gap Prediction

Ying-Ting Yeh, Janghoon Ock, Amir Barati Farimani · 2025

Read Text to Band Gap: Pre-trained Language Models as Encoders for Semiconductor Band Gap Prediction by Ying-Ting Yeh, Janghoon Ock, Amir Barati Farimani (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-03456.txt.

Introduction

Ying-Ting Yeh and co-authors feed word tales of stone goods to tongue models and read off the spark gap of each, in a paper from 2025. The gap sets how a chip stone drinks light, a hinge trait for tools. Past reads need hand built traits or atom maps, both slow to make.

Word tales in fixed form and in free prose both serve as feed. Four tongue models, bent with a small read head, call the gap with mean miss near a quarter step. The best is a vast plain model, but a small trade tuned one ties it at a tenth of its bulk. Their gaze rests on mix and spin marks, not shape marks, which flags a text blind spot.

Catalog record

CREATORYing-Ting Yeh, Janghoon Ock, Amir Barati Farimani
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-03456

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Text to Band Gap: Pre-trained Language Models as Encoders for Semiconductor Band Gap Prediction Ying-Ting Yeh, Janghoon Ock, Amir Barati Farimani Abstract In this study, we explore the use of a transformer-based language model as an encoder to predict the band gaps of semiconductor materials directly from their text descriptions. Quantum chemistry simulations, including Density Functional Theory (DFT), are computationally intensive and time-consuming, which limits their practicality for high-throughput material screening, particularly for complex systems. Shallow machine learning (ML) models, while effective, often require extensive data preprocessing to convert non-numerical material properties into numerical inputs. In contrast, our approach leverages textual data directly, bypassing the need for complex feature engineering. We generate material descriptions in two formats: formatted strings combining features and natural language text generated using the ChatGPT API. We demonstrate that the RoBERTa model, pre-trained on natural language processing tasks, performs effectively as an encoder for prediction tasks. With minimal fine-tuning, it achieves a mean absolute error (MAE) of approximately 0.33 eV, performing better than shallow machine learning models such as Support Vector Regression, Random Forest, and XGBoost. Even when only the linear regression head is trained while keeping the RoBERTa encoder layers frozen, the accuracy remains nearly identical to that of the fully trained model. This demonstrates that the pre-trained RoBERTa encoder is highly adaptable for processing domain-specific text related to material properties, such as the band gap, significantly reducing the need for extensive retraining. This study highlights the potential of transformer-based language models to serve as efficient and versatile encoders for semiconductor materials property prediction tasks. keywords American Chemical Society, L a T e X † † affiliation: Department of Chemical Engineering, Carnegie Mellon University, 5000 Forbes Street, Pittsburgh, PA 15213, USA † † email: [email protected] † † affiliation: Department of Mechanical Engineering, Carnegie Mellon University, 5000 Forbes Street, Pittsburgh, PA 15213, USA † † abbreviations: IR,NMR,UV 1 Introduction The band gap of semiconductor materials is a fundamental property that directly impacts their electronic and optical behaviors. This parameter dictates crucial attributes such as conductivity, light absorption, and emission, making it essential for the performance of various electronic, optoelectronic, and photovoltaic devices. Therefore, the precise prediction and control of the band gap are vital for optimizing semiconductor applications in these fields Kim et al. 2020 . Band gaps are determined primarily through experimental methods, with UV-visible absorption spectroscopy and photoluminescence spectroscopy being the most commonly used techniques Masood et al. 2023 . However, these experimental methods can only measure the band gaps of synthesized materials and are not applicable to new materials designed theoretically. On the computational side, Density Functional Theory (DFT) has been the primary tool for studying the electronic structure of materials Koch and Holthausen 2015 ; Kohn and Sham 1965 . While DFT can accurately simulate electronic properties such as band structures and band gaps, its high computational cost and resource-intensive nature make it less practical for high-throughput material screening, especially for complex systems Schleder et al. 2019 . Machine learning (ML) methods have become powerful tools for addressing the computational challenges of Density Functional Theory (DFT). Shallow ML models, such as Random Forest (RF) Wang et al. 2021 and Support Vector Regression (SVR) Rajan et al. 2018 ; Zhuo et al. 2018 , are commonly used to predict band gaps efficiently by significantly reducing computational costs compared to solving complex quantum mechanics equations. However, these models often struggle with non-numerical features, requiring extensive preprocessing to convert material properties into numerical formats. This not only increases the complexity of feature engineering but also risks losing critical, hard-to-quantify information. Recent advances in deep learning, particularly Graph Neural Networks (GNNs) Taniguchi et al. 2023 , have improved the ability to capture the interconnectivity of atomic structures by representing them as graphs. However, this approach requires the conversion of atomic arrangements into graph representations, which adds another layer of preprocessing. Furthermore, GNNs still face limitations in integrating non-numerical properties, such as compound names, into the training process without additional preprocessing steps. These challenges underscore the need for approaches that can seamlessly handle both numerical and non-numerical features in material property predictions while minimizing complex preprocessing requirements. Language models offer unique advantages by directly utilizing human-readable text data, eliminating the need for elaborate pre-processing while preserving critical information embedded in material descriptions Ock et al. 2023 ; Ock et al. 2024 . This streamlines the prediction process. Recent advances in natural language processing, particularly with large language models (LLMs), have introduced transformative possibilities for materials science. This simplifies the prediction process compared to conventional ML approaches, which rely on precise atomic coordinates or complex data preprocessing to obtain numerical features, LLMs can directly use text-based descriptions to predict material properties. This bypasses the need for detailed structural data and removes the necessity for elaborate feature engineering. Recent advances in natural language processing, particularly with LLMs, have introduced transformative opportunities for materials science. For instance, AlloyBERT demonstrates the potential of transformer-based models to predict material properties from descriptive text Chaudhari et al. 2024 . Similarly, AMGPT showcases the benefits of using composition-based input strings and fine-tuned LLMs, enabling accurate and efficient predictions for materials science tasks Jacobs et al. 2024 ; Chandrasekhar et al. 2024 . Additionally, CatBERTa, a RoBERTa-based predictive model, has been developed to predict adsorption energy in catalyst systems Ock et al. 2023 ; Ock et al. 2024 . In this study, we leverage a transformer-based language model, RoBERTa (Robustly Optimized BERT Pre-training Approach), to predict the band gaps of semiconductor materials. This approach allows us to directly process text-based descriptions of material properties, such as chemical composition and crystal structures, without requiring extensive feature engineering. Our approach can handle both numerical and non-numerical features flexibly without complex data preprocessing, enhancing prediction accuracy and simplifying the input process. The inherent attention mechanisms of the transformer architecture also provide interpretability by identifying key material descriptors that contribute most to band gap predictions. By harnessing the power of language models, our approach captures subtle relationships and patterns within materials data, offering a robust framework for accurate and insightful band gap predictions. 2 Methods 2.1 Pre-trained Model RoBERTa is an enhanced model based on Bidirectional encoder representations from transformers (BERT), focused on improving language pre-processing performance Liu 2019 . Widely applied across various natural language processing (NLP) tasks, RoBERTa is known for its strong ability to capture contextual relationships between tokens in text. We hypothesized that this capability would also be effective in identifying relationships between material features. Compared to BERT, RoBERTa omits the next-sentence prediction task, focusing solely on the masked language modeling task, and introduces multiple full sentences in each input during training to help the model understand long-range dependencies. RoBERTa also employs a byte-level Byte Pair Encoding tokenizer, which segments text into subword units at the byte level, allowing it to effectively handle rare and Unicode characters. These design choices, combined with dynamic masking, larger training batches, and improved sentence packing, enable RoBERTa to produce more representative embeddings, capturing both intra- and inter-sentence context Liu 2019 . 2.2 Fine-tuning Model Our model is based on the RoBERTa-base architecture, which comprises 12 transformer layers, each with 12 attention heads and a hidden size of 768. We

Read the complete text