Transformer-Enhanced Variational Autoencoder for Crystal Structure Prediction
Ziyi Chen, Yang Yuan, Siming Zheng, Jialong Guo, Sihan Liang, Yangang Wang, Zongguo Wang · 2025
Read Transformer-Enhanced Variational Autoencoder for Crystal Structure Prediction by Ziyi Chen, Yang Yuan, Siming Zheng, Jialong Guo, Sihan Liang, Yangang Wang, Zongguo Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-09423.txt.
Introduction
Ziyi Chen and co-authors present a generative model that predicts and creates crystal structures, in a research paper from 2025. A crystal's structure fixes much of a material's behavior, so forecasting stable structures speeds materials design. Generative models are a fresh route, but crystals demand care for their repeating patterns and symmetry.
Their model, TransVAE-CSP, pairs a variational autoencoder with a transformer that respects the geometry of the task. A distance expansion trick and symmetry-aware features help it learn what stable materials look like. It can both rebuild known structures and generate new ones. On three benchmark sets covering carbon, perovskites, and mixed materials, it beat rival methods on standard scores.
Catalog record
| CREATOR | Ziyi Chen, Yang Yuan, Siming Zheng, Jialong Guo, Sihan Liang, Yangang Wang, Zongguo Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-09423
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Transformer-Enhanced Variational Autoencoder for Crystal Structure Prediction Ziyi Chen, Yang Yuan, Siming Zheng, Jialong Guo, Sihan Liang, Yangang Wang, Zongguo Wang Abstract Crystal structure forms the foundation for understanding the physical and chemical properties of materials. Generative models have emerged as a new paradigm in crystal structure prediction(CSP), however, accurately capturing key characteristics of crystal structures, such as periodicity and symmetry, remains a significant challenge. In this paper, we propose a Transformer-Enhanced Variational Autoencoder for Crystal Structure Prediction (TransVAE-CSP), who learns the characteristic distribution space of stable materials, enabling both the reconstruction and generation of crystal structures. TransVAE-CSP integrates adaptive distance expansion with irreducible representation to effectively capture the periodicity and symmetry of crystal structures, and the encoder is a transformer network based on an equivariant dot product attention mechanism. Experimental results on the carbon_24, perov_5, and mp_20 datasets demonstrate that TransVAE-CSP outperforms existing methods in structure reconstruction and generation tasks under various modeling metrics, offering a powerful tool for crystal structure design and optimization. † † affiliationnotice: Equal contribution 1 Introduction Crystal structure prediction (CSP) methods play a crucial role in the discovery of novel materials and remain a persistent challenge in condensed matter physics and materials chemistry. With the advancement of first-principles calculations, such as Density Functional Theory (DFT) ( Hohenberg & Kohn 1964 ) and structure search algorithms, including simulated annealing ( Kirkpatrick et al. 1983 ) and genetic algorithms ( Wu et al. 2013 ) , the precision and efficiency of CSP have improved significantly. These methods typically involve generating candidate crystal structures and performing DFT calculations to screen them. Notable examples include USPEX ( Lyakhov et al. 2013 ) and AIRSS ( Pickard & Needs 2011 ) . However, these approaches are computationally intensive and time-consuming. Traditional computationally driven CSP methods often lack effective learning priors and are generally limited to structural searches within a single system. Furthermore, due to the inherent randomness of the search space, they require substantial computational resources and are susceptible to getting trapped in local optima. With the rapid advancement of deep learning technologies, CSP has progressed into a new phase of data-driven. Neural network models now substitute DFT calculations to predict material properties ( Merchant et al. 2023 ) , significantly enhancing the efficiency of structure searches. Generative models, including variational autoencoder ( Kingma 2013 ) , adversarial generative networks ( Goodfellow et al. 2014 ) , and diffusion models ( Ho et al. 2020 ) with prior knowledge, are increasingly used in crystal structure generation. These models are capable of learning the feature space distribution of sample crystal structures and directly sampling from this space to generate crystal structures that adhere to the learned distribution. This approach significantly improves search efficiency compared to traditional CSP methods. Figure 1: TransVAE-CSP, based on the Variational Autoencoder (VAE) paradigm, excels in both crystal structure reconstruction and ab initio generation tasks. Experimental results show exceptional performance across evaluation metrics. Compared to previous work ( Xie et al. 2021 ) , we have introduced innovations and conducted verifications in crystal structure representation and the encoder network. CSP methods based on generative models have progressively evolved from utilizing composition information to directly constructing complex crystal structures. Early works, such as MatGAN ( Dan et al. 2020 ) and Binded-VAE ( Oubari et al. 2012 ) , rely exclusively on crystal structure composition information for training, capturing implicit rules of chemical composition. However, these models serve primarily as reference models for structure generation and have a limited capacity to produce diverse structures. More advanced methods, including Composition-Conditioned Crystal GAN ( Kim et al. 2020 ) and CCDCGAN ( Long et al. 2021 ) , have made significant progress in feature representation, but they still constrain the crystal structure space and are limited to specific compound groups. In contrast, works for complex systems, such as CDVAE ( Xie et al. 2021 ) , DiffCSP ( Jiao et al. 2023 ) , UniMat ( Yang et al. 2024 ) , and MatterGen ( Zeni et al. 2025 ) , have achieved crystal structure generation without being restricted by representation space, aligning with the primary research goals in materials discovery. Therefore, optimizing existing model frameworks and exploring new approaches to continuously enhance both the efficiency and quality of structure generation represent key directions for advancing the application of artificial intelligence in materials science. The development of CSP methods highlights that the accuracy of algorithms and models for crystal structure characterization is a critical factor influencing the stability and precision of generated structures. This accuracy remains a significant challenge in generative models for CSP. In particular, capturing the periodicity, symmetry, and other fundamental characteristics of crystal structures continues to be a major obstacle in CSP methodologies. To address these challenges, we propose a Transformer-Enhanced Variational Autoencoder (TransVAE-CSP) for crystal structure prediction, as shown in Fig. 1 . This model facilitates the accurate reconstruction and generation of crystal structures. The primary contributions of this work are as follows: • We developed an adaptive distance expansion method to characterize crystal structures and introduced an innovative hybrid function method to optimize a single distance expansion function. Our verification showed that this hybrid strategy can yield optimal results under different crystal sample environments; • We proposed a Transformer network based on an equivariant dot product attention mechanism as encoder for Variational Autoencoder (VAE) model, enhancing its ability to learn the characteristics of crystal samples; • Our work was evaluated using the carbon_24, perov_5 and mp_20 datasets. The results demonstrated significant advantages in structure reconstruction and generation tasks under various modeling metrics. This research offers a powerful tool for crystal structure design and optimization. 2 Related Work In this section, we will introduce the primary challenges encountered by CSP methods (Section 2.1 ), commonly used crystal structure characterization techniques (Section 2.2 ), encoding network models (Section 2.3 ) and crystal generation models (Section 2.4 ). 2.1 Problem Formulation A crystal structure consists of atoms interacting with each other, where each atom is characterized by its element type and coordinates. The goal of learning its representation is to train an encoder that accurately maps the crystal structure into a representation within a sampling space. To preserve the properties of the crystal structure after transformations, the structure must maintain translation, rotation, inversion, and permutation invariance ( Keating 1966 ; Lin et al. 2022 ) . Among these, translation, rotation, and inversion operations form E(3) symmetry, while translation and rotation operations form SE(3) symmetry. However, current models often struggle to effectively capture key features of crystal structures like symmetry and periodicity, particularly global symmetry, which can result in generated crystal structures failing to retain correct physical properties. Effectively incorporating these symmetries into material structure representation and neural networks has emerged as a significant challenge in crystal structure prediction. This issue also represents a classic problem in structural encoding. 2.2 Crystal Representation The crystal structure representation methods form the foundation for studying the relationship between crystal properties and structures. Previous representation techniques, such as those based on simple matrices ( Oubari et al. 2012 ; Dan et al. 2020 ; Kim et al. 2020 ; Long et al. 2021 ) , are inadequate for representing complex systems. The