CLOUD: A Scalable and Physics-Informed Foundation Model for Crystal Representation Learning
Changwen Xu, Shang Zhu, Venkatasubramanian Viswanathan · 2025
Read CLOUD: A Scalable and Physics-Informed Foundation Model for Crystal Representation Learning by Changwen Xu, Shang Zhu, Venkatasubramanian Viswanathan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-17345.txt.
Introduction
New materials are usually found by slow lab work or heavy physics calculations, and both routes limit how many crystals can be screened. Changwen Xu, Shang Zhu, and Venkatasubramanian Viswanathan present CLOUD in a 2025 paper, a foundation model that reads crystals as text. The model learns from structure alone, then predicts properties for crystals it has never seen.
CLOUD rests on a string encoding that packs a crystal's symmetry, atom positions by symmetry class, and makeup into a compact, coordinate-free line. The transformer model was pre-trained on more than six million crystal structures. Fine-tuning then adapted it to many property tasks, where it matched strong rivals and scaled well with size. As a proof of concept, the team also used it in a differentiable way to predict heat traits of crystals.
Catalog record
| CREATOR | Changwen Xu, Shang Zhu, Venkatasubramanian Viswanathan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 98 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-17345
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CLOUD: A Scalable and Physics-Informed Foundation Model for Crystal Representation Learning Changwen Xu, Shang Zhu, Venkatasubramanian Viswanathan Abstract The prediction of crystal properties is essential for understanding structure-property relationships and accelerating the discovery of functional materials. However, conventional approaches relying on experimental measurements or density functional theory (DFT) calculations are often resource-intensive, limiting their scalability. Machine learning (ML) models offer a promising alternative by learning complex structure-property relationships from data, enabling faster predictions. Yet, existing ML models often rely on labeled data, adopt representations that remain limited in capturing essential structural characteristics, and lack integration with physical principles—factors that constrain their generalizability and interpretability. Here, we introduce CLOUD (Crystal Language mOdel for Unified and Differentiable materials modeling), a transformer-based framework trained on a novel Symmetry-Consistent Ordered Parameter Encoding (SCOPE) that encodes crystal symmetry, Wyckoff positions, and composition in a compact, coordinate-free string representation. Pre-trained on over six million crystal structures, CLOUD is fine-tuned on multiple downstream tasks and achieves competitive performance in predicting a wide range of material properties, demonstrating strong scaling performance with respect to data and model size. Furthermore, as a proof of concept of differentiable materials modeling, CLOUD is applied to predict the phonon internal energy and heat capacity, which integrates the Debye model to preserve thermodynamic consistency. This approach enforces thermodynamic consistency and enables temperature-dependent property prediction without requiring additional data, illustrating the broader potential of combining ML with physical laws for materials applications. These results demonstrate the potential of CLOUD as a scalable and physics-informed foundation model for crystalline materials, unifying symmetry-consistent representations with physically grounded learning for property prediction and materials discovery. Introduction Predicting the properties of crystalline materials is of scientific and technological significance, as it enables insight into structure-property relationships and facilitates the discovery of novel functional materials Wang and Pan 2008 ; Pilania et al. 2013 ; Ahmad et al. 2018 ; Tabor et al. 2018 ; Pyzer-Knapp et al. 2022 . Accurate knowledge of material properties underpins the design of functional materials across a broad range of applications, including electronics Giustino 2014 , energy storage Dave et al. 2022 ; Annevelink et al. 2022 , and catalysis Nørskov et al. 2009 ; Annevelink et al. 2022 . One illustrative example is the heat capacity at constant volume ( C_{v} ) which serves as a representative example of a thermodynamic property that plays a key role in vibrational analysis and phase stability Grimvall 1999 ; Kauwe et al. 2018 ; Shankar 2012 . As such, the ability to predict these properties from a material’s structure is essential for accelerating the development of new materials and deepening our understanding of structure–property relationships. Conventionally, crystal properties are evaluated through experimental characterization or theoretical simulations. In the lab, properties such as crystal structure, conductivity, and stability are measured using techniques like X-ray diffraction, spectroscopy, or electrochemical analysis. On the theoretical side, Density Functional Theory (DFT) is widely used to compute formation energies, electronic structures, and elastic constants from first principles. However, these approaches are often time-consuming and resource-intensive, limiting their scalability for high-throughput screening and materials discovery McCullough et al. 2020 ; Chen et al. 2024 . Consequently, materials development has been constrained to a limited portion of the vast chemical space of synthesizable crystalline materials due to the high costs associated with evaluating material properties within specific systems Hellenbrandt 2004 ; Davies et al. 2016 . To overcome these limitations, the materials science community has increasingly embraced data-driven methods for predicting material properties with high throughput, reducing reliance on costly simulations and resource-intensive experiments. Among these, machine learning surrogates have proven especially effective, as they can learn complex structure-property relationships from data and enable rapid, scalable property predictions. Leveraging these strengths, ML models have the potential to substantially broaden the scope of material performance evaluation in the vast chemical space of crystals Phuthi et al. 2024 . Pioneering work has significantly improved the prediction accuracy for crystals ( Gilmer et al. 2017 ; Xie and Grossman 2018 ; Goodall and Lee 2020 ; Choudhary and DeCost 2021 ; Ruff et al. 2023 ) . Rational design of representations that map crystals to continuous vector space has enabled the development of more accurate and generalizable machine learning models for crystal property prediction. Early models focused on composition-based representations Goodall and Lee 2020 ; Ihalage and Hao 2022 ; Wang et al. 2021 , which are simple and broadly accessible but suffer from limited expressiveness due to their inability to distinguish between structural polymorphs. Structure-based models overcome this limitation by leveraging atomic coordinates, often through graph neural networks, to capture geometric and spatial information Gilmer et al. 2017 ; Schütt et al. 2018 ; Xie and Grossman 2018 ; Satorras et al. 2021 ; Choudhary and DeCost 2021 . However, these models struggle with long-range interactions Gong et al. 2023 and require costly structural data. To bridge the gap, coordinate-free models have emerged, offering a balance between structural awareness and computational efficiency by encoding symmetry and connectivity without explicit coordinates Goodall et al. 2022 ; Xiao et al. 2023 ; Huang et al. 2023 . Nevertheless, current coordinate-free representations still face challenges in efficiently encoding critical structural information for accurate property predictions, motivating the need for improved representations that balance structural richness and computational simplicity. Despite the advances in material property prediction with machine learning, state-of-the-art (SOTA) models typically rely on labeled training data obtained through costly wet-lab experiments or density functional theory (DFT) calculations. This reliance limits their applicability and scalability due to the scarcity, high cost, and variability of labeled materials datasets Fujinuma et al. 2022 . Moreover, these models often exhibit poor out-of-distribution (OOD) generalization, which is critical for real-world applications where crystals with novel compositions or structures are common Phuthi et al. 2024 . Recently, foundation models (FMs) Bommasani et al. 2021 , which are trained on broad data at scale and can be adapted to a wide range of downstream tasks, have emerged as a promising solution. Most FMs adopt a transformer-based architecture Vaswani et al. 2017 , enabling them to be pre-trained at scale using vast amounts of unlabeled data Devlin et al. 2018 ; Brown 2020 ; Lu et al. 2025 , which are orders of magnitude more abundant than labeled data. The key motivation for pre-training on large-scale unlabeled datasets is to learn universal representations of crystals through self-supervised learning. These representations can be effectively transferred to a wide range of downstream tasks, improving model performance even in low-data regimes. Furthermore, the implicit parallelism of attention allows for massively scalable transformer architectures, whose performance can improve predictably with the size of the model and training data, following neural scaling laws Kaplan et al. 2020 ; Hoffmann et al. 2022 ; Nayak and Varshney 2024 . If such a scaling behavior holds, FMs are expected to achieve SOTA performance simply by increasing the data and model capacity. FMs leverage self-supervised pretraining to capture high-level structural and chemical patterns from unlabeled data Bommasani et al. 2021 ; Frey et al. 2023 , which has shown promise in predicting crystal properties Merchant et al. 2023 ; Antunes et al. 2023 ; Gruver et al. 2024 . In the molecular domain,