Latent Diffusion Pretraining for Crystal Property Prediction

Shrimon Mukherjee, Kishalay Das, Partha Basuchowdhuri, Pawan Goyal, Niloy Ganguly · 2026

Read Latent Diffusion Pretraining for Crystal Property Prediction by Shrimon Mukherjee, Kishalay Das, Partha Basuchowdhuri, Pawan Goyal, Niloy Ganguly (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-00776.txt.

Introduction

Mukherjee and co-authors teach models the shape of crystals in this 2026 paper. Fast, sound forecasts of crystal traits drive the hunt for new materials. Graph networks and Transformers read the local setting of atoms well. But they need much labeled data, and labels for crystal traits are scarce.

Their frame, CrysLDNet, pretrains with latent diffusion to ease that lack. A variational autoencoder maps 3D crystal structures into a smooth latent space. The diffusion process then runs inside that space, on large sets of unlabeled crystals. This teaches the encoder rich facts about structure and chemistry before any trait labels are used. The model is then fine-tuned for the trait at hand. On the JARVIS and MP datasets, the authors report gains of 4.26 and 4.90 percent over strong baselines. The learned features also hold up when labels are sparse.

Catalog record

CREATORShrimon Mukherjee, Kishalay Das, Partha Basuchowdhuri, Pawan Goyal, Niloy Ganguly
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE77 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.00776

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Latent Diffusion Pretraining for Crystal Property Prediction Shrimon Mukherjee, Kishalay Das, Partha Basuchowdhuri, Pawan Goyal, Niloy Ganguly Abstract Fast and accurate prediction of crystal properties is a central challenge in new materials design. Graph neural networks and Transformer-based models have emerged as powerful tools for this task due to their ability to encode the local structural environment of atoms within a crystal. However, these models are data-hungry and in practice labeled data for crystal properties are very scarce. Pretraining–finetuning strategies, particularly those based on diffusion models, have shown promise in addressing these limitations. In this work, we introduce a novel latent-diffusion based pretraining framework CrysLDNet, designed to mitigate the data scarcity. Our approach integrates a Variational Autoencoder (VAE) with a diffusion model during the pretraining stage. The VAE encoder maps 3D crystal structures into a smooth latent space, within which the diffusion process is applied. This latent diffusion pretraining enables the graph encoder to effectively capture structural and chemical semantics from large-scale unlabeled data, which can then be finetuned for specific property prediction tasks. Comprehensive experiments on popular DFT datasets for property prediction reveal that CrysLDNet significantly outperforms both training-from-scratch and pretrained baselines, with improvements of 4.26% and 4.90% on the JARVIS and MP datasets. Additionally, the learned representations remain robust in sparse-data conditions and are expressive enough to correct DFT errors when finetuned with limited experimental data. Code is available at https://github.com/shrimonmuke0202/CrysLDNet.git . † † affiliationnotice: Equal contribution 1 Introduction Accurate and efficient prediction of chemical properties is a critical step in the materials design pipeline. While Density Functional Theory (DFT) ( 45 ) has long been the standard tool for estimating crystal properties, its high computational cost severely limits large-scale screening. Recent advances in machine learning have provided data-driven alternatives ( 18 ; 38 ; 19 ; 68 ; 11 ) that achieve DFT-comparable accuracy at a fraction of the cost. In particular, graph neural network (GNN)–based models ( 66 ; 3 ; 37 ; 46 ; 52 ; 4 ) , followed by transformer-based architectures ( 70 ; 32 ; 33 ; 69 ; 55 ) , have progressively advanced the state of the art. Despite their success, these supervised (train-from-scratch) models require large labeled datasets, which are scarce in materials science and unevenly distributed across properties (Table 7 ). Also, curating property-labeled datasets through simulations or experiments is both time- and resource-intensive. As a result, existing supervised models often exhibit suboptimal performance in low-data regimes ( 9 ; 61 ) , particularly for properties that lack sufficient training data. In contrast, large collections of unlabeled crystal structures with only 3D structural information are readily available. This has motivated a line of work on self-supervised pretraining for crystals, including CrysXPP ( 10 ) , CrysGNN ( 9 ) and Crystal Twins ( 41 ) . These methods pretrain encoders on unlabeled crystal structures and subsequently fine-tune them on small property-labeled datasets, following a pretraining–finetuning paradigm. Recent diffusion-based approaches, such as CrysDiff ( 61 ) , and DPF ( 54 ) further enhance pretraining by reconstructing perturbed crystal structures. However, they operate directly in the high-dimensional feature space, jointly modeling atom types, fractional coordinates, and lattice parameters. This space is inherently heterogeneous: atom types are discrete and modeled via discrete diffusion (e.g., D3PM ( 2 ) ), lattice parameters are continuous and handled by DDPMs ( 21 ) , while fractional coordinates are periodic and therefore require score-based modeling with wrapped normal distributions ( 60 ) . Such heterogeneous modeling necessitates complex architectures and many diffusion steps, resulting in inefficient training and representations that remain constrained by this non-smooth input space. In this work, we adopt a latent diffusion-based pretraining framework to overcome the limitations of conventional diffusion-based pretraining. Crystal properties such as formation energy and bandgap are fundamentally governed by atomic arrangement and lattice structure. Therefore, learning compact, expressive representations that faithfully capture both composition and 3D geometry is more beneficial than modeling the raw feature space directly. Building on this insight, we propose CrysLDNet, a novel pretraining strategy based on latent diffusion that learns robust and expressive crystal representations to improve downstream property prediction. Our framework consists of two components: a Variational Autoencoder (VAE) and a Latent Diffusion Model (LDM). The VAE encoder compresses high-dimensional crystal structures into a smooth, compact latent space, while the decoder reconstructs the original structure. The LDM then learns the distribution of these latent representations by progressively noising and denoising them using a transformer-based architecture. This latent diffusion process guides the encoder to capture richer structural semantics, yielding representations that are better aligned with downstream property prediction tasks. Empirically, we show (Figure 3 ) that embeddings learned via latent diffusion more accurately reconstruct atomic composition and 3D structure than feature-space diffusion methods. Finally, the pretrained encoder is fine-tuned with a lightweight property predictor using limited labeled data. Extensive experiments on widely used DFT benchmark datasets demonstrate that CrysLDNet consistently outperforms both training-from-scratch models and existing pretrained baselines across all evaluated properties. In particular, CrysLDNet achieves substantial relative error reductions (typically in the range of 4.26% – 19.34% ) compared to strong diffusion- and GNN-based baselines, with the largest gains observed in low-data regimes, where labeled samples are scarce. Beyond performance improvements, CrysLDNet offers an important practical advantage: it is backbone-agnostic. The encoder architecture can be seamlessly replaced with more powerful future models without modifying the latent diffusion framework or retraining strategy. In particular, we observe that replacing the VAE encoder from Matformer to PDDFormer leads to an average performance improvement of 10.46% on JARVIS and 12.39% on the MP dataset. This design makes CrysLDNet inherently future-proof, allowing it to benefit directly from ongoing advances in crystal representation learning and transformer architectures. Conflict of Interest Disclosure The authors declare that there are no conflicts of interest. Figure 1 : Overview of our proposed pretrain–finetune framework, CrysLDNet. In the pretraining stage, a VAE encodes crystal structures into latent representations, on which the LDM is applied to refine the encoded latent space. CrysLDNet is designed to be fully agnostic to the choice of different backbone encoder architectures. The pretrained encoder is then finetuned on labeled data for downstream tasks. 2 Preliminaries 2.1 Crystal Representation Crystal materials can be viewed as a 3D point cloud (Fig. 4 (a)) of atoms arranged in an orderly repeating pattern. For a material with N atoms in its unit cell, the structure can be defined as \textbf{{M}}=(\textbf{{A}},\textbf{{X}},\textbf{{L}}) . Atom Type Matrix: \textbf{{A}}=[\mathbf{{a_{1}}},\mathbf{a_{2}},...,\mathbf{a_{N}}]^{T}\in\mathbb{R}^{N\times k} , where each \mathbf{a_{i}} is a one-hot vector denoting the atomic type of the i^{th} atom, and k is the maximum number of possible atom types. Coordinate Matrix: \textbf{{X}}=[\mathbf{x_{1}},\mathbf{x_{2}},...,\mathbf{x_{N}}]^{T}\in\mathbb{R}^{N\times 3} , where \mathbf{x_{i}}\in\mathbb{R}^{3} represents the 3D coordinates of the i^{th} atom in the unit cell. Lattice Matrix: \textbf{{L}}=[\mathbf{l_{1}},\mathbf{l_{2}},\mathbf{l_{3}}]^{T}\in\mathbb{R}^{3\times 3} , which specifies how the unit cell repeats itself in 3D space along directions \mathbf{l_{1}},\mathbf{l_{2}}, and \mathbf{l_{3}} to form the periodic crystal. Symmetry in Crystal Structure. Crystal materials exhibit fundamental physical symmetries that any learned representation must respect. One such property is rotational invariance, which ensures that rotating the atom coordinates and lattice matrices by any orthogonal matrix \mathbf{Q} results in an equivalent representation of the same material. Another key property is periodic translation invariance, meaning that

Read the complete text