Improving crystal material property prediction with multi-view geometric graph transformer Authors: Liang Zhang, Ziyue Wang, Xin Wang, Xinyu Zhang, Yu Mao, Kong Chen, Yuen Wu Published: 2026 ABSTRACT Accurately and comprehensively representing crystal structures is critical for advancing machine learning in large-scale crystal materials simulations. However, effectively capturing and leveraging the intricate geometric and topological characteristics of crystal structures remains a significant challenge for most existing methods in crystal property prediction. Here, we propose MGT, a multi-view graph transformer that jointly models SE(3) invariant scalar representations and SO(3) equivariant directional representations, enabling the capture of both rotational–translational invariance and rotation-equivariant directional information in crystal structures. A mixture of experts inspired router serves as the key integration mechanism, adaptively weighting these complementary embeddings for each target task. Through multi-task self-supervised pretraining, MGT achieves up to 14% reduction in mean absolute error compared with previous state-of-the-art models on crystal property benchmarks. Comprehensive ablation and interpretability analyses confirm that both the self-supervised pretraining strategy and the mixture of experts inspired router contribute to the overall model performance. In transfer learning scenarios—including crystal catalyst adsorption energy and hybrid perovskite bandgap prediction—MGT achieves performance improvements of up to 58% over existing baselines, demonstrating strong domain-agnostic scalability. Overall, all the results suggest that MGT is an effective and generalizable framework for crystal material property prediction, with significant potential to accelerate the discovery of novel materials. Introduction Crystal materials are widely used in areas such as energy storage, catalysis, drug discovery and semiconductor design due to their unique structures and properties1–4. Accurate prediction of the properties of crystal materials is crucial for accelerating the discovery of new materials with specific functions5,6. However, traditional methods such as Density Functional Theory (DFT) calculations or molecular dynamics (MD) simulations are both time-consuming and costly. The computational complexity of DFT scales cubically with the system size, which requires substantial computational resources and time, especially when dealing with material systems containing a large number of atoms7–9. In recent years, significant progress has been made through the development of graph neural network (GNN)-based models, enabling rapid and accurate crystal material property prediction. In 2018, Xie et al. introduced the CGCNN framework, capable of learning material properties directly from atomic connections in crystal graphs10. Since then, a variety of GNN-based models have emerged. For instance, Matformer11, a graph transformer designed to be invariant to periodicity, explicitly captures repeating patterns and characterizes crystal structures through geometric distances between identical atoms in neighboring cells. PotNet12 innovatively models the complete set of interatomic potentials among all atoms by computing physics-principled potentials, such as the Coulomb, London dispersion and Pauli repulsion potentials, achieving the infinite summations of pairwise atomic interactions. However, employing graph representations that only consider local information for crystal structures cannot guarantee a one-to-one correspondence between the representation and the crystal graph13. This limitation makes it difficult for GNN models to distinguish distinct crystal materials, thus impacting prediction performance. To address this challenge, PerCNet13 incorporates dihedral angles to resolve the many-to-one problem in crystal graph representations. Ablation experiments confirm that this additional information is essential for the geometric completeness of periodic structures. Specially, in crystal materials, atoms are arranged in a periodic lattice in three-dimensional (3D) space, where the geometric structure is SE(3) invariant or SO(3) equivariant under specific translations and rotations14. These properties reflect the underlying spatial symmetries of the crystal, corresponding to symmetry operations of a subgroup of the Euclidean group E3. Indeed, recent works based on SE(3) invariant and SO(3) equivariant have already shown great potential in atomic systems15–17. Equiformer16 leverages the transformer architecture18 to achieve strong empirical results on 3D atomic benchmarks such as QM9, MD17 and OC20, highlighting the advantage of transformer-based model with equivariant features in atomistic systems. ComFormer14 introduces two variants, iComFormer and eComFormer, by incorporating SE(3) invariant and SO(3) equivariant crystal graph representations, respectively. As a result, ComFormer demonstrates excellent performance across multiple crystal property prediction tasks, underscoring the efficiency and expressiveness of SE(3) invariant and SO(3) equivariant graph features. Although there have been significant advancements in leveraging SE(3) invariant and SO(3) equivariant graph features, current state-of-the-art models still fall short of fully harnessing their potential. For instance, prior works have predominantly focused on utilizing these features in isolation within neural frameworks, neglecting their intrinsic synergistic capabilities14. Furthermore, while one19 has incorporated domain knowledge and rules of dual-graph features, it primarily relies on a simple concatenation method that fail to exploit the advantages of the leveraged knowledge. Consequently, the absence of an integrated framework to effectively fuse multi-view features and maximize their synergistic benefits remains a long-standing unresolved challenge20, highlighting the need for a unified approach that can unlock their full potential and enhance the overall performance of crystal material property prediction tasks. In this work, we develop MGT (Multi-view Graph Transformer), designed to fully exploit the comprehensive and accurate geometric features of both SE(3) invariance and SO(3) equivariance. We leverage pretraining and fine-tuning strategies21,22 with MGT. Specifically, a multi-task self-supervised learning23,24 (SSL) pretraining strategy, including denoising and contrastive learning, is employed to enhance the model’s capability to effectively recognize crystal structures. In the fine-tuning stage, we introduce a lightweight mixture of experts inspired (MoE-inspired)25,26 module within MGT to dynamically and efficiently process SE(3) and SO(3) embeddings, which meaningfully improves the performance of the model in downstream tasks. We compare MGT with a series of models for crystal material property prediction and conduct ablation studies to systematically evaluate the contribution of each component within MGT. Our results confirm that MGT outperforms the corresponding best baseline models across multiple standard crystal property prediction benchmarks, including the Materials Project and JARVIS datasets, achieving performance improvements ranging from 2.1 to 13.6%. Our illustrations demonstrate that MGT has effective clustering in the latent space, as revealed by t-Distributed Stochastic Neighbor Embedding (t-SNE)27 visualization. Furthermore, we evaluate the generalization ability of MGT in transfer learning scenarios, including catalyst global minimum adsorption energy (GMAE)28 prediction and hybrid organic–inorganic perovskite (HOIP)29 bandgap prediction. In these task-specific settings, MGT is compared against established state-of-the-art models within each domain and achieves relative performance gains ranging from 15.4% to 57.8%. Our main contributions in this work are as follows. We develop a multi-view graph transformer framework named MGT for crystal property prediction, consisting of two graph transformer encoders and a light MoE-inspired router to efficiently process SE(3) invariant and SO(3) equivariant graph features. We introduce an SSL pretraining strategy for MGT that incorporates denoising and contrastive learning in the pre-training stage. The effectiveness of this pretraining approach is validated through ablation experiments. We demonstrate that MGT can effectively distinguish distinct crystal materials in the latent space. Further, these insights elucidate the underlying factors for the superior performance of MGT. We successfully implement MGT in catalyst GMAE and HOIP bandgap prediction tasks, achieving competitive prediction performance and demonstrating its generalizability and scalability in practical applications. Results Preliminaries A crystal material \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$M$$\end{document}M can generally be represented as a periodic arrangement of atoms within a unit cell30. Formally, give \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$M=({{\bf{A}}},\,{{\bf{X}}},{{\bf{L}}})$$\end{document}M=(A,X,L), where the infinite periodic structure can be expressed as:1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left\{\left({\hat{a}}_{i},{\hat{{{\bf{x}}}}}_{i}\right),|,{\hat{a}}_{i}={a}_{i},{\hat{{{\bf{x}}}}}_{i}={{{\bf{x}}}}_{i}+{k}_{1}{{\mathcal{l}}}_{1}+{k}_{2}{{\mathcal{l}}}_{2} +{k}_{3}{{\mathcal{l}}}_{3},{\,k}_{1},{k}_{2},{k}_{3}{\mathbb{\in }}{\mathbb{Z}}{\mathbb{,}}1\le i\le n\right\}$$\end{document}a^i,x^i,∣,a^i=ai,x^i=xi+k1l1+k2l2+k3l3,k1,k2,k3\inZ,1≤i≤nwhere \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\bf{A}}}=\left[{a}_{1},{a}_{2},\ldots,{a}_{n}\right]\in {{\mathbb{A}}}^{N}$$\end{document}A=a1,a2,…,an∈AN denotes the list of atomic species, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\bf{X}}}=\left[{{{\bf{x}}}}_{1},{{{\bf{x}}}}_{2},\ldots,{{{\bf{x}}}}_{n}\right]\in {{\mathbb{R}}}^{N\times 3}$$\end{document}X=x1,x2,…,xn∈RN×3 is the corresponding atomic coordinates, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\bf{L}}}=\left[{{\mathcal{l}}}_{1},{{\mathcal{l}}}_{2},{{\mathcal{l}}}_{3}\right]\in {{\mathbb{R}}}^{3\times 3}$$\end{document}L=l1,l2,l3∈R3×3 is the lattice matrix defining the periodic translations in three dimensions and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${k}_{1},{k}_{2},{k}_{3}$$\end{document}k1,k2,k3 are integer translation indices along the respective lattice directions, used to extend the unit cell periodically in 3D space. A crystal representation \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\rm{f}}}:({{\bf{A}}},{{\bf{X}}},{{\bf{L}}})\to {\rm Z}$$\end{document}f:(A,X,L)→Z should preserve fundamental geometric symmetries such as SE(3) invariance and SO(3) equivariance, to ensure rotational consistency and geometric completeness. The representation is SE(3) invariant if it satisfies: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\rm{f}}}\left({{\bf{A}}},{{\bf{X}}},{{\bf{L}}}\right)={{\rm{f}}}({{\bf{A}}},{{\bf{X}}}{{{\bf{R}}}}^{T}+{{\bf{b}}},{{\bf{RL}}})$$\end{document}fA,X,L=f(A,XRT+b,RL), for any rotation matrix \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\bf{R}}}\in {{\mathbb{R}}}^{3\times 3}$$\end{document}R∈R3×3, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left|{{\bf{R}}}\right|$$\end{document}R = 1 and translation vector \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\bf{b}}}\in {{\mathbb{R}}}^{3}$$\end{document}b∈R3. This ensures that the output remains unchanged under rotations and translations applied simultaneously to atomic coordinates \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\bf{X}}}$$\end{document}X and lattice vectors \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\bf{L}}}$$\end{document}L. The representation is SO(3) equivariant if it satisfies: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\rm{f}}}\left({{\bf{A}}},{{\bf{X}}}{{{\bf{R}}}}^{T},{{\bf{RL}}}\right)={{\rm{f}}}\left({{\bf{A}}},{{\bf{X}}},{{\bf{L}}}\right){{\bf{R}}}$$\end{document}fA,XRT,RL=fA,X,LR, for any rotation matrix \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\bf{R}}}\in {{\mathbb{R}}}^{3\times 3}$$\end{document}R∈R3×3, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left|{{\bf{R}}}\right|$$\end{document}R = 1. This ensures that vectorial outputs rotate consistently with the input. Moreover, to capture symmetries and periodicity, graph features are constructed by associating each atom \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i$$\end{document}i with periodic images \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${j}^{{\prime} }$$\end{document}j′ of other atoms \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$j$$\end{document}j. The position of each periodic image \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${x}_{{j}^{{\prime} }}$$\end{document}xj′ is defined as:2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\bf{x}}}}_{{j}^{{\prime} }}={{{\bf{x}}}}_{i}+{k}_{1}{{\mathcal{l}}}_{1}+{k}_{2}{{\mathcal{l}}}_{2}+{k}_{3}{{\mathcal{l}}}_{3},{\,k}_{1},{k}_{2},{k}_{3}{\mathbb{\in }}{\mathbb{Z}}$$\end{document}xj′=xi+k1l1+k2l2+k3l3,k1,k2,k3\inZ An edge is formed between atom \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i$$\end{document}i and a periodic image \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${j}^{{\prime} }$$\end{document}j′ if the Euclidean distance satisfies: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{||}{{{\bf{x}}}}_{{j}^{{\prime} }}-{{{\bf{x}}}}_{i}{||}}_{2}\le r$$\end{document}∣∣xj′−xi∣∣2≤r, where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$r$$\end{document}r is a predefined cutoff radius. For SE(3) invariance, edges incorporate scalar geometric features, including Euclidean distances \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{||}{{{\bf{x}}}}_{{j}^{{\prime} }}-{{{\bf{x}}}}_{i}{||}}_{2}$$\end{document}∣∣xj′−xi∣∣2 and angles \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\theta }_{{j}^{{\prime} }i,i{i}_{k}},{k}\in \{1,\,2,\,3\}$$\end{document}θj′i,iik,k∈{1,2,3} between edge vectors \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\bf{x}}}}_{{j}^{{\prime} }}-{{{\bf{x}}}}_{i}$$\end{document}xj′−xi and lattice reference vectors \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left\{{{{\bf{e}}}}_{i{i}_{1}},{{{\bf{e}}}}_{i{i}_{2}},{{{\bf{e}}}}_{i{i}_{3}}\right\}$$\end{document}eii1,eii2,eii3, which are constructed from the closest periodic images of atom \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i$$\end{document}i and preserve the periodic invariance. For SO(3) equivariant, edges include both the scalar distances \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{||}{{{\bf{x}}}}_{{j}^{{\prime} }}-{{{\bf{x}}}}_{i}{||}}_{2}$$\end{document}∣∣xj′−xi∣∣2 and the equivariant vectors \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\bf{x}}}}_{{j}^{{\prime} }}-{{{\bf{x}}}}_{i}$$\end{document}xj′−xi, which transforms under rotation. In addition, the geometric completeness of SE(3) invariant and SO(3) equivariant crystal graph features respects both lattice periodicity and geometric symmetries, which can uniquely determine a crystal structure, as demonstrated in prior work14. Framework As shown in Fig. 1, MGT is composed of two graph encoders and a MoE-inspired router. Each graph encoder incorporates a graph transformer architecture and a lightweight projection head. Specifically, the SE(3) graph encoder employs both edge-wise and node-wise transformer layers to update node, edge and angle features via attention-based message passing, effectively capturing SE(3) invariant geometric information. In contrast, the SO(3) graph encoder leverages spherical harmonics31 and tensor product (TP) layers32 to ensure SO(3) rotational equivariance while enriching structural representations. In addition, to leverage both SE(3) invariant and SO(3) equivariant features during the fine-tuning on downstream tasks, The lightweight MoE-inspired router adaptively integrates SE(3) and SO(3) embeddings via a self-attention-based module26, enabling task-specific fusion of geometry-aware representations.Fig. 1Schematic diagram of MGT framework workflow.a Database construction for pretraining and fine-tuning. The pretraining database is used for self-supervised representation learning, whereas the fine-tuning database is used for downstream property prediction. b SE(3) invariant features include the edge length \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{||}{x}_{{j}^{{\prime} }}-{x}_{i}{||}}_{2}$$\end{document}∣∣xj′−xi∣∣2 and the angle \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\theta }_{{j}^{{\prime} }i,i{i}_{k}}$$\end{document}θj′i,iik. SO(3) equivariant features include the edge length \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{||}{x}_{{j}^{{\prime} }}-{x}_{i}{||}}_{2}$$\end{document}∣∣xj′−xi∣∣2 and the equivariant vectors \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${x}_{{j}^{{\prime} }}-{x}_{i}$$\end{document}xj′−xi. Specifically, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${x}_{i}$$\end{document}xi and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${x}_{{j}^{{\prime} }}$$\end{document}xj′ denote the atomic coordinates of the atom \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i$$\end{document}i and its neighboring atom \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${j}^{{\prime} }$$\end{document}j′, respectively, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${i}_{k}$$\end{document}ik denotes the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k$$\end{document}k-th neighboring atom used for angle, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k\in \{1,\,2,\,3\}$$\end{document}k∈{1,2,3}. c For pretraining, using multi-objective SSL strategy that simultaneously performs denoising and contrastive learning. d For fine-tuning, MoE-inspired router is employed to adaptively adjust the weight assigned to SE(3) and SO(3) embeddings based on the specific target task. Schematic diagram of MGT framework workflow. a Database construction for pretraining and fine-tuning. The pretraining database is used for self-supervised representation learning, whereas the fine-tuning database is used for downstream property prediction. b SE(3) invariant features include the edge length \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{||}{x}_{{j}^{{\prime} }}-{x}_{i}{||}}_{2}$$\end{document}∣∣xj′−xi∣∣2 and the angle \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\theta }_{{j}^{{\prime} }i,i{i}_{k}}$$\end{document}θj′i,iik. SO(3) equivariant features include the edge length \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{||}{x}_{{j}^{{\prime} }}-{x}_{i}{||}}_{2}$$\end{document}∣∣xj′−xi∣∣2 and the equivariant vectors \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${x}_{{j}^{{\prime} }}-{x}_{i}$$\end{document}xj′−xi. Specifically, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${x}_{i}$$\end{document}xi and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${x}_{{j}^{{\prime} }}$$\end{document}xj′ denote the atomic coordinates of the atom \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i$$\end{document}i and its neighboring atom \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${j}^{{\prime} }$$\end{document}j′, respectively, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${i}_{k}$$\end{document}ik denotes the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k$$\end{document}k-th neighboring atom used for angle, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k\in \{1,\,2,\,3\}$$\end{document}k∈{1,2,3}. c For pretraining, using multi-objective SSL strategy that simultaneously performs denoising and contrastive learning. d For fine-tuning, MoE-inspired router is employed to adaptively adjust the weight assigned to SE(3) and SO(3) embeddings based on the specific target task. Recent studies in representation learning have demonstrated that SSL pretraining can significantly enhance the generalization of models on downstream tasks33–35. Admittedly, to enhance the effectiveness of model and reduce its reliance on large labeled datasets, MGT also adopts a multi-task SSL pretraining strategy tailored to the challenges of crystal property prediction. This strategy includes two distinct pretraining strategies: denoising learning and contrastive learning. For the denoising learning strategy36, Gaussian noise is added to angle and edge features for SE(3) and SO(3) representations, respectively. Following the approach of GPIP framework37, MGT is trained to restore the disturbed noise, rather than reconstructing the original features. This training strategy forces MGT to learn invariant and equivariant features that remain stable under local perturbations, while preserving geometric symmetries. For the contrastive learning strategy38, MGT aligns SE(3) invariant with SO(3) equivariant representations, maximizing the mutual information across these geometric features. This contrastive strategy forces the model to learn complementary features from each representation, enhancing the model’s capability to generalize across various crystal structures. The combination of these two pretraining strategies enables MGT to capture meaningful spatial structural features and enhance cross-view consistency between SE(3) and SO(3) embeddings. Performance evaluation We evaluated the performance of MGT framework on nine downstream tasks using two benchmark crystal material datasets: Materials Project39 and JARVIS40. The evaluation was compared against several advanced baselines, including CGCNN10, ALIGNN41, Matformer11, PotNet12, ComFormer14 and ReGNet42. To ensure a fair comparison, we employed the same data splitting seed as used in prior studies and evaluated model performance with mean absolute error (MAE) and R2. As shown in Supplementary Tables 1, 2 and 12, MGT outperforms these baseline methods in eight downstream prediction tasks, achieving MAE reductions ranging from 2.1% to 13.6%. However, MGT achieves the best performance on all evaluated tasks except the Total Energy prediction task for JARVIS dataset, where ReGNet attains a slightly lower MAE. This is likely due to ReGNet’s reciprocal-space-based inductive bias, which is particularly well suited for modeling long-range energy contributions in periodic crystals42,43. Notably, as shown in Fig. 2, MGT demonstrates robust fitting performance across all downstream tasks, with the predicted data points predominantly clustering around the diagonal and all R2 values exceeding 0.95. Furthermore, to assess robustness, we conducted 9 additional independent runs with different random seeds across all downstream prediction tasks. Detailed results and performance comparisons are provided in Supplementary Note 6 and Supplementary Tables 9 and 10. These results demonstrate the effectiveness of multi-view features combining SE(3) invariance and SO(3) equivariance in MGT, leading to improved and stable model performance in crystal material property prediction tasks.Fig. 2Performance comparison of MGT on Materials Project and JARVIS datasets with MAE, RMSE and R-squared value (R2).a–d Actual and predicted results of Formation Energy, Band Gap, Bulk Moduli and Shear Moduli in Materials Project dataset. e–i Actual and predicted results of Formation Energy, Bandgap (OPT), Total Energy, Bandgap (MBJ) and Ehull in JARVIS dataset. Performance comparison of MGT on Materials Project and JARVIS datasets with MAE, RMSE and R-squared value (R2). a–d Actual and predicted results of Formation Energy, Band Gap, Bulk Moduli and Shear Moduli in Materials Project dataset. e–i Actual and predicted results of Formation Energy, Bandgap (OPT), Total Energy, Bandgap (MBJ) and Ehull in JARVIS dataset. Ablation studies To study the impact of each technique we adopted on model performance, we carried out ablation experiments in three aspects, which are the performance comparison with and without pretraining, the effectiveness of each pretraining strategy and the performance comparison with and without the MoE-inspired module in downstream task fine-tuning. A more detailed explanation of our settings is provided in Supplementary Note 5. Firstly, we compared MGT with and without pretraining. As shown in Fig. 3a and Supplementary Table 3, MGT with pretraining achieves an MAE of 0.0165, which is better than the MAE of 0.0174 obtained without pretraining on the Formation Energy prediction task for Materials Project dataset. This demonstrates that the pretraining strategy is more effective than training MGT framework from scratch in achieving high performance.Fig. 3Ablation experiment results of MGT with different training strategies. w/o: without. w/: with.a Performance comparison of MGT in the Formation Energy prediction task for Materials Project and JARVIS datasets, respectively. b Performance comparison of MGT in the Band Gap and Bandgap (OPT) prediction tasks for Materials Project and JARVIS datasets, respectively. Ablation experiment results of MGT with different training strategies. w/o: without. w/: with. a Performance comparison of MGT in the Formation Energy prediction task for Materials Project and JARVIS datasets, respectively. b Performance comparison of MGT in the Band Gap and Bandgap (OPT) prediction tasks for Materials Project and JARVIS datasets, respectively. Secondly, we studied MGT with the denoising learning strategy, the contrastive learning strategy and both strategies combined. As shown in Fig. 3 and Supplementary Table 3, MGT with both the denoising and contrastive learning strategies achieves the best model performance. However, it is worth noting that MGT with the denoising learning strategy achieves MAE of 0.116, outperforming MAE of 0.120 achieved by MGT with the contrastive learning strategy on the Bandgap (OPT) prediction task for JARVIS dataset. This performance trend also holds across both datasets in the Formation Energy prediction task. These findings suggest that the denoising pretraining strategy may be more critical than the contrastive learning strategy in crystal property prediction. By focusing on restoring noise, the denoising strategy enhances the perceptive ability of MGT to recognize SE(3) invariant and SO(3) equivariant geometric features. Furthermore, when the denoising pretraining is combined with the contrastive learning strategy, it improves the generalization of MGT across downstream prediction tasks, achieving the best performance. Lastly, we also evaluated the performance of MGT with and without the MoE-inspired router. In the absence of the MoE-inspired router, we replaced it with a simple NN architecture44. Specifically, the features from both SE(3) and SO(3) embeddings were concatenated and then passed through two fully connected layers. A more detailed description is provided in Supplementary Note 1. As shown in Fig. 3 and Supplementary Table 3, MGT with the MoE-inspired router achieves MAEs of 0.0261 and 0.114, outperforming MAEs of 0.0263 and 0.115 achieved by MGT without the MoE-inspired router on the Formation Energy and Bandgap (OPT) prediction tasks for JARVIS dataset, respectively. MGT with the MoE-inspired router also outperforms the one without the MoE-inspired router on Materials Project dataset. Therefore, these results strongly demonstrate that the MoE-inspired router in MGT enhances model performance by effectively leveraging SE(3) and SO(3) embeddings and dynamically integrating these features during the fine-tuning phase. This adaptive capability offers a clear advantage over traditional NN architectures in fusing dual representations. Moreover, we can also see that MGT still achieves better performance than baseline methods even without pretraining or MoE-inspired router, demonstrating the inherent effectiveness of multi-view architecture alone already outperforms the baselines. Interpretability analysis To effectively assess the contributions of each component within MGT framework, we utilized t-SNE27 for visualization analysis with SE(3), SO(3) and MoE-inspired router embeddings. Details can be found in Supplementary Note 2. As illustrated in Fig. 4a–d, the SE(3) and SO(3) graph encoders successfully cluster crystal materials based on the Formation Energy for Materials Project and JARVIS datasets. Furthermore, as illustrated in Fig. 4e, f, the MoE-inspired router strengthens the clustering, making the groupings more distinct. This improved clustering is crucial for MGT to distinguish distinct crystal materials. The clustering effectiveness is consistent across all prediction tasks, as shown in Supplementary Figs. 1–7. These results highlight the significant role of the MoE-inspired router in enhancing overall model performance on downstream tasks.Fig. 4Visualizations of the Formation Energy prediction task for Materials Project and JARVIS datasets.a SE(3) embeddings for Materials Project dataset. b SE(3) embeddings for JARVIS dataset. c SO(3) embeddings for Materials Project dataset. d SO(3) embeddings for JARVIS dataset. e MoE-inspired router embeddings for Materials Project dataset. f MoE-inspired router embeddings for JARVIS dataset. Visualizations of the Formation Energy prediction task for Materials Project and JARVIS datasets. a SE(3) embeddings for Materials Project dataset. b SE(3) embeddings for JARVIS dataset. c SO(3) embeddings for Materials Project dataset. d SO(3) embeddings for JARVIS dataset. e MoE-inspired router embeddings for Materials Project dataset. f MoE-inspired router embeddings for JARVIS dataset. Next, we also analyzed the actual contribution scores of SE(3) and SO(3) embeddings in the MoE-inspired router. Details are provided in Supplementary Note 3. As shown in Supplementary Fig. 12, in the Formation Energy prediction task, SO(3) embeddings exhibit a more significant contribution than SE(3) embeddings for Materials Project dataset. However, the opposite is observed for JARVIS dataset, where SE(3) embeddings contribute more. In addition, SE(3) and SO(3) embeddings contribute almost equally in the Band Gap prediction task for Materials Project dataset. In contrast, the Bandgap (OPT) prediction task for JARVIS dataset exhibits negative and positive contribution scores for SE(3) and SO(3) embeddings, respectively. These findings indicates that the MoE-inspired module within MGT can autonomously adjust the weight assigned to SE(3) and SO(3) embeddings depending on the specific prediction task. We also presented the contribution scores of other crystal property prediction tasks, as shown in Supplementary Figs. 13 and 14. The SE(3) and SO(3) graph encoders each demonstrate task-dependent strengths, and their weighting relationship appears to be complementary. For example, in the Bulk Moduli, Bandgap (OPT) and Bandgap (MBJ) prediction tasks for Materials Project and JARVIS datasets, respectively, this complementary nature is particularly evident. It is also worth noting that Ehull prediction task for JARVIS dataset, which reflects global thermodynamic stability, is more naturally captured by SE(3) embeddings. Moreover, these task-dependent preferences are physically reasonable. Properties dominated by directional deformation modes (e.g., Shear Moduli) require orientation-dependent geometric cues that may be better preserved by SO(3) equivariant embeddings45, whereas stability-related scalar quantities defined on an energy landscape (e.g., Ehull), which are inherently invariant to global rotations and translations of the crystal structure46, are more naturally captured by SE(3) invariant representations. Importantly, such preferences are not universal and can be dataset-dependent, as they are influenced by data distribution, crystal structural diversity and other dataset-specific factors. Admittedly, these observations underscore the necessity of a dynamic mechanism rather than a fixed fusion rule, providing strong justification for the MoE-inspired router in adaptively balancing SE(3) and SO(3) embeddings across different crystal property prediction tasks and thereby improving overall model performance. Transfer learning It is essential to evaluate the effectiveness of MGT framework by assessing its generalizability and scalability through transfer learning47,48. To this end, we applied MGT to two diverse transfer learning scenarios. The first involved predicting GMAE, a crucial reactivity descriptor that helps identify the most stable adsorption configurations of catalysts. A more detailed explanation of GMAE is provided in Supplementary Note 9. Accurate prediction of GMAE is essential for the effectiveness of machine learning algorithms in catalyst screening. As shown in Fig. 5a–c, MGT demonstrates satisfactory fitting performance across the Alloy-GMAE, FG-GMAE and OCD-GMAE datasets. Compared with the state-of-the-art AdsMT28 baseline, as detailed in Supplementary Tables 4 and 12, MGT achieves significant improvements of 46.4, 57.8 and 15.4%, respectively. The visualizations and contribution scores for these prediction tasks also confirm our observations, as shown in Supplementary Figs. 8–10 and 15.Fig. 5Performance comparison of MGT on Alloy-GMAE, FG-GMAE, OCD-GMAE and HOIP datasets with MAE, RMSE and R2.a Actual and predicted results of Alloy-GMAE prediction task. b Actual and predicted results of FG-GMAE prediction task. c Actual and predicted results of OCD-GMAE prediction task. d Actual and predicted results of HOIP bandgap prediction task. Performance comparison of MGT on Alloy-GMAE, FG-GMAE, OCD-GMAE and HOIP datasets with MAE, RMSE and R2. a Actual and predicted results of Alloy-GMAE prediction task. b Actual and predicted results of FG-GMAE prediction task. c Actual and predicted results of OCD-GMAE prediction task. d Actual and predicted results of HOIP bandgap prediction task. The second scenario focused on predicting the bandgap of HOIP29, for which precise prediction is critical for optimizing optoelectronic applications such as solar cells and light-emitting diodes. As shown in Fig. 5d and Supplementary Tables 5 and 12, MGT exhibits competitive performance compared to other baseline methods in the bandgap prediction task on the HOIP dataset, with a 25.7% improvement. We also presented all visualizations and contribution scores, as shown in Supplementary Figs. 11 and 15, respectively. Collectively, these results demonstrate the excellent generalization ability and scalability of MGT framework across diverse transfer learning scenarios, highlighting its promising potential for practical applications. Discussion In this work, we introduced MGT, a geometry-aware multi-view graph transformer that unifies SE(3) invariant scalar geometry and SO(3) equivariant vector geometry for crystal property prediction. Our extensive experiments have clearly demonstrated that MGT outperforms existing methods across multiple benchmark tasks, achieving accurate and reliable predictive performance. These results validate the rationality and effectiveness of our multi-task pretraining strategy and the robustness of our architecture design. We have also applied MGT to transfer learning scenarios, where it has demonstrated superior performance compared to state-of-the-art models in GMAE and HOIP bandgap prediction tasks. Importantly, our ablation studies illustrate that self-supervised pretraining is designed not only to improve performance on individual tasks, but also to enhance the model’s adaptability across different crystal property prediction tasks and strengthen its generalization ability. The MoE-inspired module enables dynamic fusion of SE(3) invariant and SO(3) equivariant graph representations by learning task-relevant weighting, allowing the model to adaptively adjust different geometric views for different prediction targets. Overall, these findings provide compelling evidence that MGT constitutes a robust, generalizable and effective framework for crystal material property prediction, with strong potential to support materials discovery. Limitations and future work MGT currently focuses on SE(3) and SO(3) symmetries of static structures and does not yet incorporate the full crystallographic space group, magnetic or time-reversal symmetries, nor finite-temperature or defect-induced disorder. Extending the framework to space-group-equivariant message passing, stochastic symmetry transformations and multi-fidelity active-learning loops will be key next steps49–52. Coupling MGT with symmetry-preserving diffusion or autoregressive models will further enable direct inverse design of crystals with target properties53–55, paving the way toward autonomous, AI-accelerated discovery of next-generation functional materials. In addition, exploring the incorporation of geodesic distance into different graph representations represents a promising direction for future research56,57. Methods Dataset The pretraining dataset is sourced from the Open Quantum Materials Database (OQMD)58, specifically version 1.5, which is released in October 2021. The data within this version had been optimized through density functional theory (DFT) simulations. In cases where multiple structures shared the same cell composition and space group, we carefully selected the structure that exhibited the smallest unit volume per chemical formula following Antunes et al.59. This selection process ultimately yielded a dataset comprising 587483 structures, which are then partitioned into training, validation and testing sets in a ratio of 8:1:1. Element distribution of OQMD was shown in Supplementary Fig. 16. The downstream datasets used in our experiments are Materials Project39 and JARVIS40. The Materials Project, version is 2018.6.1, which contains 69239 structures for the Formation Energy and Band Gap prediction tasks. However, for the Bulk Moduli and Shear Moduli prediction tasks, only 4664 structures are available. JARVIS includes 55722 structures for the Formation Energy, Bandgap (OPT), Total Energy prediction tasks. For the Bandgap (MBJ) and Ehull prediction tasks, JARVIS provides 18171 and 55370 structures, respectively. We evaluated model performance on Materials Project and JARVIS datasets, which involve four and five crystal property prediction tasks, respectively. All downstream datasets used the same training, validation and testing splits and data sizes, following the methodology of Yan et al.11 and Lin et al.12. Element distributions of Materials Project and JARIVS were shown in Supplementary Figs. 17 and 18. The datasets employed for transfer learning include Alloy-GMAE, FG-GMAE and OCD-GMAE, which are derived from the Catalysis Hub60, the ‘functional groups’ (FG) dataset61 and the OC20-Dense dataset62, respectively. Alloy-GMAE dataset comprises the largest number of structures, totaling 11260. FG-GMAE dataset features a medium scale with 3038 structures. OCD-GMAE dataset consists of 973 structures. HOIP dataset, made publicly available by Kim et al.29, includes 1346 HOIP structural files. To ensure fair comparisons across models, we also utilized OC20-LMAE63 as the pretraining dataset for Alloy-GMAE, FG-GMAE and OCD-GMAE as Chen et al.28. OC20-LMAE dataset was filtered from the OC2063, which eventually comprises 363937 structures. The GMAE-related datasets, including the pretraining dataset, underwent rigorous data cleaning and organization as demonstrated by Chen et al.28. Each dataset was also split into training, validation and testing sets with a ratio of 8:1:1. A more detailed display is shown in Supplementary Table 6. MGT MGT is designed to capture the geometry information contained in the two distinct representations of SE(3) and SO(3) graphs. It is a dual-graph encoder architecture that processes node features, edge features and angle features in the graphs, respectively. Specifically, the initial node features are derived from CGCNN embedding features10. For edge features, such as node \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$j$$\end{document}j to node \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i$$\end{document}i, the initial features are obtained using Radial Basis Function (RBF) kernel64 embedding features. The initial angle features are based on RBF kernel embedding with the cosine function employed. The first encoder of MGT consists of the SE(3) graph transformer and the SE(3) projection head layer. As illustrated in Fig. 6a, b, the SE(3) graph transformer includes edge-wise transformer and node-wise transformer layers14 for updating correspondent messages. The edge-wise transformer layer updates edge features with angle information and crystal lattice basis vectors, which can be expressed as:3\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\bf{Q}}}}_{{ij}}^{e}= {{{\rm{f}}}}_{Q}^{e}\left({{{\bf{e}}}}_{{ij}}\right),{{{\bf{K}}}}_{{ij}}^{e}={{{\rm{\phi }}}}_{K}^{e}\left({{{\rm{f}}}}_{K}^{e}\left({{{\bf{e}}}}_{{ij}}\right)\,\left|\left|{{{\rm{f}}}}_{{K}_{m}}^{l}\left({{\mathcal{l}}}_{m}\right)\right|\right|{{{\rm{f}}}}_{E}^{a}\left({{{\boldsymbol{\theta }}}}_{{ij}}\right)\right),{{{\bf{V}}}}_{{ij}}^{e} \\= {{{\rm{\phi }}}}_{V}^{e}\left({{{\rm{f}}}}_{V}^{e}\left({{{\bf{e}}}}_{{ij}}\right){{\rm{||}}}{{{\rm{f}}}}_{{V}_{m}}^{l}\left({{\mathcal{l}}}_{{{\rm{m}}}}\right){{\rm{||}}}{{{\rm{f}}}}_{E}^{a}\left({{{\boldsymbol{\theta }}}}_{{ij}}\right)\right)$$\end{document}Qije=fQeeij,Kije=ϕKefKeeijfKmllmfEaθij,Vije=ϕVefVeeij∣∣fVmllm∣∣fEaθijwhere \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\rm{f}}}}_{Q}$$\end{document}fQ, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\rm{f}}}}_{K}$$\end{document}fK, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\rm{f}}}}_{V}$$\end{document}fV and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\rm{f}}}}_{E}$$\end{document}fE are linear transformation layers, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\rm{\phi }}}}_{K}$$\end{document}ϕK and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\rm{\phi }}}}_{V}$$\end{document}ϕV are nonlinear layers, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\bf{e}}}}_{{ij}}$$\end{document}eij is the edge features, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\boldsymbol{\theta }}}}_{{ij}}$$\end{document}θij is the angle features and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\mathcal{l}}}_{{{\rm{m}}}}$$\end{document}lm (m = 3) are the lattice basis vectors of the unit cell. Then the attention weight computation and message aggregation output can be formulated as:4\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\bf{Output}}}={{\rm{Softplus}}}\left({{{\bf{e}}}}_{{ij}}+{{\rm{BNorm}}}\left({\sum }_{i=1}^{m}{{\upsigma }}\,{{\rm{BNorm}}}\left(\frac{{{{\bf{Q}}}}_{{ij}}{\,{{\bf{K}}}}_{{ij}}}{\sqrt{{{{\rm{d}}}}_{{{{\bf{Q}}}}_{{ij}}}^{k}}}\right)\,{{{\bf{V}}}}_{{ij}}\right)\right)$$\end{document}Output=Softpluseij+BNorm∑i=1mσBNormQijKijdQijkVijwhere \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\upsigma }}$$\end{document}σ denotes the sigmoid function, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\rm{d}}}}_{{{{\bf{Q}}}}_{{ij}}}^{k}$$\end{document}dQijk is the output dimension of each query vector \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\bf{Q}}}}_{{ij}}$$\end{document}Qij, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\rm{Softplus}}}$$\end{document}Softplus denotes the activation function and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\rm{BNorm}}}$$\end{document}BNorm denotes the batch normalization. Then, the node-wise transformer layer aggregates messages from neighboring nodes and edge features, where the edge features are computed from the edge-wise transformer layer, which can be expressed as:5\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\bf{Q}}}}_{i}^{h}= {{{\rm{f}}}}_{Q}^{h}\left({{{\bf{h}}}}_{i}\right),{{{\bf{K}}}}_{{ij}}^{h}={{{\rm{\phi }}}}_{K}\left({{{\rm{f}}}}_{K}^{i}\left({{{\bf{h}}}}_{i}\right){||}{{{\rm{f}}}}_{K}^{j}\left({{{\bf{h}}}}_{j}\right){||}{{{\rm{f}}}}_{E}^{e}\left({{{\bf{e}}}}_{{ij}}\right)\right),{{{\bf{V}}}}_{{ij}}^{h} \\= {{{\rm{\phi }}}}_{V}\left({{{\rm{f}}}}_{V}^{i}\left({{{\bf{h}}}}_{i}\right){||}{{{\rm{f}}}}_{V}^{j}\left({{{\bf{h}}}}_{j}\right){||}{{{\rm{f}}}}_{E}^{e}\left({{{\bf{e}}}}_{{ij}}\right)\right)$$\end{document}Qih=fQhhi,Kijh=ϕKfKihi∣∣fKjhj∣∣fEeeij,Vijh=ϕVfVihi∣∣fVjhj∣∣fEeeij6\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\bf{Output}}}={{\rm{Softplus}}}({{{\bf{h}}}}_{i}+{{\rm{BNorm}}}\left({\sum }_{i=1}^{n}{{\rm{\sigma }}}\,{{\rm{BNorm}}}\left(\frac{{{{\bf{Q}}}}_{i}{\,{{\bf{K}}}}_{{ij}}}{\sqrt{{{{\rm{d}}}}_{{{{\bf{Q}}}}_{i}}^{k}}}\right)\,{{{\bf{V}}}}_{{ij}}\right))$$\end{document}Output=Softplus(hi+BNorm∑i=1nσBNormQiKijdQikVij)where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\rm{h}}}}_{i}$$\end{document}hi is the node features, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\rm{d}}}}_{{{{\rm{Q}}}}_{i}}^{k}$$\end{document}dQik is the output dimension of each query vector \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\bf{Q}}}}_{i}$$\end{document}Qi and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n$$\end{document}n denotes the number of neighboring nodes connected to node \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i$$\end{document}i. The final component is a lightweight SE(3) projection head layer, which can be formulated as:7\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\bf{z}}}={{\bf{z}}}+{{{\rm{LNorm}}}}\left({{\rm{f}}}_{{l}_{2}}\left(\left({{{\rm{f}}}}_{{l}_{1}}{{\bf{z}}}+{{{\bf{b}}}}_{1}\right)+{{{\bf{b}}}}_{2}\right)\right)$$\end{document}z=z+LNormfl2fl1z+b1+b2where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\rm{f}}}}_{l}$$\end{document}fl is the linear layer and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\rm{LNorm}}}$$\end{document}LNorm is the layer normalization.Fig. 6MGT architecture, including multiple different modules.a Edge-wise transformer for updating edge representations through projected edge features, feature concatenation, Hadamard product, batch normalization (BN), nonlinear activation \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma $$\end{document}σ, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\rm{Softplus}}}$$\end{document}Softplus activation and residual addition \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\bigoplus $$\end{document}⨁. b Node-wise transformer for updating node representations through interactions among node-pair features and edge features, followed by normalization, activation and residual update operations. c Equivariant node update module, in which the node representation \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\bf{h}}}}_{j}$$\end{document}hj and aggregated edge representation \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\vec{{{{\bf{e}}}}_{{ij}}}$$\end{document}eij⃗ are combined by tensor product to generate equivariant node updates. d Mix of experts inspired module, in which the SE(3) embedding \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\bf{E}}}}_{{{\rm{SE}}}(3)}$$\end{document}ESE(3) and SO(3) embedding \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\bf{E}}}}_{{{\rm{SO}}}(3)}$$\end{document}ESO(3) are concatenated and adaptively weighted through a self-attention-based router. MGT architecture, including multiple different modules. a Edge-wise transformer for updating edge representations through projected edge features, feature concatenation, Hadamard product, batch normalization (BN), nonlinear activation \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma $$\end{document}σ, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\rm{Softplus}}}$$\end{document}Softplus activation and residual addition \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\bigoplus $$\end{document}⨁. b Node-wise transformer for updating node representations through interactions among node-pair features and edge features, followed by normalization, activation and residual update operations. c Equivariant node update module, in which the node representation \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\bf{h}}}}_{j}$$\end{document}hj and aggregated edge representation \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\vec{{{{\bf{e}}}}_{{ij}}}$$\end{document}eij⃗ are combined by tensor product to generate equivariant node updates. d Mix of experts inspired module, in which the SE(3) embedding \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\bf{E}}}}_{{{\rm{SE}}}(3)}$$\end{document}ESE(3) and SO(3) embedding \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\bf{E}}}}_{{{\rm{SO}}}(3)}$$\end{document}ESO(3) are concatenated and adaptively weighted through a self-attention-based router. To effectively capture the SO(3) equivariant features of crystal graphs, we also adopted an equivariant node update module within the SO(3) graph encoder. As illustrated in Fig. 6c, This module is based on spherical harmonics31 and TP operations following by Geiger et al.32 and Yan et al.14. Specifically, node features are first projected to a scalar representation through linear transformations. Then, two TP layers are applied, each combining node features, edge-wise spherical harmonics and edge features. The TP layer updates the node features by computing a weighted TP of the neighboring node features \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\bf{h}}}}_{j}$$\end{document}hj, the spherical harmonics \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\bf{Y}}}}_{{ij}}$$\end{document}Yij derived from relative position vectors \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\bf{r}}}}_{{ij}}$$\end{document}rij and a learned edge-dependent weight \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\rm{W}}}({{{\bf{e}}}}_{{ij}})$$\end{document}W(eij). Formally, each TP layer is defined as:8\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\bf{h}}}}_{i}^{(l+1)}={{{\rm{Agg}}}}_{j\in {{{\rm N}}}({{\rm{i}}})}\left[{{\rm{TP}}}\left({{{\bf{h}}}}_{j}^{\left(l\right)},{{{\bf{Y}}}}_{{ij}},{{\rm{W}}}\left({{{\bf{e}}}}_{{ij}}\right)\right)\right]+{{{\bf{h}}}}_{i}^{(l)}$$\end{document}hi(l+1)=Aggj∈N(i)TPhjl,Yij,Weij+hi(l)where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\rm{Agg}}}$$\end{document}Agg is an aggregation function, and residual connections were also used to preserve representational depth. Then, the output feature is followed by a \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\rm{Softplus}}}$$\end{document}Softplus activation, batch normalization and a linear projection layer, with another residual connection from the initial input feature \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\bf{h}}}}_{i}^{(0)}$$\end{document}hi(0):9\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\bf{Output}}}={{\rm{Softplus}}}\left({{{\rm{f}}}}_{l}\,{{\rm{Softplus}}}\left({{\rm{BNorm}}}\left({{{\bf{h}}}}_{i}^{\left(2\right)}\right)\right)\right)+{{{\bf{h}}}}_{i}^{(0)}$$\end{document}Output=SoftplusflSoftplusBNormhi2+hi(0) this architecture guarantees that the learned node representations remain equivariant to rotations, effectively capturing both scalar and higher-order geometric interactions through the TP operation. Similar to the SE(3) encoder, we also incorporated a node-wise transformer followed by a lightweight SO(3) projection head layer for the final output. Furthermore, to effectively integrate SE(3) and SO(3) graph embeddings from different graph encoders for downstream tasks, as illustrated in Fig. 6d, we introduce a lightweight MoE-inspired25,26 module consisting of two parallel expert networks and a self-attention-based router that dynamically assigns importance weights to each expert. Given input embeddings \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\bf{E}}}}_{1}$$\end{document}E1 and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\bf{E}}}}_{2}$$\end{document}E2 from the SE(3) and SO(3) graph encoders, respectively, the fused representation is computed as:10\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\bf{H}}}=\left[{{{\rm{f}}}}_{{expert}1}\left({{{\bf{E}}}}_{1}\right),{{{\rm{f}}}}_{{expert}2}\left({{{\bf{E}}}}_{2}\right)\right],{{{\bf{W}}}}_{{router}}={{{\rm{Router}}}}_{{attn}}({{\bf{H}}}),{{\bf{Y}}}={{{\rm{f}}}}_{o}({{\bf{H}}}\,{{{\bf{W}}}}_{{router}})$$\end{document}H=fexpert1E1,fexpert2E2,Wrouter=Routerattn(H),Y=fo(HWrouter)where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\bf{H}}}\in {{\mathbb{R}}}^{B\times N\times d}$$\end{document}H∈RB×N×d, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$B$$\end{document}B is the batch size, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N=2$$\end{document}N=2 is the number of experts, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d$$\end{document}d is the embedding dimension. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\rm{Router}}}}_{{attn}}$$\end{document}Routerattn is the self-attention mechanism18 and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\rm{f}}}}_{o}$$\end{document}fo is the output linear layer. This design enables MGT to adaptively fuse geometry-aware representations based on the relevance of each expert, thereby enhancing its performance across diverse downstream tasks. Pretraining We adopted a multi-objective SSL strategy that simultaneously performs denoising and contrastive learning across SE(3) and SO(3) graph representations. Specifically, for each crystal graph structure, we introduce random noise into geometric attributes, including angle and edge features in SE(3) and SO(3) graphs, respectively. The pretraining framework is trained to predict the perturbed noise directly from the corresponding graph embeddings, which can be expressed as:11\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\mathscr{L}}}}_{{SE}3}={\sum }_{i=1}^{m}\left[{||{\hat{{{\boldsymbol{\varepsilon }}}}}_{i}^{\theta }-({\widetilde{{{\boldsymbol{\theta }}}}}_{i}-{{{\boldsymbol{\theta }}}}_{i})||}^{2}\right]$$\end{document}LSE3=∑i=1m∣∣ε^iθ−(θ~i−θi)∣∣212\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\mathscr{L}}}}_{{SO}3}={\sum }_{i=1}^{m}\left[{||{\hat{{{\boldsymbol{\varepsilon }}}}}_{i}^{e}-({\widetilde{{{\bf{e}}}}}_{i}-{{{\bf{e}}}}_{i})||}^{2}\right]$$\end{document}LSO3=∑i=1m∣∣ε^ie−(e~i−ei)∣∣2where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{{{\boldsymbol{\varepsilon }}}}$$\end{document}ε^ is the noise predicted by the model, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\boldsymbol{\theta }}}$$\end{document}θ and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\bf{e}}}$$\end{document}e are the original attributes, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$m$$\end{document}m is the batch size, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\widetilde{{{\boldsymbol{\theta }}}}$$\end{document}θ~ and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\widetilde{{{\bf{e}}}}$$\end{document}e~ are the modified angle and edge features obtained by adding noise \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\boldsymbol{\varepsilon }}}$$\end{document}ε, which is sampled from a Gaussian distribution. In parallel, we empowered the model to capture shared geometric semantics through maximizing the mutual information between the SE(3) and SO(3) embeddings. This is achieved by normalizing the temperature-scaled cross-entropy (NT-Xent) loss65:13\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\mathscr{L}}}}_{{constrast}}=-\frac{1}{2N}{\sum }_{k=1}^{N}\left[\log \frac{{{{\bf{e}}}}^{{{\rm{sim}}}({{{\bf{z}}}}_{i},{{{\bf{z}}}}_{j})/\tau }}{{\sum }_{j=1,j\ne i}^{2N}{{{\bf{e}}}}^{{{\rm{sim}}}({{{\bf{z}}}}_{i},{{{\bf{z}}}}_{j})/\tau }}+\log \frac{{{{\bf{e}}}}^{{{\rm{sim}}}({{{\bf{z}}}}_{j},{{{\bf{z}}}}_{i})/\tau }}{{\sum }_{j=1,j\ne i+N}^{2N}{{{\bf{e}}}}^{{{\rm{sim}}}({{{\bf{z}}}}_{j},{{{\bf{z}}}}_{i})/\tau }}\right]$$\end{document}Lconstrast=−12N∑k=1Nlogesim(zi,zj)/τ∑j=1,j≠i2Nesim(zi,zj)/τ+logesim(zj,zi)/τ∑j=1,j≠i+N2Nesim(zj,zi)/τwhere \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N$$\end{document}N is the batch size, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\bf{z}}}}_{i}$$\end{document}zi and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\bf{z}}}}_{j}$$\end{document}zj represent the graph embeddings from the SE(3) and SO(3), respectively. The cosine similarity between these two embeddings is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\rm{sim}}}\left({{{\bf{z}}}}_{i},{{{\bf{z}}}}_{j}\right)=\frac{{{{\bf{z}}}}_{i},{{{\bf{z}}}}_{j}}{\left|\left|{{{\bf{z}}}}_{i}\right|\right|{||}{{{\bf{z}}}}_{j}{||}}$$\end{document}simzi,zj=zi,zjzi∣∣zj∣∣. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau $$\end{document}τ is the temperature parameter, which scales the similarity values and controls the smoothness of the distribution. The pretraining objective integrates these components with balancing coefficients \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\lambda }_{1}$$\end{document}λ1, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\lambda }_{2}$$\end{document}λ2 and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\lambda }_{3}$$\end{document}λ3:14\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\mathscr{L}}}}_{{total}}={\lambda }_{1}{{{\mathscr{L}}}}_{{constrast}}+{\lambda }_{2}{{{\mathscr{L}}}}_{{SE}3}+{\lambda }_{3}{{{\mathscr{L}}}}_{{SO}3}$$\end{document}Ltotal=λ1Lconstrast+λ2LSE3+λ3LSO3 We employed the AdamW66 optimizer with an initial learning rate of 10⁻5 for model pretraining with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\lambda }_{1}=1.0$$\end{document}λ1=1.0, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\lambda }_{2}=0.5$$\end{document}λ2=0.5 and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\lambda }_{3}=0.5$$\end{document}λ3=0.5. The learning rate was scheduled using cosine annealing with a predefined minimum value, gradually decreasing the learning rate over training epochs. In addition, to enable that MGT can capture meaningful spatial structural information while avoiding excessive distribution shifts between clean and noisy conformations, the noise scale was set to 0.15. The warmup step was set to 10. The batch size was set to 128 and training epoch was set to 200. More detailed model pretraining settings are provided in Supplementary Table 7. Fine-tuning All downstream tasks, including transfer learning settings, were trained using the mean squared error (MSE) loss, defined as:15\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\mathscr{L}}}}_{{fine}-{tune}}=\frac{1}{n}{\sum }_{i=1}^{n}{({{{\bf{y}}}}_{i}-{\hat{{{\bf{y}}}}}_{i})}^{2}$$\end{document}Lfine−tune=1n∑i=1n(yi−y^i)2where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n$$\end{document}n is the batch size, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\bf{y}}}}_{i}$$\end{document}yi and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\hat{{{\bf{y}}}}}_{i}$$\end{document}y^i denote the ground truth and predicted values of the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i$$\end{document}i-th sample, respectively. We employed the AdamW optimizer with an initial learning rate of 0.0005 for downstream tasks and 0.0006 for transfer learning. A cosine annealing scheduler was applied to gradually decay the learning rate throughout training. All models were trained for 500 epochs, with a batch size of 16 for downstream tasks and 8 for transfer learning tasks. More detailed fine-tuning settings are provided in Supplementary Table 8. Implementation The entire model pretraining and fine-tuning process was implemented using the open-source frameworks PyTorch67 and PyTorch Geometric68. The data processing was conducted using the FairChem69, ASE70, pymatgen71 and JARVIS-Tools40 packages. All experiments were conducted on a 64-bit CentOS v8.0 server equipped with eight NVIDIA RTX 4090 GPUs and 512 GB of RAM. Supplementary information Supplementary Information Transparent Peer Review file Supplementary Information Transparent Peer Review file