LeMat-Bulk: aggregating, and de-duplicating quantum chemistry materials databases

Martin Siron, Inel Djafar, Ali Ramlaoui, Etienne du Fayette, Amandine Rossello, Edvin Fako, Matthew McDermott, Felix Therrien, Luis Barroso-Luque, Flaviu Cipcigan, Philippe Schwaller, Thomas Wolf, Alexandre Duval · 2025

Read LeMat-Bulk: aggregating, and de-duplicating quantum chemistry materials databases by Martin Siron, Inel Djafar, Ali Ramlaoui, Etienne du Fayette, Amandine Rossello, Edvin Fako, Matthew McDermott, Felix Therrien, Luis Barroso-Luque, Flaviu Cipcigan, Philippe Schwaller, Thomas Wolf, Alexandre Duval (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-05178.txt.

Introduction

Martin Siron and co-authors present LeMat-Bulk, a merged data set that joins three big quantum chemistry stores for materials. Fast growth in such stores aids machine-led discovery, but overlap and clashing formats make joint use hard.

LeMat-Bulk spans more than five million materials computed with one main method, plus the largest sets for two other methods. The team set one standard form and coined a hashing trick, the Bonding Algorithm Weisfeiller-Lehman, to spot copies. Tests of that hash check how well it keeps true matches while it drops exact repeats.

Catalog record

CREATORMartin Siron, Inel Djafar, Ali Ramlaoui, Etienne du Fayette, Amandine Rossello, Edvin Fako, Matthew McDermott, Felix Therrien, Luis Barroso-Luque, Flaviu Cipcigan, Philippe Schwaller, Thomas Wolf, Alexandre Duval
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE76 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-05178

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LeMat-Bulk: aggregating, and de-duplicating quantum chemistry materials databases Martin Siron, Inel Djafar, Ali Ramlaoui, Etienne du Fayette, Amandine Rossello, Edvin Fako, Matthew McDermott, Felix Therrien, Luis Barroso-Luque, Flaviu Cipcigan, Philippe Schwaller, Thomas Wolf, Alexandre Duval Abstract The rapid expansion of materials science databases has driven machine learning-based discovery while also posing challenges in data integration, duplication, and interoperability. Robust standardization and de-duplication methods are needed to address these issues and streamline materials research. We present LeMat-Bulk , a unified dataset combining Materials Project, OQMD, and Alexandria, encompassing over 5.3 million PBE-calculated materials and also representing the largest collection of PBESol and SCAN functional calculations. Our methodology standardizes calculations across databases that utilize different parameters, effectively addressing redundancy and enhancing cross-compatibility. To de-duplicate, we propose a hashing function which we termed the Bonding Algorithm Weisfeiller-Lehman (BAWL). We comprehensively benchmark this fingerprint under atomic noise, lattice strain, and symmetry transformations, demonstrating that it outperforms existing fingerprinting techniques such as SLICES, and CLOUD in robustness while offering greater computational efficiency than similarity-based approaches such as Pymatgen’s StructureMatcher. Additionally, the fingerprint facilitates the analysis of functional-dependent trends (PBE, PBESol, SCAN) offering a scalable framework for data-driven materials science. 1]Entalpic, Paris, France 2]LIAC, EPFL, Lausanne, Switzerland 3]NCCR, EPFL, Lausanne, Switzerland 4]Newfound Materials, Houston, TX, USA 5]MILA, Quebec, Canada 6]FAIR at Meta, Menlo Park, CA, USA 7]IBM, Daresbury, UK 8]Huggingface, Brooklyn, NY, USA \contribution [*]Corresponding authors 1 Introduction Recent advancements in generative materials science aim to accelerate the discovery of novel materials with target properties by leveraging deep learning models trained on atomic-scale data [ 1 , 2 , 3 , 4 , 5 ] . These AI-driven approaches offer the potential to reshape the design process by generating entirely new material structures with tailored functionalities [ 6 ] . In contrast to traditional heuristics and rules, generative models can explore vast design spaces and reduce human bias, providing a promising path for data-driven materials discovery and inverse design. However, the performance and reliability of these models depend heavily on access to high-quality and well-structured datasets, as well as on robust methods for evaluating materials novelty – both of which remain open challenges. Over the past decades, computational chemistry and in-silico modeling have played an essential role in advancing materials science. Density Functional Theory (DFT) [ 7 ] has enabled the replication and rationalization of known material behaviors, marking the transition to the third paradigm of materials science [ 8 , 9 ] . This was followed by the emergence of large-scale, high-throughput computational databases such as the Materials Project [ 10 , 11 ] , OQMD [ 12 ] , and Alexandria [ 13 ] . These initiatives have shifted the field toward data-centric workflows – an evolution sometimes described as the fourth paradigm of materials science [ 14 ] – where data generation, curation, and reuse are foundational to discovery. While Machine Learning (ML) plays an increasingly important role in this paradigm, it builds on the infrastructure of large, structured datasets derived from ab initio simulations. However, even with these resources, major limitations persist due to inconsistent and inhomogeneous data. A central issue is the lack of interoperability between major quantum chemistry datasets, which prevents efficient integration and utilization. Each repository varies in calculation parameters, computational features, and data scope, which complicates cross-referencing and integration. These discrepancies can lead to inconsistencies in key thermodynamic and structural properties, which complicates the construction of ML-ready datasets and impairs the ability to draw global insights [ 15 ] . Moreover, duplicate entries — arising from multiple calculations of the same structure — are often not linked or identified, further reducing the efficiency of data usage. For ML researchers, this lack of harmonization presents a barrier to benchmarking and evaluating whether generated structures are truly novel or simply rediscovered. It also creates hurdles to access and unify data across multiple sources, often requiring expertise in multiple database interfaces. Standardization efforts like the Optimade protocol [ 16 , 17 ] have attempted to address this issue by enabling federated querying across databases. Yet, such efforts remain limited in both scope and adoption. For instance, the Materials Project’s Optimade endpoint currently focuses on a curated subset of properties [ 18 ] , while broader coverage is typically delivered through the dedicated Materials Project API. Additionally, endpoint providers often label similar properties with different names. This highlights limitations in the federated approach compared to a fully unified dataset strategy, as pursued in this study. Efforts to reconcile discrepancies in calculation methodologies led to the Alexandria database, integrating structures from sources such as Materials Project (MP) and AFLOW [ 19 ] to enhance interoperability and standardize parameters. Similarly, OQMD incorporates Materials Project data [ 20 ] . However, these efforts lack transparency regarding data provenance (e.g., in Alexandria, material origins—MP, AFLOW, or prototype generation—are unclear), limiting broader applicability. For example, we cannot compare an Alexandria material taken from MP with its MP-computed counterpart. Besides, these initiatives often duplicate rather than consolidate computations. As a result, despite increasing data availability, standardized, interoperable, and openly accessible databases remain scarce. Beyond data integration, a key yet unresolved challenge in generative materials discovery is defining material novelty in a computationally rigorous and scalable way. Current high-throughput pipelines frequently generate duplicate structures, yet there are no universally accepted benchmarks for determining whether generated materials are genuinely novel or already exist in databases. Existing methods for structural comparison, including widely used approaches such as Pymatgen’s Structure Matcher [ 21 ] , as well as similarity metrics proposed, such as Widdowson et al.’s Average Minimum Distance [ 22 , 23 ] , and Yang et al. [ 24 ] , Bartók et al. [ 25 ] , De et al. [ 26 ] , Zhu et al. [ 27 ] , Oganov et al. [ 28 ] , and Therrien et al. [ 29 ] . However these metrics require pairwise structure comparisons across large datasets, resulting in computational inefficiencies at scale. A more scalable alternative is to use hashing or fingerprinting methods to represent structures as compact strings, enabling efficient direct comparisons rather than computing vector-based similarities with thresholds. Recent hashing approaches include SLICES [ 30 ] , CLOUD [ 31 ] , and a graph-based hashing method proposed by Ongari et al. [ 32 ] . These methods are each discussed in detail in the Supplementary Information (SI). While Ongari’s method showed promise for identifying similar Metal-Organic Framework (MOF) structures, it has not yet been systematically evaluated on large inorganic crystalline materials databases, nor have CLOUD and SLICES. Moreover, there is currently no comprehensive benchmarking assessing the robustness or sensitivity of these hashing and similarity scoring methods under structural perturbations or symmetry transformations, nor have these methods been directly compared. This methodological gap has led to growing criticisms within the generative materials domain, particularly regarding the treatment of disordered materials [ 5 , 33 , 34 ] . While materials science has traditionally defined materials based on their chemical composition, atomic structure, properties and applications the emergence of generative models has blurred these boundaries, further complicating how novelty should be framed and evaluated. To address these issues, we introduce LeMat-Bulk , a large-scale unified dataset that consolidates

Read the complete text