Beyond Atomic Geometry Representations in Materials Science: A Human-in-the-Loop Multimodal Framework

Can Polat, Hasan Kurban, Erchin Serpedin, Mustafa Kurban · 2025

Read Beyond Atomic Geometry Representations in Materials Science: A Human-in-the-Loop Multimodal Framework by Can Polat, Hasan Kurban, Erchin Serpedin, Mustafa Kurban (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00302.txt.

Introduction

Can Polat and co-authors build a richer dataset for materials science that goes beyond atomic positions in a 2025 paper. Most crystal datasets hold only geometry files, which limits modern multimodal learning. Their framework adds two-dimensional projections and structured text notes, such as lattice measures and coordination data, to each structure.

Experts in the loop helped keep the annotations accurate and uniform. The dataset supports two tasks: predicting properties and summaries from mixed inputs, and generating new crystals under partial supervision. Tests with leading language and vision models exposed large gaps between input types and showed that annotation quality drives generalization. The data and code are shared openly for benchmarking.

Catalog record

CREATORCan Polat, Hasan Kurban, Erchin Serpedin, Mustafa Kurban
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-00302

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond Atomic Geometry Representations in Materials Science: A Human-in-the-Loop Multimodal Framework Can Polat, Hasan Kurban, Erchin Serpedin, Mustafa Kurban Abstract Most materials science datasets are limited to atomic geometries (e.g., XYZ files), restricting their utility for multimodal learning and comprehensive data-centric analysis. These constraints have historically impeded the adoption of advanced machine learning techniques in the field. This work introduces MultiCrystalSpectrumSet (MCS-Set), a curated framework that expands materials datasets by integrating atomic structures with 2D projections and structured textual annotations, including lattice parameters and coordination metrics. MCS-Set enables two key tasks: (1) multimodal property and summary prediction, and (2) constrained crystal generation with partial cluster supervision. Leveraging a human-in-the-loop pipeline, MCS-Set combines domain expertise with standardized descriptors for high-quality annotation. Evaluations using state-of-the-art language and vision-language models reveal substantial modality-specific performance gaps and highlight the importance of annotation quality for generalization. MCS-Set offers a foundation for benchmarking multimodal models, advancing annotation practices, and promoting accessible, versatile materials science datasets. The dataset and implementations are available at https://github.com/KurbanIntelligenceLab/MultiCrystalSpectrumSet . 1 Introduction Materials science drives innovation in energy storage, catalysis, and microelectronics, yet progress remains limited by the challenge of mapping atomic geometry to material function ( Shen et al. 2022 ; Choudhary et al. 2022 ; Huang et al. 2024 ; Jaafreh et al. 2022 ) . While first-principles methods such as density functional theory (DFT) offer accurate predictions of ground-state properties, their performance depends heavily on high-quality initial structures and exhaustive sampling of the configurational landscape—constraints that scale poorly with system complexity ( Orio et al. 2009 ; Cohen et al. 2012 ) . Semi-empirical surrogates like density functional tight binding (DFTB) reduce computational cost but still require extensive geometry optimization, sustaining a “structure bottleneck” that hinders high-throughput discovery ( Hourahine et al. 2007 ) . Recent advances in machine learning have introduced data-driven surrogates, including graph neural networks (GNNs), diffusion models, and large language models (LLMs), to predict energies or synthesize plausible crystal configurations ( Xie et al. 2021 ; Jiao et al. 2023 ; Antunes et al. 2024 ; Li et al. 2024 ; Hessmann et al. 2025 ) . These methods have been enabled by curated datasets such as Perov-5 ( Castelli et al. 2012 ) , Carbon-24 ( Pickard & Needs 2006 ; Pickard & Needs 2011 ) , and MP-20 ( Jain et al. 2013 ) . However, two systemic issues persist. First, heterogeneous data curation introduces sampling bias and spurious correlations that limit model robustness and out-of-distribution performance ( Davariashtiyani et al. 2024 ) . Second, most existing benchmarks are restricted to atomic coordinate data, omitting the multimodal context—such as visual projections and textual descriptors—that human crystallographers routinely use to reason about structure. MultiCrystalSpectrumSet (MCS-Set) is a human-in-the-loop, multimodal benchmark for evaluating structure–property relationships in materials science. It consists of atomic clusters of Ag, Au, PbS, and ZnO, spanning R6–R10 (0.6 nm to 1 nm) geometries and ranging from 55 to 351 atoms. Each cluster is augmented with 780 unique 3D rotations, producing over 15,600 triplets that align XYZ coordinate files with high-resolution (512 \times 512) orthographic projections and structured textual annotations. These annotations include lattice parameters, unit cell volume, density, and nearest-neighbor statistics. Annotation quality is maintained through an expert-in-the-loop process that combines automated structural descriptors with manual review. MCS-Set supports two core research directions: (1) multimodal property and summary prediction , leveraging integrated structural and visual inputs, and (2) constrained crystal generation under partial cluster supervision, such as extrapolating R9 structures from R6, R7, R8, and R10 data. The benchmark enables cross-modal evaluation using both geometry-aware and language-based metrics, and facilitates analysis of annotation fidelity, rotational augmentation effectiveness, and modality-specific generalization. By unifying structured, visual, and textual modalities, MCS-Set promotes data-centric practices in materials science and contributes to reproducible, generalizable benchmarks for scientific machine learning. The rest of the paper is organized as follows. Section 2 presents background and related work. Section 3 describes the data generation process. Section 4 outlines the tasks, implementation, and experiments. Section 5 discusses limitations, and Section 6 marks the conclusion. Figure 1: Overview of the MultiCrystalSpectrumSet (MCS-Set) dataset. Top Left: Atomic clusters of Au, Ag, PbS, and ZnO spanning R6–R10 geometries. Top Right: Structural property distributions across materials. ( A ) Normalized average nearest-neighbor distances. ( B ) Radial distribution functions highlighting pairwise atomic distances. ( C ) Coordination number distributions. ( D ) Atomic count versus bounding box volume. Bottom: Representative triplets from each modality—XYZ atomic coordinates, high-resolution 2D projections, and structured textual annotations—demonstrating the dataset’s multimodal alignment. The figure illustrates both the diversity of crystal structures and the design of MCS-Set to support multimodal learning and benchmarking. 2 Background This section reviews computational techniques and data resources relevant to crystal structure modeling. Traditional approaches rely on DFT and semi-empirical search methods, while recent generative models learn structural distributions directly from data. Machine learning methods—ranging from convolutional networks to transformers and graph neural networks—encode varying degrees of structural inductive bias suited to materials domains. Public benchmarks have accelerated progress in property prediction and structure generation, yet most remain limited to raw atomic coordinate tables, lacking aligned multimodal data and transparent curation. This constraint hinders comprehensive data-centric evaluation and the development of generalizable multimodal models. These limitations highlight the need for datasets that integrate multiple modalities and embed human expertise into the annotation process—motivating the design and expert-guided construction of MCS-Set. 2.1 Crystal-Structure Generation Approaches Material properties—mechanical, optical, and electronic—are inherently determined by crystal geometry ( Bhadeshia 2001 ) . Ab-inito methods such as DFT remain the gold standard for predicting ground-state structures, but their effectiveness depends on high-quality initial coordinates and exhaustive sampling of configurational space. Semi-empirical alternatives, including DFTB, offer improved efficiency and have been extended to broader chemical regimes, including 5f elements. To overcome scalability bottlenecks, global optimization strategies have been widely adopted, including simulated annealing ( Bertsimas & Tsitsiklis 1993 ) , evolutionary algorithms ( Bartz-Beielstein et al. 2014 ) , basin hopping, and data-mining heuristics ( Huber et al. 2020 ) . These methods provide diverse candidate structures but require careful parameterization and substantial computational effort. 2.2 Machine Learning in Materials Science Machine learning has become integral to modern materials discovery pipelines. Convolutional networks ( Mishra & Rahul 2023 ; Zheng et al. 2018 ; Polat et al. 2025a ) , transformers ( Rane 2023 ; Liao et al. 2023 ) , GNNs ( Schütt et al. 2017 ; Gasteiger et al. 2020 ; Du et al. 2024 ; Duval et al. 2023 ) , and multimodal encoders ( Polat et al. 2024 ) have been applied to predict formation energies, band gaps, and elastic tensors with near-DFT accuracy. Recent advances in generative modeling are revolutionizing crystal structure generation by introducing data-driven surrogates. Models like CDVAE ( Xie et al. 2021 ) , DiffCSP ( Jiao et al. 2023 ) , and EH-Diff ( Liu et al. 2025 ) (an equivariant hypergraph diffusion framework) can explore configurational space significantly faster than traditional search methods. Despite this speed, these approaches are very sensitive to dataset quality, symmetry preservation, and modality alignment. This highlights the need for carefully curated, structurally consistent training data to achieve generalizable crystal generation. Beyond single modalities, generative frameworks

Read the complete text