Atompack: A Storage and Distribution Layer for Read-Heavy Atomistic ML Training Datasets
Ali Ramlaoui, Daniel T. Speckhard, Sagar Pal, Fragkiskos D. Malliaros, Alexandre Duval, Victor Schmidt · 2026
Read Atompack: A Storage and Distribution Layer for Read-Heavy Atomistic ML Training Datasets by Ali Ramlaoui, Daniel T. Speckhard, Sagar Pal, Fragkiskos D. Malliaros, Alexandre Duval, Victor Schmidt (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29975.txt.
Introduction
Ramlaoui and five colleagues present Atompack, a storage and distribution layer for atomistic machine learning datasets, the large collections of molecular records used to train models of materials and molecules. Training reads these immutable snapshots again and again in shuffled order.
Atompack is built for that workload: records are appended efficiently during construction, then an immutable index is committed and records are served whole through a memory-mapped read path. Against HDF5, LMDB, and ASE baselines, benchmarks cover sequential and shuffled reads, shared filesystems, write speed, and artifact size too. On a representative 64-atom workload, shuffled training-style reads run 96 times faster than the ASE LMDB baseline, with artifacts about 79 percent smaller.
Catalog record
| CREATOR | Ali Ramlaoui, Daniel T. Speckhard, Sagar Pal, Fragkiskos D. Malliaros, Alexandre Duval, Victor Schmidt |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.29975
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Atompack: A Storage and Distribution Layer for Read-Heavy Atomistic ML Training Datasets Ali Ramlaoui, Daniel T. Speckhard, Sagar Pal, Fragkiskos D. Malliaros, Alexandre Duval, Victor Schmidt Abstract Atomistic machine learning datasets are increasingly used for training: large immutable snapshots are read repeatedly, shuffled across epochs, staged across clusters’ storage systems, and republished as reusable scientific artifacts. This workload differs from interactive scientific curation, where mutable records and ad hoc inspection are often more important than random indexed throughput. We present Atompack, an append-oriented storage format and distribution layer designed around a simple workload: training pipelines usually consume complete molecular records, while the order of records is randomized by the learning algorithm. Atompack appends records efficiently during dataset construction, then commits an immutable index and serves records through a memory-mapped read path optimized for training. We compare Atompack with HDF5, LMDB, and ASE baselines representing array stores, key-value records, serialized records, and object-oriented databases. The benchmarks measure sequential reads, shuffled reads, shared-filesystem behavior, write throughput, and artifact size. On a representative 64-atom workload, Atompack is 96x faster than ASE LMDB on shuffled training-style reads while producing artifacts about 79% smaller. The results indicate that serving complete molecule records, rather than field chunks or reconstructed objects, improves shuffled training throughput while keeping artifacts compact enough for public distribution. 1]Entalpic, Paris, France 2]Université Paris-Saclay, CentraleSupélec, Inria, Gif-sur-Yvette, France \contribution [*]Corresponding author \metadata [Correspondence] 1 Introduction Atomistic machine learning (ML) datasets are no longer only small collections of structures loaded for interactive analysis or small-scale training. Modern materials and molecular ML pipelines use millions to hundreds of millions of structures to train foundation interatomic potentials and related models [ 10 , 3 , 12 , 7 , 19 , 2 , 15 , 21 , 8 , 18 ] . In that setting, the storage layer becomes part of the training system: records are repeatedly sampled in shuffled order, opened by multiple worker processes, and transferred between local and shared filesystems in high performance computing (HPC) environments. The same format also affects distribution: a published dataset should be easy for downstream users to download, reopen, and train from without reconstructing a project-specific database. The access pattern is structured but not always reflected in storage choices. A typical training objective has the form: \min_{\theta}J(\theta)=\min_{\theta}\frac{1}{N}\sum_{i=1}^{N}\ell(f_{\theta}(x_{i}),y_{i}), where \ell is a per-sample loss, x_{i} is a molecular system or crystal structure input that contains information about the elements present, the geometry, and other useful physical descriptors. x_{i} is typically encoded in a graph-based structure. Similarly, y_{i} represents one or more targets such as energy, forces, stress, or charges. Stochastic training evaluates this sum through minibatches of graphs, using a static or dynamic batching algorithm, where the graphs are sampled with a permutation or near-permutation of the ids, \{1,\ldots,N\} [ 22 ] . Thus, the storage system is used to retrieve many fields describing the molecule/crystal \pi(i) , not for one column over all molecules/crystals. This distinction matters because array stores perform best when nearby accesses reuse the same fields or chunks. This workload differs in scope with respect to several common storage tools. The Atomic Simulation Environment (ASE) is a central interchange layer for atomistic workflows [ 14 ] , designed for object-level manipulation rather than high-throughput indexed reads. HDF5 provides compact scientific arrays and a mature ecosystem [ 23 ] , strongest when access exploits array and chunk locality. LMDB provides an embedded key-value store with memory-mapped access and is widely used in training pipelines [ 5 ] , but the record payload format is left to each pipeline. Atompack targets a more specific problem: serving immutable atomistic datasets efficiently during training. Atompack stores the unit consumed by training: a complete molecule or crystal structure. Its lifecycle is append during construction, commit an immutable index, distribute, and reopen read-only for training. Atompack therefore uses append-oriented construction, a committed index for O(1) record lookup, memory-mapped read-only serving, and repository-friendly shard directories. These choices are conservative: they trade away arbitrary updates and field projection to optimize the access pattern that dominates large-scale training. We make three contributions: an Atompack file/shard design for immutable atomistic records; a comparison with HDF5, LMDB, and ASE-backed baselines across read, write, storage-size, and filesystem-sensitive workloads; and an open implementation with benchmark scripts and packaged datasets. Figure 1 : Storage access pattern for shuffled atomistic ML training. The left panel illustrates a field-wise chunked layout like HDF5: species, positions, energies, and other fields are stored as separate datasets, and the highlighted cells mark the same shuffled samples across those datasets. Atompack instead stores each molecule or structure as one indexed record, so the same shuffled sampler performs one direct record lookup per sample and returns the complete training input. 2 Background and Related Work 2.1 Atomistic Data Infrastructure Atomistic simulation software has long emphasized interoperability and rich structure objects. ASE is a representative example: it provides a high-level interface for building, manipulating, simulating, and storing atomic structures [ 14 ] . This object model is useful for curation and simulation integration. In stochastic ML training, however, the model consumes typed tensors, and repeated object materialization can cost as much as the storage access. Atompack keeps ASE-compatible ingestion paths, but serves training data as indexed tensor payloads. Large public atomistic datasets have also changed expectations. The Open Catalyst 2020 dataset and related challenges demonstrated the value of large-scale open simulation data for ML-driven chemistry [ 4 ] . Materials datasets such as MPtrj [ 6 ] and OMat24 [ 2 ] further illustrate the growth of reusable potential-energy-surface data for training universal ML interatomic potentials [ 20 , 11 , 13 ] . These datasets are valuable only if researchers can fetch, reopen, and iterate over them efficiently. For these datasets, the published storage layout affects reproducibility and downstream reuse. 2.2 Scientific Storage and Training Stores HDF5 is a standard choice for scientific arrays and is well suited to structured, typed datasets with chunked access [ 23 ] . Zarr and related formats extend this array-store lineage toward object and cloud storage [ 16 ] . These formats are strong choices when the workload can exploit field-wise or chunk-wise locality. Atomistic ML training often asks a different question: retrieve all fields for molecule i , then molecule j , under a shuffled index stream. A structure-of-arrays layout minimizes storage overhead and enables efficient scans, but a shuffled complete-record workload can turn each sample into several independent chunk lookups and reconstruction steps. LMDB is widely used in ML pipelines because it provides direct key-value lookup through an embedded memory-mapped store [ 5 ] . In practice, however, the record payload is project-specific. A packed binary payload can be compact but requires custom schema code; a pickled dictionary is easier to build but pays serialization and object reconstruction costs. General ML frameworks such as PyTorch provide data-loading abstractions [ 17 ] , but they intentionally leave storage layout to the dataset author. Atompack instead uses one structure per indexed record and a domain-specific schema for atomistic descriptors/targets. 3 System Design Figure 1 shows the intended lifecycle. Atompack targets the transition from dataset construction to repeated training access. Before publication, data are generated and appended; after publication, the dominant