Icicle: Scalable Metadata Indexing and Real-Time Monitoring for HPC File Systems
Haochen Pan *† , Ryan Chard † , Song Young Oh * , Maxime Gonthier *† , Valérie Hayot-Sasson ‡ , Geoffrey Lentner § , Joe Bottigliero * , Rachana Ananthakrishnan *† , Kyle Chard *† , Ian Foster †* · 2026
Read Icicle: Scalable Metadata Indexing and Real-Time Monitoring for HPC File Systems by Haochen Pan *† , Ryan Chard † , Song Young Oh * , Maxime Gonthier *† , Valérie Hayot-Sasson ‡ , Geoffrey Lentner § , Joe Bottigliero * , Rachana Ananthakrishnan *† , Kyle Chard *† , Ian Foster †* (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-10295.txt.
Introduction
Pan and nine colleagues present Icicle, a framework for continuous file system metadata indexing and monitoring. Traditional tools like find and du fail at billions of files, while external indexers stay batch-oriented. Icicle keeps a unified, current view of file system state, with snapshot and event ingestion from Lustre and Storage Scale. Built on Kafka and Flink, it ingests events into two indexes for file discovery and summaries by user, group, and directory. Tests on production-scale data show large throughput gains, with tunable consistency, latency, and freshness.
Catalog record
| CREATOR | Haochen Pan *† , Ryan Chard † , Song Young Oh * , Maxime Gonthier *† , Valérie Hayot-Sasson ‡ , Geoffrey Lentner § , Joe Bottigliero * , Rachana Ananthakrishnan *† , Kyle Chard *† , Ian Foster †* |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57744 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.10295
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Icicle: Scalable Metadata Indexing and Real-Time Monitoring for HPC File Systems Haochen Pan *† , Ryan Chard † , Song Young Oh * , Maxime Gonthier *† , Valérie Hayot-Sasson ‡ , Geoffrey Lentner § , Joe Bottigliero * , Rachana Ananthakrishnan *† , Kyle Chard *† , Ian Foster †* Abstract Modern HPC file systems can contain billions of files and hundreds of petabytes of data, making even simple questions increasingly intractable to answer. Traditional file system utilities such as find and du fail to scale to these sizes. While external indexing tools like GUFI and Brindexer improve query performance, they remain batch-oriented and unsuitable for heterogeneous, rapidly evolving environments. We present Icicle, a scalable framework for continuous file system metadata indexing and monitoring. Icicle maintains a unified, up-to-date, and queryable view of file system state while supporting both periodic snapshot-based ingestion for bulk metadata updates and event-based ingestion for real-time synchronization from production systems such as Lustre and IBM Storage Scale. Built on Apache Kafka and Apache Flink, Icicle provides high-throughput, fault-tolerant, and horizontally scalable ingestion of metadata events into two complementary search indexes, enabling both individual file discovery and aggregate summary statistics by user, group, and directory. This architecture enables efficient support for both coarse-grained administrative queries and interactive analytics over billions of objects. Our experimental evaluation on production-scale HPC datasets demonstrates order-of-magnitude throughput improvements over existing monitoring and indexing approaches, with tunable options for balancing consistency, latency, and metadata freshness. I Introduction High performance computing (HPC) file systems can store tens of billions of files and hundreds of petabytes of data, driven by large-scale scientific experiments, extreme-scale simulations, and AI-driven workflows. For example, Oak Ridge National Laboratory (ORNL) hosts a 700 PB Lustre file system, Argonne National Laboratory has two 100 PB Lustre file systems, and the National Energy Research Scientific Computing Center (NERSC) maintains a 650 PB High Performance Storage System (HPSS) tape archive [ 4 , 21 , 14 , 30 ] . Massive datasets produced routinely by scientific applications must be reliably stored and managed across tiered and heterogeneous storage systems, each with independent namespaces and quotas, and distinct lifecycle policies, interfaces, and representations. While this hierarchical design improves performance and optimizes data placement, it also fragments metadata across multiple systems, making user and administrative tasks increasingly complex and time-consuming. Unfortunately, traditional utility tools such as find and du fail to scale to modern HPC file systems. Already in 2010, ORNL reported that a single find or du traversal on their 10 PB Spider file system could take more than 48 hours to complete [ 29 ] . As file systems continue to grow beyond petabyte scales, administrators have turned to external metadata indexing systems to improve observability and conduct data lifecycle management. Modern solutions such as Grand Unified File Indexer (GUFI) [ 27 ] and Brindexer [ 37 ] construct external, database-backed indexes of file system metadata, enabling faster query performance than in-place scans. However, these systems remain fundamentally batch-oriented: they rely on periodic scans to update their databases and require parallel scripts to perform queries and analysis. As a result, obtaining even basic statistics such as per-user summaries or top- k usage reports remains slow, resource-intensive, and operationally cumbersome. Modern HPC file systems such as Lustre and IBM Storage Scale (GPFS) [ 40 ] provide real-time metadata event feeds, yet few indexing systems exploit these capabilities to enable both real-time event monitoring and cost-aware, periodic indexing. At the same time, metadata consumers have widely varying freshness requirements: anomaly detection may need near-real-time updates, while usage trend analysis may tolerate daily snapshots. Existing approaches, while effective for file counting or hierarchical aggregation, are limited in their ability to support semantic or percentile-based queries. Examples include identifying which users created the most small files last week or calculating the 99th percentile of directory sizes per group. Executing such queries across a large collection of databases remains inefficient with current monitoring systems. Table I lists representative queries that motivate these needs, spanning individual file discovery, anomaly detection, and aggregate usage analysis. TABLE I : Representative queries for file system metadata management. Role Example Query Query Expression Granularity General Where is a specific file or directory? name LIKE "*pattern*" Individual General Which files or directories have world-writable permissions? mode = 777 Individual General Which files have not been accessed in over 12 months? atime < now() - 1y Individual General Which large files have low access frequency? size > 100GB AND atime < now() - 6m Individual General Which files are replicated across directories? GROUP BY checksum HAVING count > 1 Individual General Which directories contain more than 100,000 files? file_count > 100000 Aggregate General What is the total storage consumption per project? SUM(size) GROUP BY project Aggregate General Which projects are approaching their quota limits? usage / quota > 0.9 Aggregate Admin Which files are owned by deleted users? uid NOT IN active_users Individual Admin Which files exceed their retention period? mtime < retention_date Individual Admin Which users have the most small files? COUNT(file_size < 1MB) DESC Aggregate Admin What is the per-user file count and storage usage? SUM(size), COUNT(*) GROUP BY uid Aggregate Admin What is the 99th percentile directory size per project? PERCENTILE(size, 0.99) GROUP BY project Aggregate To address these emerging needs, we present Icicle, a scalable and extensible framework for continuous file system metadata indexing and monitoring across HPC file systems. Icicle aims to maintain an external, queryable, and unified view of disparate file systems to support semantically rich queries. Icicle operates in two complementary modes: snapshot mode , which ingests periodic metadata exports or GUFI-style database dumps; and update mode , which continuously consumes and reduces real-time metadata event streams such as Lustre changelogs and GPFS mmwatch [ 18 ] events. By integrating both modes using a modern stream-processing architecture built on Apache Kafka [ 20 ] and Apache Flink [ 6 ] , Icicle provides scalable, fault-tolerant, and idempotent processing of metadata events. Its open architecture can support any POSIX-compliant environment that exposes metadata event streams or snapshots, and it can directly feed downstream policy engines, dashboards, and workflow systems for real-time automation. This paper makes three contributions: 1. The design of a unified and scalable metadata indexing framework. We introduce Icicle, a scalable metadata indexing and monitoring framework that unifies heterogeneous sources of file system state, including snapshots, GUFI-style databases, and real-time changelogs from Lustre and GPFS. Icicle is open source and available at https://github.com/globus-labs/icicle/ . 2. High-throughput and idempotent stream processing. Icicle comprises three core components: (i) a high-throughput ingestion layer that collects file system metadata from existing tools and services; (ii) a distributed aggregation and processing layer built on Kafka and Flink that performs stream-based event reduction and state computation, to transform events into search-engine ingestion requests; and (iii) a web-based interface that enforces data visibility for administrators and users to perform queries and custom analytics. 3. Evaluation of scalability and throughput. We conduct a comprehensive evaluation of Icicle’s snapshot pipeline and event monitor on Lustre and GPFS file systems. In snapshot mode, we show that Icicle scales with both dataset size and available CPU resources, enabling efficient processing of multi-petabyte metadata exports. In update