Metadata Intelligent Distribution Algorithm for Servers (MIDAS) : Adaptive Proxy Middleware for Mitigating Metadata Hotspots in High-Performance Computing (HPC) I/O at Scale
Sangam Ghimire, Nigam Niraula, Nirjal Bhurtel, Paribartan Timalsina, Bishal Neupane, James Bhattarai, Sudan Jha · 2025
Read Metadata Intelligent Distribution Algorithm for Servers (MIDAS) : Adaptive Proxy Middleware for Mitigating Metadata Hotspots in High-Performance Computing (HPC) I/O at Scale by Sangam Ghimire, Nigam Niraula, Nirjal Bhurtel, Paribartan Timalsina, Bishal Neupane, James Bhattarai, Sudan Jha (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-18124.txt.
Introduction
Sangam Ghimire, Nigam Niraula, Nirjal Bhurtel, and four co-authors present MIDAS, middleware that eases metadata hotspots in high performance computing storage. When thousands of jobs start or checkpoint at once, a few metadata servers can be swamped, causing long queues and slow, uneven response.
MIDAS sits transparently between clients and servers, with no kernel or backend changes. It balances load using live telemetry, caches cooperatively while preserving backend meaning, and adjusts its own aggressiveness through a stabilizing control loop. Experiments cut average queue lengths by about a quarter and worst case hotspots by up to four fifths versus round robin scheduling.
Catalog record
| CREATOR | Sangam Ghimire, Nigam Niraula, Nirjal Bhurtel, Paribartan Timalsina, Bishal Neupane, James Bhattarai, Sudan Jha |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-18124
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Metadata Intelligent Distribution Algorithm for Servers (MIDAS) : Adaptive Proxy Middleware for Mitigating Metadata Hotspots in High-Performance Computing (HPC) I/O at Scale Sangam Ghimire, Nigam Niraula, Nirjal Bhurtel, Paribartan Timalsina, Bishal Neupane, James Bhattarai, Sudan Jha Abstract Metadata hotspots remain one of the key obstacles to scalable Input/Output (I/O) in both High-Performance Computing ( HPC ) and cloud-scale storage environments. Situations such as job start-ups, checkpoint storms, or heavily skewed namespace access can trigger thousands of concurrent metadata requests against a small subset of servers. The result is long queues, inflated tail latencies, and reduced system throughput. Prior efforts including static namespace partitioning, backend-specific extensions, and kernel-level modifications address parts of the problem, but they often prove too rigid, intrusive to deploy, or unstable under shifting workloads. We present MIDAS , an adaptive middleware layer that operates transparently between clients and metadata servers, requiring no changes to kernels or storage backends. The design brings together three mechanisms: (i) a namespace-aware load balancer that enhances consistent hashing with power-of- d d sampling informed by live telemetry, (ii) a cooperative caching layer that preserves backend semantics through leases, invalidations, or adaptive timeouts, and (iii) a self-stabilizing control loop that dynamically adjusts routing aggressiveness and cache lifetimes while avoiding oscillations under bursty workloads. Analysis of the model and controlled experiments show that MIDAS reduces average queue lengths by roughly 23% and mitigates worst-case hotspots by up to 80% when compared to round-robin scheduling. These findings highlight that a stability-aware, middleware-based strategy can provide backend-agnostic improvements to metadata management, enabling better scalability in bursty scenarios, more predictable tail latencies, and stronger overall system performance. I Introduction In large HPC systems, metadata frequently emerges as the bottleneck for I/O scalability. Real-world traces show that metadata workloads are highly bursty: during job initialization or checkpointing, activity can spike by more than two orders of magnitude across a cluster. For example, monitoring tools such as Darshan report fluctuations exceeding 100 \times in cluster-wide metadata operations [ 1 ] . When thousands of processes simultaneously launch or checkpoint, their requests concentrate on a small subset of directories, rapidly overwhelming the Metadata Server (MDS) . The result is long queues and sharp increases in worst-case response times (tail latency), making hotspots a recurring challenge rather than a rare anomaly [ 1 ] . A variety of techniques have been proposed to mitigate this problem, but critical gaps remain. Static partitioning methods such as subtree or hash-based schemes achieve balance under uniform workloads, yet they struggle with skewed access patterns, causing cross-server traversals and expensive repartitioning [ 2 , 3 , 4 ] . Dynamic approaches, including Lustre Distributed Namespace Environment (DNE) rebalancing, can partially relieve imbalance but often destabilize the system until carefully tuned for specific workloads [ 5 , 6 ] . Middleware-based projects such as IndexFS and PACON demonstrate the promise of adaptive routing and caching, but they generally lack strong stability guarantees and are difficult to generalize across different backends [ 7 , 8 ] . MIDAS addresses these limitations by mitigating metadata hotspots while preserving stability and consistency. Operating transparently between clients and metadata servers, it terminates metadata Remote Procedure Call (RPC) s to enable safe request re-routing and cooperative caching without requiring changes to client kernels or server code. Its design builds on three principles: (i) namespace-aware load balancing, (ii) cooperative caching with leases, invalidations, or adaptive Time-To-Lives (TTLs) , and (iii) a self-stabilizing control loop that dynamically adapts routing and cache lifetimes while avoiding oscillations under bursty workloads. II Literature Review Metadata management has long been a critical scalability bottleneck in High-Performance Computing ( HPC ) parallel file systems. Bursty job start-ups, checkpoint storms, and skewed namespace access patterns generate surges of requests to metadata servers ( MDS ), leading to queuing delays and significantly impacting tail latencies [ 9 , 10 ] . At exascale, the intensity and burstiness of metadata operations are further amplified, making hotspot mitigation one of the central challenges in HPC I/O research [ 11 ] . Early systems attempted to address this challenge using static namespace partitioning techniques such as subtree or hash-based distribution. Subtree partitioning offered simplicity and reasonable balance under uniform workloads, but suffered from severe load imbalance when access patterns were skewed [ 2 , 3 ] . Dynamic subtree partitioning was later introduced to repartition overloaded subtrees for improved balance [ 4 ] . However, frequent metadata migration destabilized the system and introduced complexity [ 3 ] . Hash-based partitioning subsequently mapped identifiers such as pathnames to servers using modulus functions, ensuring uniform load distribution but disrupting metadata locality [ 2 , 3 ] . This disruption often caused costly cross-server traversals during directory operations and introduced additional overhead as systems scaled [ 2 , 3 ] . To reduce the limitations of static approaches, researchers have explored middleware-based solutions that act as an indirection layer between clients and file systems. Projects such as IndexFS [ 7 ] and PACON [ 8 ] employ adaptive routing, caching, and distributed indexing to achieve scalability and flexibility without modifying kernel or server internals. This shift marked a pivot from backend-specific mechanisms toward broader adaptive, system-agnostic techniques that could be deployed incrementally. Real-world studies further highlight the severity of metadata bursts. For example, monitoring tools such as Darshan have observed fluctuations of more than 100\times in cluster-wide metadata activity during job initialization and checkpoints [ 1 ] . These findings confirm that challenges of skew and burstiness remain despite progress in parallel I/O , emphasizing the need for adaptive mechanisms that can respond dynamically in real time [ 12 ] . Some file systems now incorporate internal load-balancing mechanisms. For example, Lustre supports DNE , enabling automatic Metadata Target (MDT) restriping and rebalancing [ 5 ] . While effective in many cases, persistent imbalance remains when directory distribution is suboptimal [ 13 ] . Similarly, CFS introduces scalable critical sections for metadata operations in Portable Operating System Interface (POSIX) -compliant file systems to enhance concurrency [ 6 ] . However, these methods require significant manual tuning and offer limited flexibility in dynamic environments. Dynamic load balancing strategies have also been explored [ 14 ] . The power-of- d choices algorithm reduces imbalance compared to random placement [ 15 , 16 ] . Yet, namespace locality and consistency constraints restrict its freedom, preventing straightforward application in metadata workloads [ 17 ] . Cooperative metadata caching has been studied as well. Straightforward caching can introduce staleness and inconsistency, so advanced mechanisms have been proposed: server-issued leases or invalidation tokens (e.g., in CephFS [ 18 ] and HyCache+ [ 19 ] ), adaptive time-to-live policies guided by feedback control [ 20 ] , and gossip-based coordination among proxies [ 21 ] . While these improve throughput, ensuring consistency across distributed caches remains a difficult challenge. Beyond HPC , distributed and cloud-scale systems also address metadata hotspots. Google File System (GFS) [ 22 ] and Hadoop Distributed File System (HDFS) [ 23 ] mitigate hotspots by relaxing POSIX semantics and grouping namespace chunks. Similarly, systems such as ZooKeeper [ 24 ] , Cassandra [ 25 ] , and DynamoDB [ 26 ] leverage caching and gossip protocols for scalable state