XOR Bidding and Knapsack Formulations for HPC Network Resource Allocation Conference: Platform for Advanced Scientific Computing Conference; June 29-July 01, 2026; Bern, Switzerland Platform for Advanced Scientific Computing Conference (PASC ’26), June 29-July 01, 2026, Bern, Switzerland DOI: 10.1145/3815572.3815743 ISBN: 979-8-4007-2734-4/2026/06

Abrar Hossain, Kishwar Ahmed · 2026

Read XOR Bidding and Knapsack Formulations for HPC Network Resource Allocation Conference: Platform for Advanced Scientific Computing Conference; June 29-July 01, 2026; Bern, Switzerland Platform for Advanced Scientific Computing Conference (PASC ’26), June 29-July 01, 2026, Bern, Switzerland DOI: 10.1145/3815572.3815743 ISBN: 979-8-4007-2734-4/2026/06 by Abrar Hossain, Kishwar Ahmed (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-00490.txt.

Introduction

Hossain and Ahmed allot HPC network room by bids, not by queue place. Big compute hubs face floods of data from more and more fields. Plain first-come rules clog the pipes and slow lab work. Their plan weighs each stream's worth as bids state it.

Two sale tools are built for the task at hand. A greedy tool picks by worth per room in a fast pass. A knapsack sale tool, of VCG type, holds strong proof marks. Bids state data needs and the worth of the work. Sims pit both tools next to first-come rules. Done-time lags fall by vast shares at high load. Spread of lags shrinks, and load swings calm as well.

Catalog record

CREATORAbrar Hossain, Kishwar Ahmed
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.00490

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

XOR Bidding and Knapsack Formulations for HPC Network Resource Allocation Conference: Platform for Advanced Scientific Computing Conference; June 29-July 01, 2026; Bern, Switzerland Platform for Advanced Scientific Computing Conference (PASC ’26), June 29-July 01, 2026, Bern, Switzerland DOI: 10.1145/3815572.3815743 ISBN: 979-8-4007-2734-4/2026/06 Abrar Hossain, Kishwar Ahmed Abstract . Modern High Performance Computing (HPC) centers face growing challenges in ingesting large and diverse data streams. These issues often create bottlenecks that limit bandwidth use and delay scientific progress. Traditional static allocation and simple queuing methods are not sufficient. This paper presents a dynamic and value based approach to bandwidth allocation. We formalize the problem by incorporating both network and processing constraints. To address it, we introduce two new auction based mechanisms: the Greedy Value Density Auction, which is fast to compute, and the Vickrey–Clarke–Groves (VCG) Knapsack Auction, which offers strong theoretical guarantees. Both mechanisms rely on user bids that include data needs and scientific value. The goal is to maximize the total value of successful transfers, often referred to as social welfare. Simulation results show that our auction mechanisms significantly outperform First Come First Served (FCFS) baselines. In high load conditions, they reduce average and tail completion delays by over 80%. Predictability also improves, with the Coefficient of Variation of delay falling by 75 to 85%. Network stability increases as well, with load volatility (Peak to Average Ratio) dropping by up to 60 to 70%. This value driven and adaptive strategy helps reduce congestion, improves bandwidth use, and ensures fairer access based on scientific importance. † † cc-license: by 1. Introduction Motivation: Modern high-performance computing (HPC) centers increasingly function as “super-facilities,” integrating large-scale compute resources with diverse external data sources, including advanced scientific instruments like light sources and electron microscopes, as well as distributed sensor networks ( 6 ; 4 ) . Consequently, reliable and efficient data movement across potentially wide-area networks has become as critical to scientific productivity as computation itself. As outlined in Figure 1 , the primary challenge lies in ingesting massive, heterogeneous data streams originating from these sources, which exhibit wide variations in communication protocols (e.g., file transfer vs. streaming), data formats, metadata schemas, generation rates (spanning orders of magnitude), and reliability characteristics ( 19 ) . This inherent diversity often necessitates custom interfaces or gateways, complicates standardization efforts ( 9 ) , and leads to unpredictable, bursty load patterns at the HPC edge, challenging traditional infrastructure assumptions ( 12 ) . As data traverses staging systems towards core HPC storage and processing, it frequently encounters a significant bandwidth bottleneck at the ingestion point, affecting not just network links but potentially intermediate buffers and file systems ( 2 ) . Uncoordinated demands from disparate users and instruments exacerbate this bottleneck, resulting in network congestion, inefficient resource utilization characterized by oscillations between overload and underuse ( 12 ) , and significant transfer delays that directly impede end-to-end scientific workflows and leave expensive compute resources idle ( 7 ; 23 ) . Effectively managing access to this shared, limited ingestion bandwidth is therefore paramount for enabling timely, large-scale scientific analysis. Where Existing Data Movement Tools Fall Short: While various tools—such as SCP/SFTP ( 15 ) , Rsync ( 22 ) , GridFTP ( 13 ) , and proprietary UDP-based solutions like Aspera ( 14 ) —are commonly employed for data transfers, they often exhibit significant limitations when faced with the scale, heterogeneity, and dynamic nature of modern scientific workflows. Fundamentally, these point-to-point tools lack the adaptability required for dynamic end-to-end path conditions and are not inherently fault-tolerant in less predictable, highly congested environments ( 19 ) . As detailed further in Section 2.2 , the inability of these tools to optimally arbitrate mixed-priority workloads at the edge motivates the need for a more intelligent, value-aware data movement framework. Figure 1 . The scientific data pipeline, highlighting heterogeneous sources feeding into a common ingestion interface. Our Contributions: To address the critical challenges of managing constrained ingestion bandwidth amidst heterogeneous and dynamic demands, this paper makes the following contributions: • We formalize the HPC ingestion bandwidth allocation problem considering heterogeneous demands and priorities. • We propose two novel, value-aware auction mechanisms: Greedy Value-Density and VCG Knapsack for HPC data ingestion. • We detail the value-driven allocation and payment logic specific to each proposed mechanism. • We analyze the trade-offs between the VCG mechanism’s truthfulness and the Greedy mechanism’s efficiency. • We demonstrate through simulation the potential of auction-based strategies to mitigate ingestion bottlenecks and improve resource utilization compared to traditional methods. 2. Background (a) Hourly capacity demand at ALCF often exceeds 90% utilization, indicating frequent network congestion. (b) Evolution of average transfer size (GB) and total number of transfers per year at ALCF (2014-2024). (c) Yearly distribution of GridFTP transfer sizes (GB) at ALCF, illustrating increasing scale and persistent variability. Figure 2 . Observed data ingestion characteristics at ALCF ( 10 ) . HPC systems are increasingly burdened by data ingestion, where data transfer and I/O constraints critically limit end-to-end scientific workflows. Analysis of extensive operational data from large-scale facilities like the Argonne Leadership Computing Facility (ALCF) ( 10 ) reveals persistent challenges in managing data ingress efficiently. These challenges, stemming from workload characteristics and user behaviors, motivate the need for improved ingestion strategies. 2.1. Bottlenecks and Inefficiencies Symptoms of ingestion bottlenecks include significant network saturation and unpredictable latency spikes, frequently arising from competing data flows that degrade each other’s throughput, even across high-speed research networks ( 12 ) . The volatility of network demand is clearly illustrated in Figure 2(a) , which shows hourly GridFTP data volume at ALCF, normalized against a representative operational pathway capacity; demand frequently spikes above a 90% threshold, indicating periods of intense contention for this shared resource. Current infrastructure often struggles to handle such competing transfers efficiently ( 12 ) , leading to substantial delays in data availability and potentially underutilized compute resources awaiting necessary input ( 2 ) . This contention translates to significant and unpredictable transfer latencies; for instance, traditional methods in microscopy workflows incurred waits of 10-20 minutes, a delay drastically reduced by optimized streaming approaches ( 23 ) . Manual or semi-automated “store-and-forward” processes further contribute to these inefficiencies and can even lead to data loss ( 23 ) . The nature of the workload itself, as characterized by the ALCF data, contributes to these inefficiencies. As can be seen in Figure 2(b) , while the number of transfers has fluctuated, the average transfer size has shown a significant increasing trend, particularly in recent years, placing concentrated pressure on the ingest infrastructure. The wide range of transfer sizes, spanning orders of magnitude from megabytes to terabytes, is evident year-over-year (Figure 2(c) and Figure 4(a) ), demanding a system that can accommodate diverse data scales. Furthermore, uncoordinated user-initiated transfers create bursty network patterns, causing transient congestion during peak times ( 12 ) , while leaving bandwidth underutilized at other times. This is compounded by heterogeneous user behaviors: a small fraction of users often contributes a disproportionate amount to peak traffic (Figure 3(a) ), and users exhibit diverse predictability in their transfer patterns (Figure 3(b) ) and varying data interaction modes, such as differing PUT versus GET size characteristics (Figure 4(b) ) or a

Read the complete text