Workload composition smooths aggregate power demand while sustaining short-horizon ramps in AI data centers

Subir Majumder, Minlan Yu, Le Xie · 2026

Read Workload composition smooths aggregate power demand while sustaining short-horizon ramps in AI data centers by Subir Majumder, Minlan Yu, Le Xie (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-10769.txt.

Introduction

Majumder, Yu, and Xie show that workload mix shapes grid-facing power dynamics in shared-GPU AI data centers. As inference share rises, total power variability traces a U-shape while short-horizon ramping traces a hump, with sizes and turning points depending on system load. A trace-calibrated design linking arrivals, queueing, scheduling, and GPU power reveals an asymmetric mechanism: at middle mixes, queued batch jobs fill idle capacity left by fluctuating inference demand. Yet short-horizon ramping stays high because inference swings feed straight into realized power.

Catalog record

CREATORSubir Majumder, Minlan Yu, Le Xie
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42377
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.10769

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Workload composition smooths aggregate power demand while sustaining short-horizon ramps in AI data centers Subir Majumder, Minlan Yu, Le Xie Abstract Artificial intelligence (AI) is driving rapid growth in electricity demand, yet the grid-facing power dynamics of AI data centers remain poorly understood. Here we show that, in shared-GPU systems, the composition of batch and inference workloads decouples aggregate power variability from short-horizon ramping. As the inference share rises, variability becomes U-shaped, whereas ramping becomes hump-shaped, particularly under higher loading. The magnitude and turning points of these patterns also depend on system loading. Using a trace-calibrated framework linking workload arrivals, queueing, scheduling, and GPU power, we show that the underlying mechanism is asymmetric. At intermediate workload mixes, queued batch jobs fill capacity left idle by fluctuating inference demand, reducing aggregate power variability. However, short-horizon ramping remains elevated because inference-side fluctuations propagate more directly into realized power. AI data centers should therefore be understood as dynamic systems whose workload composition shapes their grid impact. keywords AI, data center, electricity demand, grid integration, power variability, ramping, workload composition 1 Introduction Artificial intelligence (AI) is rapidly becoming a major source of electricity demand. As deployment expands, AI data centers are expected to account for a growing share of power-system load in the United States and globally 29 ; 16 . Yet their grid impact cannot be inferred from annual energy use alone. Power systems respond to demand over minutes to hours, so what matters operationally is not only how much electricity AI data centers consume, but also how quickly their loads rise, fall, and fluctuate. These short-horizon dynamics affect reliability, reserve requirements, infrastructure planning, emissions accounting, and the value of flexibility 37 ; 26 ; 15 ; 10 ; 12 ; 23 . Most estimates of AI electricity demand, however, are based on aggregate GPU counts or annual energy use 19 ; 21 , and planning studies often represent data centers as exogenous hourly loads with stylized flexibility limits 5 ; 27 ; 1 . Such approaches miss the internal operating dynamics through which computational work arrives, waits for service, and is translated into realized power demand. This gap matters increasingly because many AI data centers now colocate latency-sensitive inference and queue-managed batch workloads on shared-GPU systems 25 ; 35 ; 4 . We refer to such systems as hybrid AI data centers. Batch and inference workloads differ sharply in temporal structure, resource requirements, and power trajectories 24 ; 6 ; 31 ; 7 . Batch jobs can often wait in scheduler-managed queues 34 ; 38 ; 3 and, in some settings, be interrupted and resumed through checkpointing 9 . Online inference, by contrast, is latency-sensitive: production serving systems typically keep queues short 32 and sustain throughput through mechanisms such as continuous batching 18 or rerouting under saturation 40 . As a result, colocating batch and inference on shared-GPU systems may produce power dynamics that cannot be inferred from studying either workload in isolation. Here we show that hybrid AI data centers do not behave as weighted averages of batch-only and inference-only systems. Instead, workload composition changes grid-facing power demand nonlinearly: aggregate variability becomes U-shaped as inference share rises, whereas short-horizon ramping becomes hump-shaped. Hybrid systems can therefore be smoother in aggregate than the batch-only and inference-only extremes while remaining harder for the grid to follow on short timescales. This non-additive behavior arises because batch and inference workloads propagate fluctuations into power differently. Batch workloads buffer burstiness through queues, whereas latency-sensitive inference transmits it more directly into realized power, with only partial smoothing from capacity saturation and longer request residence times. When the two workloads share GPUs, queued batch work can fill residual-capacity valleys left by fluctuating inference demand, reducing aggregate variability without comparably reducing short-horizon ramping. AI data centers should therefore be understood not simply as large electric loads but as dynamic systems whose internal workload interactions shape grid-facing demand. Variability and ramping are distinct grid-facing properties of AI data centers and cannot be inferred from annual energy use alone. 2 Hybrid workload composition reshapes variability and ramping in AI data centers Hybrid batch–inference systems exhibit nonlinear load dynamics that cannot be inferred by interpolating between the batch-only and inference-only extremes. We find that hybrid systems are often less variable than either batch-only or inference-only systems, producing a U-shaped relationship between aggregate variability and inference share (Figure 1 (a); Supplementary Figure 20). Short-horizon ramping behaves differently. At higher utilization, the median short-horizon ramp rate becomes hump-shaped rather than monotonic as inference share rises (Figure 1 (b); Supplementary Figures 20 and 25). Thus, aggregate smoothness and short-horizon ramping decouple in shared-GPU AI data centers. Figure 1 : Hybrid AI data center power varies nonlinearly with workload composition. (a) Coefficient of variation (COV) of total power versus the inference share of total offered load across utilization levels. (b) Median normalized 15-minute ramp rate versus the inference share of total offered load for the same utilization levels. (c) Hourly distributions of normalized total power in a representative hybrid batch–inference configuration, decomposed into inference and batch contributions. In (a) and (b), faint curves show individual configurations and bold curves show the median within each utilization level. We show this using a trace-calibrated model of a hybrid AI data center in which batch jobs and inference requests share a common GPU pool. The model combines three empirical inputs: 98,000 batch jobs with per-job multi-GPU power profiles from MIT’s high-performance computing facility 28 , 44 million Azure LLM code and conversation requests 32 , and serving measurements that map inference token throughput to GPU demand and power 7 . In the model, latency-sensitive inference receives priority access to GPUs, while batch jobs use the remaining capacity 35 ; 4 . To isolate the role of workload composition, we hold GPU count and utilization fixed while varying only the share of total work assigned to batch and inference, and we repeat this comparison across a range of scenarios (Supplementary Note 2, Supplementary Table 6; Methods). For each simulation, power profiles are normalized so that the comparison reflects workload interaction rather than system scale. The reduction in variability at intermediate workload mixes arises mainly because batch jobs can wait. When inference demand is high, it takes GPU capacity immediately. Batch jobs, by contrast, can remain in the queue and start when inference leaves GPUs idle. This allows queued batch work to fill some of the gaps created by fluctuating inference demand, smoothing total GPU use and therefore aggregate power demand. The supplementary diagnostics support this interpretation. Batch GPU use increases when more capacity is left over after inference, and aggregate power closely follows total GPU use (Supplementary Figures 22–24). This smoothing effect becomes more important as the system gets busier and spare capacity becomes scarcer, which helps explain why the U-shape becomes more pronounced at higher utilization (Figure 1 (a); Supplementary Figure 20). How much smoothing batch jobs can provide also depends on how flexible they are. If batch jobs can be paused and resumed more frequently, they can respond more easily to changing leftover capacity. If checkpoint intervals are longer, batch work becomes less responsive, the minimum in variability shifts to lower inference shares, and variability rises sooner as

Read the complete text