WVA: A Global Optimization Control Plane for llmd
Abhishek Malvankar 1 , Lionel Villard 1 , Mohammed Abdi 1 , Tommaso Sgreccia 2 , Evgeny Shindin 1 , Braulio Dumba 1 , Vishakha Ramani 1 , Asser Tantawi 1 , Tamar Eilam 1 · 2026
Read WVA: A Global Optimization Control Plane for llmd by Abhishek Malvankar 1 , Lionel Villard 1 , Mohammed Abdi 1 , Tommaso Sgreccia 2 , Evgeny Shindin 1 , Braulio Dumba 1 , Vishakha Ramani 1 , Asser Tantawi 1 , Tamar Eilam 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-09730.txt.
Introduction
Abhishek Malvankar and coauthors describe WVA, a control plane that scales language model serving in the llmd system. Standard Kubernetes scaling ignores service goals, mixed hardware and the load inside each inference engine. WVA instead scales with spare headroom in mind and shrinks fleets without breaking live state. Trials show far higher useful throughput and far fewer failed requests than the standard scaler.
Catalog record
| CREATOR | Abhishek Malvankar 1 , Lionel Villard 1 , Mohammed Abdi 1 , Tommaso Sgreccia 2 , Evgeny Shindin 1 , Braulio Dumba 1 , Vishakha Ramani 1 , Asser Tantawi 1 , Tamar Eilam 1 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46795 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.09730
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
WVA: A Global Optimization Control Plane for llmd Abhishek Malvankar 1 , Lionel Villard 1 , Mohammed Abdi 1 , Tommaso Sgreccia 2 , Evgeny Shindin 1 , Braulio Dumba 1 , Vishakha Ramani 1 , Asser Tantawi 1 , Tamar Eilam 1 Abstract As Large Language Models (LLMs) scale to handle massive concurrent traffic, optimizing the infrastructure required for inference has become a primary challenge. To manage the high cost of GPU resources while ensuring strict service-level objectives (SLOs), operators increasingly deploy models across heterogeneous hardware clusters that multiplex latency-sensitive online requests and throughput-oriented offline requests. However, traditional resource-centric autoscalers like the Kubernetes horizontal pod autoscaler (HPA) do not consider application-specific SLOs, hardware heterogeneity, or internal engine state (like KV cache utilization) globally. This leads to unnecessary scaling, severe resource underutilization, and disrupted stateful inference. To address these limitations, we introduce the Workload Variant Autoscaler (WVA), a specialized control plane co-designed with llmd that tightly couples scaling decisions with the inference server’s internal saturation state. By utilizing proactive headroom-based scaling and fragmentation-aware scale-down, our experiments demonstrate that WVA achieves a 37% improvement in effective throughput and a 10x reduction in request failures compared to HPA. Furthermore, WVA’s cost-aware tiering intrinsically reduces overall power consumption by prioritizing lower-cost, energy-efficient hardware variants over homogeneous scaling on high-end accelerators. I Introduction The inference cost of serving Large Language Models (LLMs) has emerged as a critical bottleneck for the AI industry [ 1 , 2 ] . In contrast to stateless microservices where throughput scales linearly with compute resources, LLM inference is a memory-bound, stateful process. Performance is constrained by high-bandwidth memory (HBM) capacity for Key-Value (KV) caches and varies significantly based on input context length and decoding parameters [ 1 ] . Legacy autoscaling mechanisms like the Kubernetes HPA, designed for the homogeneous clusters of the microservices era, are ill-equipped for this new reality. Generic autoscalers treat the application as a black box, optimizing for Resource Utilization (e.g., keep CPU at 80%). We argue that this abstraction is leaky for LLMs. High-performance inference requires white-box autoscaling, where the scaler deeply understands the engine’s internal state. This impedance mismatch leads to massive inefficiency. GPU clusters are often over-provisioned to handle peak load, wasting energy during idle periods, or they fail to distinguish between expensive H100s and cost-effective A100s, treating different types of hardware as fungible units. To solve this, we must shift from resource-based scaling to SLO-based, heterogeneity-aware scaling. Fig. 1: WVA integrated with llmd We introduce the Workload Variant Autoscaler (WVA), a control plane framework designed for these complexities (see Figure 1 ). WVA integrates deeply with llmd [ 3 ] , a Kubernetes-native high-performance distributed LLM inference framework, to improve latency and throughput while providing cost and energy efficiency. Unlike monolithic scalers, WVA features a pluggable architecture that allows operators to inject custom optimization strategies for conflicting objectives. Leveraging this framework, we implement a saturation-based optimization that enables advanced capabilities such as headroom-based scaling and fragmentation-aware scale-down. II Background and Motivation II-A Background: LLM Inference Dynamics LLM inference is fundamentally distinct from traditional stateless microservices, characterized by its stateful nature and heterogeneous hardware requirements. The process comprises two distinct phases: Prefill and Decode . The prefill phase processes the input prompt in parallel to compute the initial Key-Value (KV) cache, a highly compute-bound operation. The decode phase subsequently generates tokens sequentially; this phase is memory-bandwidth bound and relies heavily on the cached KV states [ 4 ] . The KV Cache stores these intermediate attention states to avoid redundant computation, but it consumes significant GPU memory, making memory capacity the primary bottleneck. This architectural complexity introduces significant variability in latency expectations. User requests vary drastically in input prompt length and desired output tokens, leading to unpredictable processing times. Furthermore, due to the extreme capital expenditure required for AI infrastructure, clusters are increasingly heterogeneous, mixing older, cost-efficient GPUs (e.g., NVIDIA A100s) with newer, highly performant but expensive GPUs (e.g., NVIDIA H100s). II-B Motivation: Balancing Cost and SLOs The core motivation for our work is the urgent need to balance inference cost with stringent Service Level Objectives (SLOs). In production Model-as-a-Service (MaaS) environments, providers must navigate conflicting constraints: maximizing hardware utilization to reduce the per-token serving cost, while simultaneously maintaining strict tail-latency guarantees for end users. Achieving this balance requires an autoscaling control plane that can intelligently scale traffic across heterogeneous hardware tiers. For example, a system should prioritize scaling inexpensive A100 instances for baseline traffic to minimize cost, while reserving expensive H100 capacity to absorb sudden load spikes and protect latency SLOs. II-C State of the Art: The Shortcomings of Legacy Scaling Legacy autoscaling systems, such as the Kubernetes horizontal pod autoscaler (HPA), are insufficient for the complexities of LLM inference because they were designed for homogeneous, stateless workloads. Simple HPA operates on aggregated, generic resource metrics (e.g., CPU or memory utilization) and optimizes for a target average Resource Utilization (e.g., maintaining CPU at 80%). This approach treats the inference engine as a black box and fails to capture application-specific congestion signals like KV cache fragmentation or active queue depth. Even if HPA is extended via custom metric adapters, it still fundamentally treats all pods as identical, fungible replicas. It offers no native mechanism for cost-aware tiering (e.g., prioritizing A100s over H100s). Finally, HPA’s scale-down logic results in agnostic termination, removing replicas based on cluster-wide averages without regard for locally saturated KV caches on specific nodes, which severely disrupts stateful inference. Ultimately, solving these challenges requires the deep vertical integration of application-aware saturation, heterogeneity support, and decoupled actuation that legacy systems cannot provide. A common theoretical alternative is to modify the core Kubernetes HPA to natively understand these LLM-specific paradigms. However, doing so actively contradicts the cloud-native philosophy of modularity. HPA is intentionally designed as a generic, lowest-common-denominator controller for stateless, fungible pods. Embedding domain-specific logic—such as KV cache phase awareness, prefill/decode scaling requirements, or heterogeneous hardware tiering—into the core control plane would break backward compatibility and severely bloat the orchestrator. As highlighted by emerging CNCF initiatives [ 5 ] , managing the unique resource-utilization asymmetry of generative AI requires specialized, decoupled data-plane extensions rather than monolithic alterations to foundational orchestration primitives. Consequently, WVA is designed as an extensible, inference-aware overlaid control plane rather than a fork of HPA. III System Model III-A The Variant Abstraction Central to WVA is the introduction of the Variant as a first-class abstraction. A single base model (e.g., ‘Llama3-70b‘) could be deployed in multiple inference server configurations. We define a Variant as a unique tuple of: Variant=\langle Hardware,Parallelism,Quantization\rangle (1) For the current iteration of the WVA implementation evaluated in this work, we specifically constrain the variant space to two primary dimensions: Hardware , denoting the specific GPU model architecture hosting the inference server, and Parallelism , representing the discrete number of GPUs allocated per model replica. For example, ‘Variant A‘ might be ‘(H100, 2 GPUs)‘ while ‘Variant B‘ is ‘(A100, 4 GPUs)‘. This abstraction is operationalized via the VariantAutoscaling (VA) control-plane resource, enabling the system to evaluate the cost and performance trade-offs of each variant independently, rather than treating the service as a homogeneous pool of pods. III-B Saturation Model (Default Strategy) WVA functions as the