HBM Is Not All You Need: Efficient Disaggregated LLM Serving across Memory-heterogeneous Accelerators Conference: Preprint; 2026;
Zhixiang Wei, Yun Wang, James Yen, Mingyuan Xia, Zhengwei Qi · 2026
Read HBM Is Not All You Need: Efficient Disaggregated LLM Serving across Memory-heterogeneous Accelerators Conference: Preprint; 2026; by Zhixiang Wei, Yun Wang, James Yen, Mingyuan Xia, Zhengwei Qi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29986.txt.
Introduction
Zhixiang Wei, Yun Wang, James Yen, Mingyuan Xia, and Zhengwei Qi tackle the cost of serving large language models. Data center chips rely on expensive high bandwidth memory. Yet that memory sits mostly idle during prefill, when a model first reads a prompt. Their system, HMA-Serve, splits the work across unlike hardware instead of forcing one chip type to do everything.
Cheaper GDDR chips handle prefill, using a slower memory type. High bandwidth memory chips handle decode, the token by token phase. Three tricks make the split work well despite the mixed vendors involved. The system uses different numeric precision per phase. A pipeline overlaps data transfer with computation. Compressed values are converted only when needed on the decode side. Across four Qwen3 models and three production traces, throughput more than triples. Throughput per dollar rises nearly fivefold, with no measured quality loss.
Catalog record
| CREATOR | Zhixiang Wei, Yun Wang, James Yen, Mingyuan Xia, Zhengwei Qi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.29986
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
HBM Is Not All You Need: Efficient Disaggregated LLM Serving across Memory-heterogeneous Accelerators Conference: Preprint; 2026; Zhixiang Wei, Yun Wang, James Yen, Mingyuan Xia, Zhengwei Qi Abstract . LLM inference comprises a compute-bound prefill phase and a memory-bound decode phase, and recent systems disaggregate them onto separate hardware. Yet today’s datacenter GPUs rely on costly HBM whose bandwidth sits almost entirely idle during prefill. LLM serving across memory-heterogeneous accelerators (MemHA) pairs GDDR-based accelerators for prefill with HBM-based GPUs for decode, promising lower cost without sacrificing performance. Pushed to its most economical form, MemHA serving is inherently cross-vendor, since the best-suited chip for each phase may come from a different vendor. This breaks two assumptions that single-vendor disaggregation takes for granted—a KV format both ends consume natively, and a shared software stack. We present HMA-Serve , a MemHA-centric disaggregated serving system pairing GDDR-based accelerators for prefill with HBM-based GPUs for decode efficiently. HMA-Serve achieves this through (1) phase-wise quantization, applying vendor-native low precision for high-throughput prefill while keeping decode in high-precision BF16, (2) a compute-transfer pipeline that overlaps each layer’s KV cache transfer with later-layer prefill to reduce time-to-first-token (TTFT), and (3) deferred dequantization, shipping raw quantized bytes and reconstructing them lazily on the decode GPU to reduce network bandwidth and HBM usage. Across four Qwen3 models (4B–32B) and three production traces, HMA-Serve delivers up to 3.2 × 3.2\times higher goodput than state-of-the-art memory-homogeneous methods and 4.8 × 4.8\times higher goodput-per-dollar, with no measurable loss on generation-quality benchmarks. 1 1. Introduction Large Language Models (LLMs) are now deployed across chatbots, code assistants, and agentic applications, but serving them is extremely expensive. One early estimate placed the daily cost of running ChatGPT at roughly $700,000 in early 2023, requiring on the order of 30,000 NVIDIA A100 GPUs, and demand has only grown since. As of early 2026, NVIDIA’s flagship accelerators remain sold out with a multi-million-unit backlog, roughly two-thirds of which is expected to serve inference. Driving down the hardware cost of LLM serving is therefore a first-order systems problem. LLM inference runs in two phases with opposing resource profiles. The prefill phase processes the entire prompt in parallel to produce the KV cache and the first token, and is compute-bound. The decode phase generates tokens sequentially, each depending on the KV state of all preceding tokens, and is memory-bound, sweeping the KV cache and weights once per token. Because each phase exercises only a subset of a GPU’s resources, prior work improves utilization in two ways. Prefill-decode colocation batches prefill and decode of different requests to share weights, but suffers from prefill–decode interference and tail latency. Prefill-decode disaggregation runs the phases on separate instances connected by a KV cache transfer, removing interference and matching each phase to suitable hardware at modest transfer cost. Despite these software optimizations, hardware efficiency remains limited by a mismatch between prefill and costly HBM. HBM has become the dominant cost component of modern accelerators—18% of A100 manufacturing cost, rising to 45% on B200—yet prefill’s high arithmetic intensity leaves this premium bandwidth almost entirely idle. We measure an A100 wasting over 97% of its HBM bandwidth during 4K-token prefill. This motivates Memory-Heterogeneous Accelerator (MemHA) serving: GDDR-based accelerators, such as Tenstorrent’s chips, handle prefill, while HBM-based GPUs handle decode. This approach is highly promising in principle. For example, a state-of-the-art GDDR-based accelerator, the Tenstorrent Blackhole p150, provides 664 TFLOPS of Blocked FP8 (BFP8) compute at roughly $1,300, an order of magnitude cheaper than the A100 while offering comparable or higher low-precision throughput. Because cost-efficient GDDR accelerators have only recently shipped and share no software stack with NVIDIA, the most economical MemHA pairing is inherently cross-vendor , and little is known about whether and how it pays off. Making cross-vendor MemHA serving efficient raises two systematic challenges. First, the KV cache sits on the critical path of every request and is large enough to dominate latency unless its transfer is hidden. Qwen3-32B at 8K context produces gigabytes of KV state; even over a 100 Gbps RDMA fabric, a monolithic post-prefill transfer takes hundreds of milliseconds—comparable to prefill itself—and a naive schedule charges all of it to time-to-first-token (TTFT). The cross-vendor path is also longer than a network hop. The cache must first cross a device-to-host DMA before reaching the fabric. Second, the KV cache a GDDR prefiller produces does not match what HBM decode kernels expect, in number format or memory layout. Tenstorrent stores BFP8 in a 32{\times}32 tiled layout, while NVIDIA decode kernels read BF16 in the paged row-major layout FlashAttention ( 4 ) consumes. Both naive bridges are bad: dequantizing on the producer doubles wire traffic, while a standalone consumer pass adds a full-tensor HBM read that competes with attention. We present HMA-Serve , which turns the precision asymmetry into a performance lever and keeps transfer off the critical path through three coordinated mechanisms. Phase-wise quantization runs prefill in Tenstorrent’s vendor-native BFP8 and decode in BF16 to maximize throughput while minimizing accuracy loss. Compute-transfer pipelining exposes per-layer completion events from the prefill runtime and overlaps each layer’s KV egress—device-to-host push plus RDMA—with the prefill of later layers. Deferred dequantization ships raw quantized bytes verbatim, halving wire traffic, and reconstructs them lazily inside a fused decode-side kernel; because the reconstruction is integer bit manipulation, it runs on the GPU’s integer ALU rather than the tensor cores decode saturates, isolating its cost from decode at the hardware level instead of competing for the same compute. We implement HMA-Serve on a deployed cluster of four Tenstorrent Blackhole p150b accelerators for prefill and an NVIDIA A100 80 GB for decode over a 100 Gbps RDMA network, and evaluate four Qwen3 models (4B–32B) under three production traces. This work contributes: • HMA-Serve , to our knowledge the first MemHA serving system on real silicon pairing GDDR-based prefill with HBM-based decode across a commodity RDMA fabric. • Phase-wise quantization : vendor-native low precision for compute-bound prefill, BF16 for precision-sensitive decode. • A Compute-transfer pipeline that hides cross-vendor KV egress behind later-layer prefill. • Deferred dequantization : shipping raw quantized bytes and reconstructing them lazily in a fused decode-side kernel. • An evaluation across four LLMs and three workloads showing up to 3.2\times higher goodput than state-of-the-art memory-homogeneous serving systems and 4.8\times higher goodput-per-dollar, with no measurable quality loss. 2. Background and Motivation Table 1 positions HMA-Serve against representative serving systems along five axes: off-chip memory architecture, heterogeneous hardware across phases, disaggregation, phase-wise precision, and real-silicon demonstration. Non-HBM Het. Dis. PQ Real Lat. Thru. Cost Orca ( 9 ) ✗ ✗ ✗ ✗ ✓ Var. Low High Sarathi ( 2 ) ✗ ✗ ✗ ✗ ✓ Var. High High Groq ( 1 ) ✓ ✗ ? ✗ ✓ Low ? ? DistServe ( 11 ) ✗ ✗ ✓ ✗ ✓ Low High Med Splitwise ( 7 ) ✗ ✓ ✓ ✗ ✓ Low High Med Mix-Quant ( 6 ) ✗ ✗ ✓ ✓ ✓ Low High Med SPAD ( 10 ) ✓ ✓ ✓ ✗ sim Low High Low HMA-Serve ✓ ✓ ✓ ✓ ✓ Low High Low Table 1 . LLM serving design space. Non-HBM : a phase chip uses GDDR/SRAM; Het. : heterogeneous hardware across phases; Dis. : disaggregated scheduling; PQ :