Memory-Bound but Not Bandwidth-Limited: The Physical AI Inference Gap in Batch-1 LLM Decode A 44-Cell Cross-GPU Study of Memory Floors, CUDA Graphs and Quantized Decode
Josef Chen · 2026
Read Memory-Bound but Not Bandwidth-Limited: The Physical AI Inference Gap in Batch-1 LLM Decode A 44-Cell Cross-GPU Study of Memory Floors, CUDA Graphs and Quantized Decode by Josef Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30571.txt.
Introduction
Chen studies a puzzle in running language models on robots and edge devices, where one user awaits each token and each step streams weights and cached keys. Such batch-1 decoding is called memory-bandwidth-bound, yet testing three mid-size models on four graphics cards shows faster memory buys little speed. A slow card nears its memory floor while a fast card reaches barely a quarter of it. A CUDA Graphs test pins the gap on launch overhead; only one quantized path nears the expected speedup.
Catalog record
| CREATOR | Josef Chen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72212 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.30571
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Memory-Bound but Not Bandwidth-Limited: The Physical AI Inference Gap in Batch-1 LLM Decode A 44-Cell Cross-GPU Study of Memory Floors, CUDA Graphs and Quantized Decode Josef Chen Abstract Physical AI systems, including robots, autonomous vehicles, embodied agents and edge copilots, often run a different inference workload from cloud LLM serving: single-stream, batch-1 autoregressive decode, where one robot, camera feed or user session waits on the next token. This workload is usually described as memory-bandwidth-bound. Each decode step streams model weights and the active KV cache, so latency should scale with peak HBM bandwidth. We show that this account is true but incomplete. We measure batch-1 decode for three 7 to 8B-class GQA transformers across four NVIDIA GPUs: H100 SXM5, A100-80GB SXM4, L40S and L4. We evaluate context lengths from 2048 to 16384, producing 44 valid cells under a controlled bf16 SDPA setup. The achieved fraction of peak HBM bandwidth falls as peak bandwidth rises. On the headline Qwen-2.5-7B ctx = 2048 {=}2048 cell, an L4 reaches roughly 81% of its analytic memory floor, while an H100 reaches only 27%. Physical-AI decode is memory-dominated, but faster memory does not translate into proportional latency gains. We test the missing term with a CUDA Graphs A/B experiment. On H100 at ctx = 2048 {=}2048 , CUDA Graphs improves decode latency by 1.259 × 1.259\times across N = 10 N{=}10 fresh sessions, with a 95% bootstrap confidence interval of [ 1.253 , 1.267 ] [1.253,1.267] . On L4, the same intervention gives only 1.028 × 1.028\times . This isolates a launch-side overhead that becomes visible on fast GPUs but remains mostly hidden on slower, bandwidth-bound GPUs. The deployment implication is that memory savings matter only when the runtime realises them. On L4, bf16 decode sits close to the memory floor, but common quantised paths do not recover the expected 4 × 4\times weight-traffic reduction: bnb-nf4 reaches 59.36 ms/step and AutoAWQ + Marlin reaches 45.24 ms/step from a 62.32 ms bf16 baseline. GPTQ + ExLlamaV2, with Ada-tuned int4 kernels, reaches 17.36 ms/step. For physical-AI inference, the bottleneck is not peak bandwidth alone, but the interaction between memory traffic, launch overhead and runtime support for compressed weight movement. 1 Introduction Physical-AI inference, robotics policy heads, autonomous-driving language copilots and on-device assistants share a workload that LLM-serving research mostly does not target: single-stream, batch-1, autoregressive decode of 7–8B-class transformers. The user is one human, one robot or one camera feed; latency between tokens is the user-facing metric; throughput-oriented batching is not available because there is no batch. The standard account of batch-1 decode is that it is HBM-bandwidth-bound: each decode step streams the model weights and the per-layer KV cache through global memory once, arithmetic intensity is too low to keep the SMs busy, and step time is approximately (W+K)/B_{\mathrm{peak}} . This account motivates a sizeable fraction of recent serving work: KV cache compression [ 29 , 14 , 27 , 16 , 10 ] , weight quantisation [ 7 , 15 , 26 ] and KV offloading. It also drives the implicit assumption that deployment ladder L4 \rightarrow L40S \rightarrow A100 \rightarrow H100 is the cost-per-token ladder, because each rung adds peak HBM bandwidth. This paper is a measurement study, not a modeling paper. We measure decode step time across four NVIDIA GPUs that span more than a 10\times range in peak HBM bandwidth (L4 at 300 GB/s [ 19 ] through H100 SXM5 at 3350 GB/s [ 18 ] ), three 7–8B-class GQA architectures [ 2 ] (Qwen-2.5-7B [ 21 ] , Mistral-7B-v0.3 [ 11 ] , Llama-3.1-8B), and four context lengths between 2048 and 16384, all at batch 1 with bf16 weights and sdpa attention on Modal cloud hosts [ 17 ] . The directly-measured observed-over-floor ratio R_{\mathrm{floor}}=t_{\mathrm{obs}}/t_{\mathrm{floor}} , with t_{\mathrm{floor}}=(W+K)/B_{\mathrm{peak}} , falls from \sim 0.8 on L4 to \sim 0.25 on H100 at short contexts. Translated into achieved bandwidth, the L4 reaches roughly 81% of its peak HBM bandwidth on this workload while the H100 reaches only 27%. The HBM-bound prediction therefore overestimates achievable speedup on fast silicon by a factor of three on this workload class. Mechanism, falsified at N{=}10 . The load-bearing falsifiable claim is mechanistic: the H100 gap is per-kernel CPU launch overhead, not silicon. We test it with a CUDA Graphs A/B that touches the launch term and only the launch term. On the H100 ctx {=}2048 headline cell at N{=}10 independent Modal sessions, the within-session graphed/eager speedup is 1.259\times with 95% bootstrap CI [1.253,1.267] and cross-session CV 0.9%; on L4 at the same cell the same intervention yields 1.028\times . The prediction would have been killed by an H100 speedup under \sim 1.15\times in most sessions, or by an L4 speedup over \sim 1.15\times in any session. Neither condition occurred. Deployment implication. The substantive implication is the inverted cost-per-token ladder. With backend-pinned SDPA measurements, PyTorch’s default SDPA dispatcher ( 36.05\,\mathrm{us} /layer) outperforms explicit FLASH_ATTENTION context ( 44.35\,\mathrm{us} ), FlashInfer ( 48.20\,\mathrm{us} ) and FlashAttention-3 ( 79.25\,\mathrm{us} ) at H100 batch-1 single-decode shape; the kernel choice is not the binding constraint. The kernel choice on L4 is the binding constraint: substituting GPTQ {+} ExLlamaV2 for the default Marlin path cuts Qwen-2.5-7B step time from 62.32 ms (bf16) to 17.36 ms, a 3.59\times speedup that AutoAWQ {+} Marlin (45.24 ms) does not deliver on Ada SM89. An L4 with ExLlamaV2 runs Qwen-2.5-7B at 17.36 ms/step; an H100 with CUDA Graphs runs the same workload at 11.78 ms/step. For the streaming workloads that dominate physical AI, the GPU upgrade closes less of the gap than the kernel choice on the cheaper silicon does. Figure 1: The inverted deployment ladder for single-stream 7–8B decode. Left: achieved fraction of peak HBM bandwidth at Qwen-2.5-7B ctx {=}2048 batch 1 across four NVIDIA GPUs; L4 reaches 79% of its peak, H100 only 27%. Right: step time for the same workload after applying the best lever per GPU; L4 with GPTQ {+} ExLlamaV2 (17.36 ms) closes most of the gap to H100 with CUDA Graphs (11.78 ms) despite the H100 having 11\times the peak HBM bandwidth. Contributions. 1. A measurement study of single-stream decode at batch 1 across three 7–8B-class GQA architectures, four NVIDIA GPUs and four context lengths, with controlled software stack, warmup and 30 measured steps per cell (44 valid cells, 4 OOMs on L4 at long context, all diagnosed as A/B-protocol artifacts not capacity ceilings). 2. A CUDA Graphs A/B falsification test at N{=}10 within-session and cross-session, confirming the launch-tax interpretation on H100 short-context decode and rejecting it on L4 with a null result. 3. Backend-pinned attention-kernel measurements at H100 batch-1 single-decode shape, showing that PyTorch’s default SDPA dispatcher beats every explicit backend including FlashAttention-3 and FlashInfer at this shape, and that cuDNN’s attention path is not even supported for it. 4. A controlled L4 quantisation comparison (bf16, bnb-nf4, AutoAWQ {+} Marlin, GPTQ {+} ExLlamaV2) showing the kernel-implementation gap on Ada SM89 silicon and the cost-per-token inversion against H100. 5. A reproducible Modal-script payload covering the 44-cell sweep, the N{=}10 headline replication, the backend-pinned attention matrix and the L4 quantisation matrix, with all JSON artefacts archived. We do not claim the launch-bound observation itself is novel; CUDA Graphs and launch-bound regimes are folk