Characterizing WebGPU Dispatch Overhead for LLM Inference Across Four GPU Vendors, Three Backends, and Three Browsers
Jędrzej Maczan · 2026
Read Characterizing WebGPU Dispatch Overhead for LLM Inference Across Four GPU Vendors, Three Backends, and Three Browsers by Jędrzej Maczan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-02344.txt.
Introduction
Maczan characterizes WebGPU dispatch overhead for language model inference at batch size one. WebGPU per operation validation compounds across the many small dispatches in neural inference, yet naive benchmarks overstate dispatch cost. A sequential dispatch method reveals the true cost, with backend choice dominating and implementation choice mattering within backends. On Vulkan, kernel fusion lifts throughput by 53 percent while CUDA fusion gives no gain, confirming per operation overhead as a key differentiator. A new PyTorch backend and compiler reach about a tenth of CUDA performance.
Catalog record
| CREATOR | Jędrzej Maczan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55320 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.02344
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Characterizing WebGPU Dispatch Overhead for LLM Inference Across Four GPU Vendors, Three Backends, and Three Browsers Jędrzej Maczan Abstract WebGPU’s security-focused design imposes per-operation validation that compounds across the many small dispatches in neural network inference, yet the true cost of this overhead is poorly characterized. We present a systematic characterization of WebGPU dispatch overhead for LLM inference at batch size 1, spanning four GPU vendors (NVIDIA, AMD, Apple, Intel), two native implementations (Dawn, wgpu-native) and three browsers (Chrome, Safari, Firefox), and two model sizes (Qwen2.5-0.5B and 1.5B). Our primary contribution is a sequential-dispatch methodology that reveals naive single-operation benchmarks overestimate dispatch cost by ∼ 20 × {\sim}20\times . The true per-dispatch cost of WebGPU API overhead alone is 24–36 μ \mu s on Vulkan and 32–71 μ \mu s on Metal, while the total per-operation overhead including Python/framework cost is ∼ 95 {\sim}95 μ \mu s, which turns out to be a distinction critical for optimization. On Vulkan, kernel fusion reduces dispatches from 876 to 564, improving throughput by 53%, while CUDA fusion provides no benefit, confirming that per-operation overhead is a primary differentiator. End-to-end LLM inference was tested across three major operating systems (Linux, Windows, macOS) with five different backends (CUDA, MPS, CPU, WebGPU/Dawn, browser/WebLLM). We built torch-webgpu , our own PrivateUse1-based out-of-tree PyTorch backend and an FX-to-WebGPU compiler, which on our reference platform (NVIDIA RTX 5090/Dawn/Vulkan, float32), achieves 21.0 tok/s (0.5B) and 17.9 tok/s (1.5B), which corresponds to 11–12% of CUDA performance. At dtype-matched float32, a mobile CUDA GPU (RTX PRO 2000) achieves 1.4 × \times WebGPU’s throughput despite ∼ 6 × {\sim}6\times less compute than RTX 5090. For dispatch overhead, backend choice (Vulkan vs. Metal) is the dominant factor, although implementation choice also matters substantially within a backend (2.2 × \times for Metal). In terms of dispatch vs kernel compute efficiency, we conclude that at batch=1 with the current dispatch-heavy pipeline, per-operation overhead dominates regardless of kernel quality. All code, benchmarks, and raw data are open source. 1 Introduction WebGPU ( 32 ) provides cross-platform GPU access across all major operating systems and GPU vendors, making it a natural fit for ML deployment where portability or browser execution matters. However, WebGPU’s security-focused design imposes per-operation validation and command buffer submission, introducing overhead that compounds across the many small operations in neural network inference. The WebGPU and WGSL ecosystem is still not mature and is not widely utilized in either research or production. The performance of WebGPU implementations and specification is not well researched. Therefore, we ask what is the true per-dispatch overhead in WebGPU, how does it vary across backends, and how does it interact with framework-level and kernel-level overhead to determine end-to-end LLM inference performance? We make several contributions. First, we develop a sequential-dispatch measurement technique that reveals naive benchmarks overestimate dispatch overhead by \sim 20 \times , and apply it across four GPU vendors, three backends, and three browsers, establishing the true per-dispatch cost at 24–36 \mu s (Vulkan) and 32–71 \mu s (Metal). We distinguish per-dispatch cost (WebGPU API overhead, directly measured) from per-operation overhead ( \sim 95 \mu s, including Python/framework cost). Second, we show that kernel fusion yields 1.4 \times on Vulkan but provides no benefit on Metal or CUDA, providing causal evidence that per-operation overhead is the actionable bottleneck at batch size 1. Third, we provide end-to-end context: torch-webgpu (RTX 5090/Dawn, float32) achieves 21.0 tok/s (0.5B) and 17.9 tok/s (1.5B). At dtype-matched float32, a mobile GPU RTX PRO 2000 achieves only 1.4 \times WebGPU throughput despite \sim 6 \times less compute than RTX 5090. Per-operation overhead is consistent across model sizes, within range of \sim 95–99 \mu s. Finally, we characterize kernel efficiency: our unoptimized WGSL matmul reaches 1–2% of FP32 peak, though third-party evidence suggests \sim 17% is achievable ( 22 ) (we did not validate this ourselves). End-to-end inference was tested across all three platforms (Table 3 ), but the torch-webgpu backend specifically was tested only on RTX 5090/Dawn (Section 3.6 ). 2 Background 2.1 WebGPU Architecture WebGPU exposes GPU functionality through a command buffer model: operations are encoded into command buffers, bound to resources via bind groups, submitted to a queue for execution, and synchronized via explicit waits or buffer mapping. This provides security (operations are validated before execution) and portability (abstraction over Vulkan, Metal, D3D12), but each dispatch adds overhead from encoder creation, bind group creation, and submission. 2.2 LLM Inference Characteristics Transformer-based LLM inference ( 30 ) during autoregressive generation involves many small operations per forward pass (like attention, MLP, and normalization repeated per layer), sequential token generation where each token depends on the previous output, and a required GPU \rightarrow CPU synchronization step for token selection (argmax). All three properties add up into per-dispatch overhead. For Qwen2.5-0.5B (our primary test model), FX graph analysis reveals 1,911 total nodes per forward pass, of which approximately 876 are compute operations that potentially become WebGPU dispatches. The actual dispatch count depends on backend fusion optimizations. 2.3 Experiment Overview Table 1 classifies our experiments by scope. End-to-end LLM inference spans three platforms and five backends. Dispatch overhead benchmarks additionally cover four GPU vendors and three browsers. Table 1: Classification of experiments by scope and configuration coverage Experiment Type Dtype Configs Section End-to-end LLM inference torch-webgpu E2E fp32 1 (RTX 5090/Dawn) §4 CUDA baselines E2E fp16, fp32 2 GPUs, 2 platforms §4.2 MPS baselines E2E fp16, fp32 1 (Apple M2) §4.2 CPU baselines E2E fp32 3 platforms §4.2 ONNX Runtime (WebGPU) E2E fp32 1 (RTX 5090) §4 WebLLM (browser) E2E q4f16 6 configs † App. E Dispatch overhead benchmarks (dtype-independent) Native dispatch Micro — 4 vendors, 2 impls §7.2 Browser dispatch Micro — 3 browsers, 3 platforms §7.2 RMSNorm fusion Micro fp32 5 configs §7.3 CNN/ViT/U-Net dispatch Micro — RTX 5090 ∗ Exploratory (inconclusive, appendix only) Mega-kernel Micro fp32 RTX 5090, M2 App. C Device-side argmax Micro fp32 RTX 5090, M2 App. H ∗ CNN/ViT/U-Net dispatch overhead measured via exp9 , exp11 , exp13 (all show 24–58 \mu s, consistent with LLM results). † 2 platforms \times 3 browsers; Linux/Vulkan failed due to shader compatibility. 3 Methodology 3.1 Implementations torch-webgpu : A PyTorch ( 2 ) out-of-tree compilation backend we developed that translates torch.compile() FX graphs into WGSL ( 31 ) shaders executed with Google’s Dawn ( 11 ) WebGPU implementation, which integrates with PyTorch using PrivateUse1 mechanism. Dawn dependency : Dawn is Chromium’s WebGPU implementation. Its C API tracks the evolving WebGPU spec. Building from source requires Google’s depot_tools , creating a reproducibility challenge. The alternative, wgpu-native (Rust-based), is used in our cross-implementation validation (Section 7). ONNX Runtime (WebGPU) : Microsoft’s ML runtime ( 20 ) with WebGPUExecutionProvider . We used a development build (1.24.0.dev20251218001) with ORT_ENABLE_ALL graph optimization via optimum ’s ORTModelForCausalLM . This is the native version, not browser-based ONNX Runtime Web. 3.2 Hardware and Software Our primary test machine (WebGPU and CUDA benchmarks) uses an NVIDIA GeForce RTX 5090 (32GB VRAM) with an AMD Ryzen 7 9800X3D CPU, running Ubuntu 24.04 (kernel 6.14.0), PyTorch 2.9.1+cu128, ONNX Runtime 1.24.0.dev20251218001 (WebGPU provider) and Google’s Dawn. For cross-platform end-to-end inference we additionally test on a Windows 11 laptop with NVIDIA RTX PRO 2000 Blackwell (8GB) and Intel Core