Llamas on the Web: Memory-Efficient, Performance-Portable, and Multi-Precision LLM Inference with WebGPU

Reese Levine, Rithik Sharma, Nikhil Jain, Abhijit Ramesh, Zheyuan Chen, Neha Abbas, James Contini, Tyler Sorensen · 2026

Read Llamas on the Web: Memory-Efficient, Performance-Portable, and Multi-Precision LLM Inference with WebGPU by Reese Levine, Rithik Sharma, Nikhil Jain, Abhijit Ramesh, Zheyuan Chen, Neha Abbas, James Contini, Tyler Sorensen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20706.txt.

Introduction

Reese Levine and seven colleagues present Llamas on the Web, a WebGPU backend for llama.cpp that runs language models in the browser. Browser inference promises private, portable AI apps, but memory is tight and hardware varies widely. Their design cuts memory overhead with static planning and efficient loading, and a tunable kernel library handles cross-device differences. Templated kernels support many quantization formats, and tests span 16 devices from eight vendors.

Catalog record

CREATORReese Levine, Rithik Sharma, Nikhil Jain, Abhijit Ramesh, Zheyuan Chen, Neha Abbas, James Contini, Tyler Sorensen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74602
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.20706

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Llamas on the Web: Memory-Efficient, Performance-Portable, and Multi-Precision LLM Inference with WebGPU Reese Levine, Rithik Sharma, Nikhil Jain, Abhijit Ramesh, Zheyuan Chen, Neha Abbas, James Contini, Tyler Sorensen Abstract . Running language models in the browser presents a unique opportunity to build efficient, private, and portable AI applications, but requires contending with constrained memory availability and heterogeneous hardware targets. To realize this opportunity, we present Llamas on the Web (LlamaWeb), a WebGPU backend for llama.cpp that enables memory-efficient and performance-portable LLM inference across a wide range of model weight formats in the browser. Our design significantly reduces memory overhead through static memory planning and efficient model loading, addresses cross-device variability through a tunable kernel library, and introduces templated GPU kernels that support performant implementations of numerous quantization formats, enabling broad model support and extensibility to new formats. We evaluate LlamaWeb on 16 devices from 8 vendors, collecting data from 10 language models and four model weight formats. We compare LlamaWeb against existing browser-based LLM frameworks and find that LlamaWeb requires 29–33% less memory across several combinations of device, browser, and operating system. We also evaluate LlamaWeb’s performance against these frameworks and find that it increases decode throughput by 45–69% across four GPUs from separate vendors. In addition, we compare LlamaWeb’s performance against other llama.cpp backends, where it is competitive with and even beats vendor-specific backend performance on some devices. Table 1. Comparison of LlamaWeb (this work) to other browser-based LLM inference frameworks on key metrics. LlamaWeb WebLLM Transformers.js Available Models 1 177,691 400 41,632 Memory Usage 2 Baseline +49% +41% Performance 3 Baseline -35% -41% Weight Formats 4 23 6 7 1 Number of compatible models available in May 2026, measured using publicly accessible Hugging Face repositories (App. A ). 2 Geometric mean comparison of normalized peak browser memory usage across several configurations during inference of an f16 Llama3.2 1B model (Sec. 5 ). 3 Geometric mean comparison of normalized decode tokens per second of the same Llama3.2 model across 4 GPUs (Sec. 6 ). 4 Number of supported floating-point and quantization formats available for inference in WebGPU (App. B ). 1. Introduction The capabilities of (smaller) large language models (LLMs) are improving rapidly ( 58 ; 5 ) , with many model developers releasing open-weight models that are explicitly designed for edge deployments ( 55 ; 60 ; 1 ) . Running models locally can improve inference latency and energy efficiency ( 27 ) , especially as AI datacenter power usage continues to grow ( 59 ; 30 ) . Additionally, local models provide privacy benefits, with a recent study finding that many frontier AI companies retain and utilize user conversations for training by default and lack transparent privacy policies ( 33 ) . Capitalizing on these benefits, inference engines designed to run on consumer and edge devices like llama.cpp ( 14 ) and MLX ( 23 ) have risen in popularity over the past few years. These engines often ship with vendor-specific GPU backends, e.g., MLX is optimized for Apple GPUs, require expert knowledge to setup correctly, and must be installed separately on a user’s system, e.g., on the command line or through an app. On the other hand, web browsers present an attractive choice for local LLM deployments due to their integration into many users’ daily tasks ( 69 ) as well as their ease of use and lack of external software dependencies. New frameworks like WebGPU ( 63 ) enable GPU acceleration within the browser, opening up opportunities for interactive, low-latency usage of language models. Several existing frameworks such as Transformers.js ( 24 ) , WebLLM ( 57 ) , and wllama ( 48 ) target browser-based inference, with Transformers.js and WebLLM supporting GPU acceleration through ONNX Runtime ( 43 ) and MLC-LLM ( 46 ) , respectively. However, existing browser-based systems are limited by several issues: • Memory Inefficiencies : Existing browser inference engines dynamically allocate GPU memory and load weights into WebGPU inefficiently, leading to slowdowns as memory grows over the course of execution and even crashes as some browsers enforce hard-caps on memory per-tab. • Lack of Performance-Portable Design: Existing frameworks have limited kernel specialization for different devices, a critical limitation as WebGPU provides functional portability but not performance portability. Maximizing performance on the diverse hardware targeted by browsers motivates the need for interfaces that allow kernels to be tuned independently and easily integrated into inference engines. • Limited Quantization Support: To fully realize the opportunity of running models locally, inference engines must support diverse quantization formats that allow upstream model developers to experiment with strategies like quantization aware training and dynamic weight quantization. Kernels that operate on quantized models must be performant while allowing new formats to be added as needed. Taken together, these limitations hinder the adoption of cross-platform and browser-based LLM inference, motivating the need for new research into techniques to overcome them. 1.1. LlamaWeb In this work, we introduce LlamaWeb, a WebGPU backend for llama.cpp that runs many models supported by llama.cpp with GPU acceleration in the browser. Our approach is guided by three key constraints: (1) memory usage must be minimal and statically allocated, reducing overhead and allowing useful models to run in the browser on edge devices while avoiding unexpected slowdowns and crashes, (2) kernel libraries must be flexible and extensible to support performance portability across WebGPU’s heterogeneous hardware targets, and (3) support for diverse quantization formats needs to be a first-class concern. By building within llama.cpp, LlamaWeb also benefits from an active open-source ecosystem; for example, the first row in Tab. 1 shows the number of models available in llama.cpp’s GGUF format on Hugging Face, as compared to models available in other WebGPU frameworks’ formats. In turn, our work extends this ecosystem to the browser, enabling the broad collection of models, quantization strategies, and runtime features developed for llama.cpp to execute efficiently on WebGPU-enabled devices without requiring platform-specific native deployments. Memory-Efficient Local Inference A central design goal of our system is minimizing memory overhead during inference. To this end, we statically allocate all memory necessary to run the model at startup, including intermediate data structures for operations like FlashAttention and a fixed-size slotted memory region used for kernel parameters. We also optimize model loading and data movement, avoiding unnecessary materialization of model weights and streaming them asynchronously to GPU memory. In Sec. 5 , we compare memory consumption across multiple devices and two browsers (Chrome and Safari) and show that these strategies reduce peak memory usage by 29–33% on average compared to existing browser-based inference frameworks. Performance-Portable Kernel Library Maximizing performance across diverse GPU architectures requires adaptable kernel implementations. We develop a kernel library enabling templated shader generation with device-specific optimization and performance-portable parameters. For example, parameters like tile sizes for matrix multiplication can be specialized for different devices or use performance-portable values. Kernels can also incorporate advanced features like subgroup operations or subgroup matrix instructions when supported, while falling back to more portable implementations otherwise. At the execution level, we implement efficient grouping of kernel submission to the GPU and avoid unnecessary CPU-GPU synchronization. In Sec. 6 , we evaluate this system across a diverse set of models and devices, running 10 models across 16

Read the complete text