Open-TQ-Metal: Fused Compressed-Domain Attention for Long-Context LLM Inference on Apple Silicon
2026
Read Open-TQ-Metal: Fused Compressed-Domain Attention for Long-Context LLM Inference on Apple Silicon (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-16957.txt.
Introduction
Vegasena presents Open-TQ-Metal, the first fused compressed-domain attention system for Apple Silicon. It quantizes the key-value cache to four-bit values on the fly and computes attention straight from the compressed form, removing dequantization steps. On a single 64-GB consumer Mac, it runs a 70-billion-parameter model at a 128,000-token context. Attention runs 48 times faster than the dequantize-then-attend baseline, while cache memory shrinks from 40 GB to 12.5 GB.
Catalog record
| YEAR | 2026 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25519 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.16957
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Open-TQ-Metal: Fused Compressed-Domain Attention for Long-Context LLM Inference on Apple Silicon Abstract We present Open-TQ-Metal, the first implementation of fused compressed-domain attention on Apple Silicon, enabling 128K-context inference for Llama 3.1 70B on a single 64GB consumer Mac—a configuration impossible with all existing inference frameworks. Running large language models at their full designed context window on consumer hardware is blocked by KV cache memory: Llama 3.1 70B at 128K context requires 79.1 GB in FP16, exceeding the 64 GB physical limit. Open-TQ-Metal quantizes the KV cache to int4 on the fly and computes attention directly on the compressed representation via custom Metal compute shaders, eliminating all intermediate dequantization matrices. Across 330 experiments spanning two model families (Gemma 4 31B and Llama 3.1 70B), the fused sdpa_int4 kernel achieves 48 × \times attention speedup at 128K context over the dequantize-then-attend baseline, reduces KV cache memory from 40 GB to 12.5 GB (3.2 × \times compression), and maintains identical top-1 token predictions to FP16 inference. We further provide the first cross-architecture analysis of KV cache quantization methods, revealing that the attention scale factor—not model size—determines whether angular quantization schemes like PolarQuant succeed or fail, with Gemma 4’s attn_scale=1.0 amplifying directional error 25–100 × \times more than Llama’s standard 1 / d 1/\sqrt{d} scaling. Sai Vegasena Ensue [email protected] (a) Llama 3.1 70B at 128K context: FP16 KV requires 79.1 GB (infeasible on 64 GB). Int4 KV reduces total to 53.6 GB. (b) Fused sdpa_int4 kernel speedup on Llama 70B. Scales super-linearly from 3 \times at 1K to 48 \times at 128K. Figure 1 : The two core results of Open-TQ-Metal: (a) enabling 128K context on hardware where it was previously impossible, and (b) achieving super-linear attention speedup via fused compressed-domain computation. 1 Introduction Large language models achieve their strongest performance at long context windows ( Grattafiori et al. 2024 ) , yet deploying these models on consumer hardware is fundamentally constrained by KV cache memory ( Kwon et al. 2023 ) . The KV cache grows linearly with sequence length and model depth: for a transformer ( Vaswani et al. 2017 ) with L layers and H_{\text{kv}} key-value heads ( Shazeer 2019 ; Ainslie et al. 2023 ) , Llama 3.1 70B ( Grattafiori et al. 2024 ) at 4-bit weights ( Frantar et al. 2023 ) occupies 39.1 GB; its FP16 KV cache at 128K tokens adds 40 GB more—totaling 79.1 GB, which exceeds the 64 GB unified memory of Apple’s highest-capacity consumer chips. Every existing inference framework (mlx-lm ( Apple Machine Learning Research 2024 ) , llama.cpp ( Gerganov & contributors 2023 ) ) hits this wall. Recent KV cache quantization methods ( Liu et al. 2024 ; Hooper et al. 2024 ; Kang et al. 2024 ) compress the cache to 2–4 bits but still dequantize before attention, incurring the full bandwidth cost of materialized temporary matrices. TurboQuant ( Zandieh et al. 2025 ) combines MSE-optimal quantization with QJL residual compression, while PolarQuant ( Han et al. 2025 ) and QJL ( Zandieh et al. 2024 ) offer complementary approaches to KV cache compression. However, all three methods have only been evaluated on models up to 8B parameters (32 layers), and none have been validated on Apple Silicon’s Metal compute architecture. Their behavior at 70B (80 layers) and across architectures with non-standard attention scaling ( Gemma Team et al. 2024 ) is unknown. We address both gaps with Open-TQ-Metal, a complete inference system that implements fused compressed-domain attention in Metal compute shaders, building on the online softmax technique ( Milakov & Gimelshein 2018 ) and split-K parallelism from FlashDecoding ( Dao et al. 2023 ) . Our contributions: • Fused int4 SDPA kernel for Metal. A Metal compute shader that reads packed int4 keys and values directly from device memory, dequantizes per-element in GPU registers via bitwise operations, and computes attention with online softmax—producing zero intermediate matrices. At 128K context, this kernel is 48 \times faster than the dequantize-then-attend baseline ( Section 3.1 ). • 128K context on 64 GB hardware. By compressing the KV cache from 40 GB (FP16) to 12.5 GB (int4), Open-TQ-Metal fits Llama 3.1 70B at 128K context in 53.6 GB (including 2 GB runtime overhead)—with 10.4 GB headroom on a 64 GB Mac. Output quality is identical: both paths produce the same top-1 token under greedy decode ( Section 5 ). • Cross-architecture quantization analysis. Across 330 experiments on Gemma 4 31B and Llama 3.1 70B, we demonstrate that the attention scale factor is the critical variable for quantization method selection: PolarQuant fails on Gemma 4 ( attn_scale=1.0 ) but succeeds on Llama ( 1/\sqrt{d}=0.0884 ), with KL divergence differing by 25–100 \times ( Section 4 ). • Split-K parallelism via chained MLX Primitives. We solve the Metal dispatch race condition for long-context attention by implementing partial and reduce phases as separate MLX Primitives chained through the computation graph, achieving flat latency scaling from 1K to 128K tokens ( Section 3.2 ). All code, Metal shaders, and benchmarks are open-sourced. Figure 1 summarizes the two core results. 2 Background 2.1 KV Cache Compression Autoregressive inference stores key and value projections from all previous tokens in the KV cache ( Vaswani et al. 2017 ) . For a model with L layers, H_{\text{kv}} KV heads ( Shazeer 2019 ; Ainslie et al. 2023 ) , head dimension d , and sequence length S , the cache occupies 2LH_{\text{kv}}Sdb bytes, where b is bytes per element. At FP16 ( b=2 ), Llama 3.1 70B at 128K context requires: 2\times 80\times 8\times 128\text{K}\times 128\times 2\;=\;40\,\text{GB} (1) This single allocation exceeds the total memory of most consumer devices. Per-group asymmetric int4 quantization partitions each vector into groups of g consecutive elements and computes per-group scale and zero-point parameters: \displaystyle s \displaystyle=\frac{\max-\min}{15},\qquad z=\frac{-\min}{s} (2) \displaystyle q \displaystyle=\text{clamp}\!\left(\left\lfloor\frac{x}{s}+z\right\rceil,0,15\right),\qquad\hat{x}=(q-z)\cdot s (3) Values are packed two per byte (4 bits each). At g=32 , the effective compression is 3.2 \times including amortized scale and zero-point overhead. PolarQuant ( Han et al. 2025 ) transforms KV embeddings into polar coordinates via a recursive algorithm and quantizes the resulting angles. After random preconditioning, these angles exhibit a tightly concentrated distribution that eliminates the need for explicit normalization. The authors report over 4.2 \times compression. However, quantizing angles introduces structured angular error—a property we analyze in Section 4 . QJL ( Zandieh et al. 2024 ) projects keys through a random Gaussian matrix and stores only the sign bits of the result, yielding binary sketches in \{-1,+1\}^{m} . Attention scores are estimated asymmetrically: the query side remains in full precision while only keys are binarized. Compression exceeds 5 \times . 2.2 TurboQuant and Metal Compute TurboQuant ( Zandieh et al. 2025 ) proposes an MSE-optimal scalar quantizer based on random rotation and Lloyd-Max codebooks, with a 1-bit QJL transform applied to the residual. The paper reports quality neutrality at 3.5 bits per dimension and marginal degradation at 2.5 bits, evaluated on models up to 8B parameters. Apple Silicon GPUs use 32-wide SIMD groups as the basic execution unit. The MLX framework ( Apple Machine Learning Research 2024 ) provides a C++ Primitive system for dispatching custom Metal kernels. Critically, multiple dispatches within a single eval_gpu() call