Non-Monotonic Latency in Apple MPS Decoding: KV Cache Interactions and Execution Regimes

Willy Fitra, Hendria · 2026

Read Non-Monotonic Latency in Apple MPS Decoding: KV Cache Interactions and Execution Regimes by Willy Fitra, Hendria (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08913.txt.

Introduction

Willy Fitra Hendria logs strange lag spikes when models write text on Apple's graphics back end. Write speed is thought to grow in a smooth way with length. On three model lines, Hendria finds bands where lag jumps up to near twenty one fold. It then drops back at near settings. Set tests place the flaw in the write phase, not the read in phase. Memory strain on its own does not explain it. The flaw is not seen on CPU or on rival graphics stacks.

Key value cache still helps in the main. Its gain falls hard in the bad bands. With cache off, jumps stay in part. Back ends hide modes that coarse tests miss.

Catalog record

CREATORWilly Fitra, Hendria
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.08913

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Non-Monotonic Latency in Apple MPS Decoding: KV Cache Interactions and Execution Regimes Willy Fitra, Hendria Abstract Autoregressive inference is typically assumed to scale predictably with decoding length, with latency increasing smoothly as generated sequence length grows. In this work, we identify unexpected non-monotonic latency behavior in the Apple MPS backend, where latency changes abruptly across nearby decoding configurations during transformer decoding. Using multiple model families (GPT-2, BLOOM, and OPT), we observe latency spikes of up to 21 × 21\times within specific decoding-budget intervals, followed by recovery at neighboring configurations. Controlled experiments show that these anomalies originate primarily during the decode phase rather than prefill, are not explained by memory pressure alone, and remain absent on CPU and NVIDIA CUDA backends under identical conditions. We further show that key–value (KV) cache interacts strongly with these pathological execution regimes: KV caching remains beneficial overall, but its practical speedup collapses sharply within anomalous configurations, while cache-disabled decoding still exhibits residual non-monotonic behavior. These findings suggest that autoregressive decoding on MPS enters discrete execution regimes that are not captured by coarse-grained benchmarking, highlighting the importance of hardware-aware evaluation for long-context inference. 1 Introduction Autoregressive inference is commonly assumed to scale monotonically with decoding length, and key–value (KV) caching ( 12 ) is widely used to accelerate autoregressive decoding ( 8 ) . Prior work on Apple Silicon LLM inference has focused on system-level benchmarking and optimization ( 15 ; 1 ; 5 ; 3 ) , typically reporting aggregate latency or throughput measured over coarse decoding-budget sweeps or fixed evaluation points. While useful for system comparison, such evaluations often average over or sparsely sample decoding budgets, which can obscure non-local effects in decoding behavior. However, autoregressive decoding ( 18 ) differs fundamentally from batch inference. Generation proceeds token by token, and the KV cache grows dynamically with accumulated decoding context, producing evolving memory layouts and access patterns that are not captured by coarse-grained evaluations. This raises the possibility that performance may change non-smoothly across nearby decoding configurations, rather than following the monotonic trends typically assumed in system-level analyses. A natural systems question is: when does MPS outperform CPU? While we reproduce a model-size-dependent crossover, deeper analysis reveals that this question alone is insufficient to characterize MPS decoding behavior. Instead, we uncover a more critical phenomenon: autoregressive decoding on MPS exhibits non-monotonic latency scaling, with abrupt latency regimes emerging under specific decoding configurations and particularly pronounced under KV cache. Figure 1 previews the observed behavior. Under MPS execution, latency can increase abruptly by more than an order of magnitude before returning to lower-latency regimes at neighboring configurations. In contrast, CPU and CUDA baselines remain smooth and monotonic under identical experimental conditions. Figure 1 : Latency scaling by model and backend. Each panel shows average latency versus decoding budget ( max_tokens ) for one model. CPU and CUDA exhibit smooth monotonic scaling across all tested lengths, while MPS shows sharp non-monotonic spikes for GPT-2 Medium and GPT-2 Large at specific decoding budgets. We find that KV cache interacts with these anomalous latency regions, affecting both the magnitude of latency spikes and the collapse of speedup at pathological decoding configurations. Through controlled experiments, we demonstrate that (1) the instability is not explained by model size or memory limits alone, (2) it emerges during decoding rather than prefill, and (3) KV cache remains nominally faster at all tested lengths, but its practical advantage is largely neutralized at pathological decoding configurations (speedup collapses from 4.9 – 20.3\times to 1.9\times ), while cache-off runs still show residual non-monotonicity. These findings suggest that backend execution behavior contributes to performance variations in specific regimes that are not captured by coarse benchmarks. Contributions. • We identify non-monotonic latency behavior in MPS autoregressive decoding, with spikes up to 21\times at model-dependent decoding budgets, while CPU and CUDA T4 baselines remain strictly monotonic under identical conditions. • We demonstrate that this behavior generalizes across multiple model families (GPT-2, BLOOM, and OPT), indicating a backend-level phenomenon rather than model-specific effects. • We demonstrate sharp regime transitions in MPS decoding, where latency jumps abruptly within narrow decoding-budget intervals despite substantially smoother observed memory scaling. • We show that KV cache highlights execution sensitivity, revealing a speedup reduction (from 4.9 – 20.3\times to 1.9\times ) at pathological decoding configurations, while not removing the underlying non-monotonic structure. • We analyze the sensitivity of the anomalous regime to prompt length and decoding budget, showing that nearby decoding configurations can exhibit sharply different latency behavior under MPS decoding. 2 Related Work 2.1 Attention Efficiency and Sequence-Length Optimization A large body of work focuses on improving the efficiency of transformer attention mechanisms under long-context inference. KV cache reduces recomputation during autoregressive decoding while increasing memory usage with accumulated decoding context ( 12 ; 8 ; 22 ; 19 ) . Complementary kernel-level optimizations such as FlashAttention ( 4 ) improve efficiency through memory-aware attention computation and IO-aware kernel design. While these methods significantly improve efficiency, they generally assume monotonic cost growth with decoding length. That is, latency is expected to increase smoothly as decoding budget increases, reflecting predictable memory and compute scaling in modern accelerators. In contrast, our results show that this assumption does not hold on the Apple MPS backend, where autoregressive decoding exhibits non-monotonic latency behavior. 2.2 Language Model on Apple Silicon Recent work has explored production-grade runtime comparisons across local inference stacks ( 15 ) , alongside weight streaming approaches ( 1 ) , KV-cache optimization for edge inference ( 6 ) , and profiling studies of Apple Silicon performance ( 5 ; 3 ) . While these studies provide valuable system-level insights, they primarily report aggregate latency or throughput under coarse decoding-budget sweeps. The PyTorch MPS backend ( 13 ) enables GPU inference via Metal Performance Shaders ( 2 ) , but its internal execution behavior is not directly exposed through the PyTorch API, making performance characteristics difficult to fully model or predict. To the best of our knowledge, existing evaluations do not systematically examine fine-grained decoding-budget variations or potential non-monotonic latency regimes in autoregressive decoding. This motivates our study of MPS execution behavior under controlled decoding settings. 3 Experimental Setup Unless otherwise stated, all experiments follow the default configuration described in this section. Any deviations, such as changes in hardware, instrumentation, or ablation settings, are explicitly specified in the relevant subsection and are not mixed across experiments. 3.1 Experimental Configuration Experiments are conducted on a MacBook Pro equipped with an Apple M3 Max processor (14-core CPU, 30-core GPU, 36 GB unified memory) running macOS 14.8.1 (Sonoma). The primary software stack includes Python 3.9.6, PyTorch 2.8.0 ( 11 ) , HuggingFace Transformers 4.57.6 ( 20 ) , and LitServe 0.2.16 ( 9 ) , using the MPS backend for GPU-accelerated execution. To assess backend stability, we validated that qualitatively similar non-monotonic behavior persists across multiple PyTorch versions (2.7.0, 2.8.0, and 2.11.0), indicating the phenomenon is not tied to a specific release. Key findings were independently reproduced on an Apple M3 Pro (11-core CPU, 8-core GPU, 18 GB unified memory) under identical conditions, demonstrating the cross-device generality of the observed phenomena. We evaluate transformer language models across multiple families, including GPT-2 ( 14 ) , BLOOM ( 17 )

Read the complete text