Challenging GPU Dominance: When CPUs Outperform for On-Device LLM Inference
Haolin Zhang, Jeff Huang · 2025
Read Challenging GPU Dominance: When CPUs Outperform for On-Device LLM Inference by Haolin Zhang, Jeff Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-06461.txt.
Introduction
Haolin Zhang and Jeff Huang challenge a common belief about phone AI in a paper from 2025. Most people assume graphics chips always beat central chips for running large language models. The pair tested that claim on an iPhone 15 Pro with a one-billion-parameter model.
Running on just two CPU threads, the phone made seventeen tokens per second. The graphics chip setup made fewer, at under thirteen per second. Moving data to the graphics chip cost more time than its raw speed saved. The authors also test thread counts and number formats. Their advice helps builders pick the right chip for small, local AI.
Catalog record
| CREATOR | Haolin Zhang, Jeff Huang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-06461
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Challenging GPU Dominance: When CPUs Outperform for On-Device LLM Inference Haolin Zhang, Jeff Huang Abstract The common assumption in on-device AI is that GPUs, with their superior parallel processing, always provide the best performance for large language model (LLM) inference. In this work, we challenge this notion by empirically demonstrating that, under certain conditions, CPUs can outperform GPUs for LLM inference on mobile devices. Using a 1-billion-parameter LLM deployed via llama.cpp on the iPhone 15 Pro, we show that a CPU-only configuration (two threads, F16 precision) achieves 17 tokens per second, surpassing the 12.8 tokens per second obtained with GPU acceleration. We analyze the architectural factors driving this counterintuitive result, revealing that GPU memory transfer overhead and CPU thread optimization play a critical role. Furthermore, we explore the impact of thread oversubscription, quantization strategies, and hardware constraints, providing new insights into efficient on-device AI execution. Our findings challenge conventional GPU-first thinking, highlighting the untapped potential of optimized CPU inference and paving the way for smarter deployment strategies in mobile AI. However, fully explaining the observed CPU advantage remains difficult due to limited access to low-level profiling tools on iOS. Keywords On-Device AI \cdot SLM \cdot LLM Inference \cdot CPU Optimization \cdot ARM Architecture \cdot Mobile Machine Learning 1 Introduction Recent advances in small language models (SLMs) have challenged the conventional focus on scaling up AI models. While GPUs have been the dominant choice for inference, emerging evidence suggests that they may not always offer the best performance, particularly for SLMs on mobile devices [ 5 ] . This shift is reshaping AI deployment, as compact models demonstrate high efficiency across diverse tasks. Models such as Phi [ 1 , 13 ] , LLaMA 3.2 Herd [ 10 ] , TinyLLaMA [ 32 ] , Apple Foundation Models [ 11 ] , and Qwen-2 [ 31 ] exemplify this trend. They show that smaller architectures can achieve strong performance in natural language processing, expanding the possibilities for real-world applications beyond large-scale cloud infrastructure. Deploying SLMs on mobile devices presents unique challenges due to constraints in memory, processing power, and energy consumption. Researchers have proposed optimizations to address these limitations. Liu et al. [ 19 ] introduced MobileLLM, an architecture for sub-billion parameter models that optimizes embedding sharing, grouped-query attention, and deep-thin structures to minimize resource overhead. Pham et al. [ 23 ] developed SlimLM, fine-tuned for document assistance on smartphones, balancing model size, context length, and inference speed. Li et al. [ 17 ] presented Transformer-Lite, an inference engine that optimizes mobile GPU efficiency with dynamic shape support, FP4 quantization, and sub-tensor KV cache handling, achieving notable speedups. These advancements have fueled interest in on-device SLM deployment, reducing reliance on cloud services, improving privacy, and lowering latency and costs. A key tool in this domain is llama.cpp [ 9 ] , an open-source framework optimized for consumer hardware, including mobile devices. It supports various performance optimizations, such as quantization [ 8 ] and multi-threaded execution, making it a valuable platform for pushing the limits of on-device AI. This paper examines the performance of SLMs on mobile devices. We evaluate six models, ranging from 0.5 billion to 8 billion parameters, under different precision settings to assess the impact of quantization and hardware configurations on inference speed. Given the iPhone’s prominence in the premium market and Apple’s growing investment in on-device AI [ 12 ] , we selected the iPhone 15 Pro as our test platform. It features an Apple-designed A17 Pro SoC with an ARM v8.6-A ISA and a Metal-supported GPU. We use llama.cpp as the inference engine for our experiments. Our study highlights three key findings: • CPU Competitiveness with GPU: Under specific conditions, CPU-only inference can match or even surpass GPU performance. For smaller models such as Qwen2-0.5B and LLaMA-3.2-1B, multi-threaded CPU execution with Q4 1 1 1 Effective 4.5 bits/weight and F16 precision achieves speedups of 1.31 \times and 1.33 \times , respectively, over GPU execution. However, for models larger than 1.5B parameters, GPUs generally maintain higher throughput, as seen with Mistral-7B and LLaMA-3.2-8B, where CPU performance declines due to increased memory and computation demands. • Matrix Multiplication as the Bottleneck: Profiling reveals that matrix multiplication ( GGML_OP_MUL_MAT ) dominates computation, accounting for 87.6% of execution time in the prefill phase and 76.2% in the decode phase 2 2 2 For llama3.2-1B@f16 . This underscores the need to optimize General Matrix Multiplication (GEMM) operations to improve SLM inference efficiency. • Impact of Thread Allocation: Optimal performance occurs when the number of CPU threads matches the architecture’s performance cores. Adding more threads beyond this threshold yields diminishing returns or performance degradation, emphasizing the importance of architecture-aware thread scheduling. Through these insights, we challenge the dominant “GPU-first” paradigm, demonstrating that CPUs can be competitive for on-device AI. Our findings contribute to the development of resource-efficient, high-performance AI systems. The rest of the paper is organized as follows. Section 3 describes the llama.cpp compute graph and its role in model inference. Section 5 presents a performance analysis of models from 0.5B to 8B parameters across different backends and precision settings. Section 6 discussed profiling results from llama.cpp runtime. Section 7 proposes a hardware-aware execution strategy to maximize concurrency. Finally, Sections 8 – 9 discuss profiling challenges on mobile hardware and future directions for optimizing both CPU and GPU performance. 2 Related Work 2.1 Advancements in Small Language Models Research on small language models (SLMs) has aimed to achieve strong performance with lower computational costs, making them practical for mobile and edge applications. Several models have been introduced, including Meta’s OPT [ 33 ] and LLaMA series [ 28 , 10 ] , Microsoft’s Phi series [ 13 , 1 ] , the Qwen series [ 31 ] , StabilityAI’s StableLM [ 3 ] , and Google’s Gemma [ 27 ] and Gemini Nano [ 26 ] . Other models, such as MobileVLM [ 4 ] and Apple Foundation Models [ 11 ] , have been designed with efficiency as a priority for resource-limited environments. One key improvement in modern SLMs is the refinement of self-attention mechanisms, which traditionally scale quadratically with sequence length. Methods such as flash attention [ 6 ] optimize memory use by reducing redundant operations, improving inference speed. Grouped-query attention (GQA) [ 2 ] , used in the Qwen and LLaMA 3.2 series, reduces memory overhead by processing queries in groups, offering efficiency without sacrificing accuracy. Compression methods such as quantization further improve inference by reducing memory and bandwidth needs. Low-bit approaches, including GPTQ [ 7 ] and llama.cpp’s k-quant [ 8 ] , minimize storage and computational requirements while maintaining precision. Recent work, such as SmoothQuant [ 29 ] , extends these benefits by introducing weight-activation co-quantization to lower computational load while maintaining stable outputs. These developments improve the balance between efficiency and accuracy, making smaller models a practical alternative for on-device use. However, the efficiency of CPUs compared to GPUs for running these models under different quantization settings and model sizes is still not well understood. This study addresses this question by systematically evaluating CPU-GPU performance trade-offs across multiple configurations. 2.2 On-Device Deployment of SLMs Efficient on-device inference has led to the development of specialized engines, each