Profiling Large Language Model Inference on Apple Silicon: A Quantization Perspective DOI: XXXXXXX.XXXXXXX Journal: JACM Volume: 37 4 111 8
Afsara Benazir, Felix Xiaozhu Lin · 2025
Read Profiling Large Language Model Inference on Apple Silicon: A Quantization Perspective DOI: XXXXXXX.XXXXXXX Journal: JACM Volume: 37 4 111 8 by Afsara Benazir, Felix Xiaozhu Lin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-08531.txt.
Introduction
Afsara Benazir and Felix Xiaozhu Lin wrote this 2025 arXiv study benchmarking large language models on Apple computers. Research on efficient inference has centered on NVIDIA graphics cards, leaving Apple's unified memory chips, which share one pool between processor and graphics, poorly understood. The team profiled five hardware setups, from two Apple M-series machines to multi-GPU rigs, across models from eight to four hundred five billion parameters and fourteen compression schemes.
Low-level profiling exposed where time really goes, including the hidden cost of unpacking compressed weights. One myth falls: squeezing models to fewer bits does not guarantee faster inference everywhere. The huge unified memory, though, lets Apple machines host very large models cost-effectively beside the GPU rigs. The authors claim the first thorough characterization of Apple Silicon for on-device inference, aimed at practitioners choosing local hardware.
Catalog record
| CREATOR | Afsara Benazir, Felix Xiaozhu Lin |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-08531
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Profiling Large Language Model Inference on Apple Silicon: A Quantization Perspective DOI: XXXXXXX.XXXXXXX Journal: JACM Volume: 37 4 111 8 Afsara Benazir, Felix Xiaozhu Lin Abstract . A systematic understanding of Apple Silicon is lacking in the current landscape of hardware efficiency; research focus is largely centered on accelerating GPUs for large-scale training or inference on CUDA devices. This paper investigates Apple Silicon’s unique memory architecture that offers a unified memory integrating CPU and GPU memory and its implications for on-device LLM inference. We decipher myths about whether Apple Silicon is efficient for on-device inference compared to competitors such as NVIDIA GPUs by directly conducting latency and throughput comparison benchmarks. We explain the performance gap between them through profiling low level hardware metrics - ALU utilization, memory bandwidth, buffer usage, cache residency etc. at runtime. We draw several insights regarding performance bottlenecks such as dequantization overhead, compute throughput and memory bandwidth. We debunk existing false claims regarding large language model inference such as compressing models to lower bit precision is a defacto promise for faster inference across all hardware platforms. We find that the large unified memory enables Apple Silicon to be both cost effective and efficient against NVIDIA GPUs for ultra large language models. Our work provides a comprehensive perspective of on-device inference on commodity GPUs; our large scale evaluation on 5 hardware testbeds incorporating three Apple M-series devices: M2 Ultra, M2 Max and M4 Pro and two NVIDIA GPUs: NVIDIA RTX A6000, a multi GPU setup with 2xNVIDIA RTX A6000, 5 model scales ranging from 8B to 405B parameters and 14 quantization schemes gives an understanding of how Apple Silicon fits within the paradigm of on-device LLM inference. Our analysis reveals multiple resource interdependencies and unexpected findings, while also quantifying established insights. To the best of our knowledge, this study makes the first attempt to present a thorough characterization and analysis of Apple Silicon for on-device inference. 1. Introduction The emergence of billion scaled large language models (LLMs) and their higher reasoning capabilities has lead to a profound paradigm shift in the growth of artificial intelligence ( Minaee et al. 2024 ) . On-device execution of large LLMs is lucrative as it promises enhanced privacy, data localization, user control of data and allows for versatile personalized applications ( Xu et al. 2024 ) . With the recent advancement in machine learning (ML) accelerators, GPUs can provide peak performance; a limiting factor in accessing the supreme capabilities of these GPUs are their increasingly high cost that can be measured in terms of $$ per token generated ( Shekhar et al. 2024 ) . Although cloud providers offer on-demand GPU instances, purchasing GPUs are more cost-effective in the long run ( Strubell et al. 2020 ) . A significant constraint for LLM inference is the substantial demand for GPU memory - loading tens of GB of static model parameters alongside storing the intermediate activations and KV cache in limited GPU memory is inefficient. Devices like Apple Mac Studio ( Apple 2022 ) can support upto 192 GB of unified memory making it a prime candidate for hosting LLMs. To date, 22.9 million units of Apple Silicon chips are in circulation ( mac ) , which is on the rise. Although NVIDIA GPUs are dominating the market with their higher compute utilization and tensor cores dedicated for batch processing in LLM training and finetuning ( Cheng et al. 2024 ) , their contemporary Apple Silicon (ref. § 2.1 ) is overlooked owing to its reported lower GFLOP count ( Hübner et al. 2025 ) , despite its availability of a large unified memory. For a private LLM serving workstation intended for personal use or by a small user cohort, two factors are critical - (1) inference speed - the number of tokens generated per second and (2) cost effectiveness: how much money’s worth one can get out of a single inference run, measured through $/token. For LLM training and finetuning, tokens are processed in batches, where GPU shines owing to its supreme parallel processing power. LLM inference is autoregressive - during prefill phase , all input prompt tokens are processed in parallel to generate the very first token, but in the decode phase each subsequent token is generated conditioned on the previous one. When hosting an LLM engine that serves a number of parallel requests such as in a cloud server, batched processing can be done but for single request model serving, typical for on-device inference scenarios, decoding is truly sequential. A higher parallel processing capability (i.e higher throughput) alone does not ensure faster token generation speed as (1) for a single request LLM inference, the decode phase is memory bound ( Kim et al. 2023 ) , efficient compute units are left idle as data transfer happens (2) For inference of extremely/ultra-large language models in the range of 70B-405B parameters, a challenge is to fit the model in the GPU VRAM to be efficiently processed; a limitation for current consumer grade GPUs having a limited 24GB-48 GB VRAM ( Song et al. 2024 ) . One solution is to lower the memory traffic during each decoding pass, thereby boosting generation speed. Weights in LLM models are typically represented in high precision formats (FP16/FP32), representing these weights using fewer bits sharply reduce that traffic. Post-training quantization (PTQ) ( Yao et al. 2024 ) shines here owing to its capability to produce coherent output at extremely low bits per weight. But translating the memory reduction achieved through quantization to latency improvement can be challenging as it requires significant engineering efforts ( Jin et al. 2024 ) . Motivation Our motivation stems from the unified memory offered by Apple Silicon at lower cost per million tokens generated (ref. § 4 ) which is attractive for on-device inference. The bigger memory pool allows for effective data exchange, dynamic allocation and reduced overhead (no data duplication); even if raw compute power is lower ( Kenyon and Capano 2022 ) , it avoids the performance penalties associated with slow offloading to system RAM that discrete GPUs with comparatively lower VRAM might encounter. This calls for a closer examination of Apple Silicon’s strengths and weaknesses, and evaluating its potential as a strong competitor for on-device ML inference. Goal A lack of systematic understanding of LLM inference on Apple Silicon exists - prior work has primarily focused on LLM training ( Cheng et al. 2024 ) and on-device LLM inference on NVIDIA GPUs ( Li et al. 2024 ; Lin et al. ; Luo et al. 2024 ; Song et al. 2024 ) or ARM CPUs ( Gope et al. 2024 ) . To address this gap in literature we conduct a principled empirical study of Apple M-series devices under 26 different model precisions for absolute runtime latency, cost-effectiveness and to understand the impact of hardware characteristics on efficiency. Our observations are reported as a series of "findings" for ML practitioners to make informed choices in LLM serving. We identify the following important research questions (RQ): • RQ1: How well does Apple Silicon support end-to-end LLM inference on-device? • RQ2: Is Apple Silicon a cost effective choice for on-device LLM inference compared to other hardware accelerators such as