Compiler-Assisted Speculative Sampling for Accelerated LLM Inference on Heterogeneous Edge Devices ∗ Work realized during internship at NXP Semiconductors.
Alejandro Ruiz y Mesa ∗ 2 , Guilherme Korol 3 , Moritz Riesterer 3 , João Paulo C. de Lima 2 4 , Jeronimo Castrillon 2 4 · 2026
Read Compiler-Assisted Speculative Sampling for Accelerated LLM Inference on Heterogeneous Edge Devices ∗ Work realized during internship at NXP Semiconductors. by Alejandro Ruiz y Mesa ∗ 2 , Guilherme Korol 3 , Moritz Riesterer 3 , João Paulo C. de Lima 2 4 , Jeronimo Castrillon 2 4 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-08060.txt.
Description
Running language models on edge devices demands low latency, and speculative decoding, drafting tokens quickly then verifying them, helps, but integrating it into compiler workflows on heterogeneous chips is tricky. Alejandro Ruiz y Mesa and four co-authors address both problems with an analytical cost model that guides partitioning of model subgraphs across a system-on-chip's varied processors.
The model predicts when speculative sampling plus heterogeneous execution jointly pay off, especially for the short inputs typical at the edge. It is validated on a device with a hexacore ARM CPU and a Mali GPU. Edge AI engineers gain a sound way to decide how to split work across cores and accelerators instead of guessing, a step toward safe, responsive on-device language models.
Catalog record
| CREATOR | Alejandro Ruiz y Mesa ∗ 2 , Guilherme Korol 3 , Moritz Riesterer 3 , João Paulo C. de Lima 2 4 , Jeronimo Castrillon 2 4 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-08060