Accelerating Mobile Inference through Fine-Grained CPU-GPU Co-Execution
Zhuojin Li, Marco Paolieri, Leana Golubchik · 2025
Read Accelerating Mobile Inference through Fine-Grained CPU-GPU Co-Execution by Zhuojin Li, Marco Paolieri, Leana Golubchik (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-21081.txt.
Introduction
Zhuojin Li leads two co-authors on accelerating neural network inference on mobile devices by splitting work finely between the central and graphics processors. Phones have limited compute, yet their shared memory and narrow performance gap between the two chips invite cooperation.
Two obstacles dominate, the synchronization cost of merging partial results and the difficulty of predicting how long each task takes on each chip when implementations and parallelism shift dynamically. Their fine grained co execution scheme attacks both problems. Lower latency on device inference matters for translation, vision, and assistants that should work without the cloud.
Catalog record
| CREATOR | Zhuojin Li, Marco Paolieri, Leana Golubchik |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-21081
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Accelerating Mobile Inference through Fine-Grained CPU-GPU Co-Execution Zhuojin Li, Marco Paolieri, Leana Golubchik Abstract Deploying deep neural networks on mobile devices is increasingly important but remains challenging due to limited computing resources. On the other hand, their unified memory architecture and narrower gap between CPU and GPU performance provide an opportunity to reduce inference latency by assigning tasks to both CPU and GPU. The main obstacles for such collaborative execution are the significant synchronization overhead required to combine partial results, and the difficulty of predicting execution times of tasks assigned to CPU and GPU (due to the dynamic selection of implementations and parallelism level). To overcome these obstacles, we propose both a lightweight synchronization mechanism based on OpenCL fine-grained shared virtual memory (SVM) and machine learning models to accurately predict execution times. Notably, these models capture the performance characteristics of GPU kernels and account for their dispatch times. A comprehensive evaluation on four mobile platforms shows that our approach can quickly select CPU-GPU co-execution strategies achieving up to 1.89x speedup for linear layers and 1.75x speedup for convolutional layers (close to the achievable maximum values of 2.01x and 1.87x, respectively, found by exhaustive grid search on a Pixel 5 smartphone). BibTeX Citation: @inproceedings{LiPG26, author = {Zhuojin Li and Marco Paolieri and Leana Golubchik}, title = {Accelerating Mobile Inference through Fine-Grained {CPU}-{GPU} Co-Execution}, booktitle = {Selected Papers of {EPEW} 2025}, series = {Lecture Notes in Computer Science}, volume = {To appear}, pages = {--}, publisher = {Springer}, year = {2026}, doi = {} } 1 Introduction Machine learning (ML) techniques have achieved rapid growth in recent years, driven by the breakthroughs in large-scale architectures such as Large Language Models (LLMs) and Large Vision Models (LVMs). These state-of-the-art models exhibit impressive capabilities in a wide range of applications, including question answering, image recognition, and video analysis. While these models are typically trained on powerful cloud servers, there is a growing demand for deployment on mobile platforms to serve inference tasks, since on-device deployment not only protects user privacy, but also provides offline availability and reduces latency for real-time applications. However, deploying large-scale models on mobile platforms remains challenging, due to limited computational resources and energy constraints. To improve inference latency on mobile platforms, substantial research and industrial efforts have focused on developing efficient neural network architectures [ 16 ] and specialized hardware [ 7 ] . On top of these efforts, our work explores an additional dimension: distributing fine-grained operations (e.g., forward propagation of a neural network layer) across heterogeneous computing devices (specifically, CPU and GPU) available on modern mobile platforms. In particular, we pursue this approach by leveraging the following opportunities. (a) Unified Memory (b) GPU Workgroup Figure 1 : Introduction to Mobile Platforms The unified memory architecture on mobile platforms enables both the main processor (CPU) and specialized accelerators (e.g., GPU and NPU) to directly access shared areas of main memory, as illustrated in Fig. 1(a) . In contrast, GPUs of cloud servers maintain separate on-chip memories and require data transfers to share data with the CPU through the main memory. By eliminating the need for memory transfers to main memory, unified memory on mobile platforms facilitates collaborative execution of inference tasks across compute devices. Figure 2 : Comparison of CPU and GPU Latencies for Linear Operations with Input Shape (50, 3072), with 95% Confidence Intervals (OnePlus 11) Figure 3 : Inadequate Modeling of GPU Latency Spikes by Existing Methods for Linear Operations with Input Shape (50, 768) (OnePlus 11) Moreover, in contrast with existing works [ 8 , 19 ] which report poor performance on mobile CPUs (due to inefficient CPU implementations in some ML libraries such as OpenVINO [ 6 ] ), our empirical analysis shows that the performance gap between mobile CPUs and GPUs can be narrow for important operations when using the XNNPACK [ 4 ] library in TensorFlow Lite (TFLite), which provides high-performance implementations based on advanced SIMD instructions for ARM CPUs. For instance, as illustrated in Fig. 2 , for matrix multiplications (i.e., a linear layer in a neural network) of sizes 50\times 3072 and 3072\times C_{out} , a CPU implementation with 3 threads achieves even lower latency than a GPU kernel when C_{out}<425 . The significant performance of mobile CPUs motivates us to assign compute tasks to both CPU and GPU to achieve lower inference latency through parallel execution. However, designing an effective co-execution strategy across multiple accelerators remains challenging. In our empirical evaluation, existing approaches achieve limited performance gains mainly due to the following two reasons. First, optimal workload partitioning across CPUs and GPUs is difficult, primarily because GPU kernels exhibit complex non-linear performance characteristics. Our experiments, reported in Fig. 3 , show significant latency spikes (green line) for linear operations as the output dimension C_{out} increases, due to heuristic choices in GPU kernel implementations and in the assignment of tasks to workgroups (i.e., to groups of threads executing the same code on different subsets of the input data, as illustrated in Fig. 1(b) ). Consequently, co-execution frameworks relying on linear models for GPU latency prediction (e.g., [ 2 ] ) can make poor partitioning decisions. Although non-linear ML models based on operation configurations (e.g., input/output channels) have been proposed for more accurate latency predictions [ 9 , 22 , 15 , 13 ] , our evaluations in Fig. 3 indicate that these methods 1 1 1 GBDT hyperparameters are detailed in Section 5.2 . The MLP predictor was selected by varying the number of layers (1 to 4), neurons at each layer (32, 64 or 128), dropout rate (0 to 0.5), learning rate ( 10^{-5} to 0.1 ) and weight decay ( 10^{-6} to 10^{-3} ). still fail to capture sudden latency spikes for specific configurations. Second, synchronization overhead between compute devices is often non-negligible. Existing work [ 9 ] reports overhead of up to 1 ms for the GPU to notify the CPU that data mapping of a shared memory region has completed; as a comparison, our measurements indicate that, using the GPU on the OnePlus 11 smartphone, the longest linear operation of the ViT-Base-32 neural network [ 3 ] takes only 660 \mu s. Such overhead can completely wipe out the benefits of co-execution; thus, minimizing synchronization overhead is crucial. Our main contributions to tackling these problems are as follows: • We develop accurate latency predictors that use kernel implementation details and kernel dispatch behaviors from TFLite ( Section 3 ). Using detailed information, our predictors accurately capture complex latency characteristics, including the discontinuity due to heuristic workgroup choices and kernel selection, enabling more effective workload partitioning decisions. • To address significant synchronization overhead, we use fine-grained shared virtual memory in OpenCL to implement an efficient CPU-GPU synchronization mechanism on mobile platforms ( Section 4 ). Our design reduces expensive data mapping operations required for cache coherence and avoids notification delay through active querying. As a result, synchronization overhead is substantially reduced, e.g., from 162 \mu s to 7 \mu s for linear operations on a Motorola Edge Plus 2022 smartphone. • We create a comprehensive dataset consisting of latency measurements of 2,039 linear and 2,051 convolution operations executed on four mobile devices using co-execution strategies with 1 to