Puzzle : Scheduling Multiple Deep Learning Models on Mobile Device with Heterogeneous Processors
Duseok Kang, Yunseong Lee, Junghoon Kim · 2025
Read Puzzle : Scheduling Multiple Deep Learning Models on Mobile Device with Heterogeneous Processors by Duseok Kang, Yunseong Lee, Junghoon Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-17764.txt.
Introduction
Duseok Kang, Yunseong Lee, and Junghoon Kim present Puzzle in 2025, a scheduler that runs many deep learning models at once on a phone. Phones now mix standard chips with neural chips, but most past work plans for one model at a time. Puzzle splits each network into parts and assigns parts, order, and rank across the chips.
A genetic search drives the plan, and real device timing keeps the cost guesses honest. Across trials with nine modern networks, Puzzle served 3.7 times the request rate of an NPU-only plan. It also beat the best fixed mapping by 2.2 times at the same timing grade.
Catalog record
| CREATOR | Duseok Kang, Yunseong Lee, Junghoon Kim |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-17764
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Puzzle : Scheduling Multiple Deep Learning Models on Mobile Device with Heterogeneous Processors Thanks: Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc. Duseok Kang, Yunseong Lee, Junghoon Kim Abstract . As deep learning models are increasingly deployed on mobile devices, modern mobile devices incorporate deep learning-specific accelerators to handle the growing computational demands, thus increasing their hardware heterogeneity. However, existing works on scheduling deep learning workloads across these processors have significant limitations: most studies focus on single-model scenarios rather than realistic multi-model scenarios, overlook performance variations from different hardware/software configurations, and struggle with accurate execution time estimation. To address these challenges, we propose a novel genetic algorithm-based methodology for scheduling multiple deep learning networks on heterogeneous processors by partitioning the networks into multiple subgraphs. Our approach incorporates three different types of chromosomes for partition/mapping/priority exploration, and leverages device-in-the-loop profiling and evaluation for accurate execution time estimation. Based on this methodology, our system, Puzzle , demonstrates superior performance in extensive evaluations with randomly generated scenarios involving nine state-of-the-art networks. The results demonstrate Puzzle can support 3.7 and 2.2 times higher request frequency on average compared to the two heuristic baselines, NPU Only and Best Mapping, respectively, while satisfying the equivalent level of real-time requirements. 1. Introduction Deep Learning (DL) has achieved remarkable performance across various fields, leading to the widespread adoption of numerous and diverse DL models. As these models are deployed in diverse applications, the computational and memory demands of DL workloads have surged. This has prompted the integration of specialized Neural Processing Units (NPUs) tailored for DL workloads, offering more efficient processing than traditional hardware accelerators like CPUs and GPUs. Consequently, systems equipped with multiple heterogeneous processors have become common. This has spurred extensive research into methodologies that leverage these heterogeneous processors to handle DL workload effectively ( Kumar et al. 2022 ; Choi et al. 2022 ; Silvano et al. 2024 ; Li et al. 2024 ) . Table 1. Comparison various works utilizing heterogeneous processors to accelerate DL networks Partition Unit Multiple Models Search Space Non-linearity of Exec. Time (§ 2.1.2 ) \mu layer ( Kim et al. 2019 ) Layer \times P + BR \times CoDL ( Jia et al. 2022 ) Layer \times P \times Kang et al. ( Kang et al. 2020 ) Graph \triangle M \times Band ( Jeong et al. 2022b ) Graph \bigcirc M \bigcirc Jeong et al. ( Jeong et al. 2022a ) Graph \times M \bigcirc Collage ( Jeon et al. 2022 ) Graph \times M \times BE \bigcirc OmniBoost ( Karatzas and Anagnostopoulos 2023 ) Graph \bigcirc M \times Puzzle (ours) Graph \bigcirc M \times T \times BE \bigcirc P : Partition Ratio / BR : Branch Dist. / M : Mapping / T : Data Type / BE : Backend Impl. When it comes to platforms with limited resources, such as mobile devices, there is also a wide variety of research that focuses on utilizing heterogeneous processors ( Kim et al. 2019 ; Jia et al. 2022 ; Jeon et al. 2022 ; Kang et al. 2020 ; Seo et al. 2021 ; Jeong et al. 2022a ; Jeong et al. 2022b ; Karatzas and Anagnostopoulos 2023 ) . Unfortunately, these works often overlook critical aspects of DL workloads on mobile devices. Table 1 compares these studies that aim to accelerate DL networks using heterogeneous processors. First, the second column shows the unit of workload distribution used in these studies — either layer or graph partition. Among them, layer-partition approaches, which partition layers and assign them to heterogeneous processors, incur significant synchronization overhead because they require synchronization at every layer ( Kim et al. 2019 ; Kang et al. 2020 ) . Second, as the third column in Table 1 shows, many studies focus on scheduling only a single DL model ( Kim et al. 2019 ; Jia et al. 2022 ; Jeong et al. 2022a ; Jeon et al. 2022 ) or do not perform end-to-end execution experiments for scenarios involving multiple DL models ( Kang et al. 2020 ) , even though various DL models are executed concurrently in practice. Third, a typical mobile platform offers various configurations for DNN execution; we can select not only the processor (CPU, GPU, or NPU), but also choose the backend implementation (e.g., ONNX Runtime ( Runtime 2025 ) incorporates its own CPU, XNNPACK ( Google 2025a ) , and NNAPI ( Google 2025c ) execution providers for the CPU). Additionally, the data types of kernels (32-bit full precision, 16-bit half precision, or 8-bit integer) provide further configuration options, with a trade-off between execution time and output accuracy. While other studies ( Kang et al. 2020 ; Jeong et al. 2022b ; Jeong et al. 2022a ; Karatzas and Anagnostopoulos 2023 ; Jia et al. 2022 ) often assume a single configuration and focus only on mapping decisions (as shown in the fourth column of Table 1 ), we found that no dominant configuration provides the shortest execution time for all types of DNNs; the optimal configuration varies depending on the DNN itself. Lastly, many studies ( Kim et al. 2019 ; Kang et al. 2020 ; Jia et al. 2022 ) estimate the total execution time of a model or subgraph (or set of layers) by summing individual layers’ execution times measured offline or predicted, ignoring inter-layer (or graph-level) optimizations. However, these optimizations can significantly affect DL inference performance. Moreover, modern accelerators, such as Qualcomm’s NPU, can execute multiple operations concurrently ( Qualcomm 2025 ) , making the previous estimation method prone to significant errors. We refer to this characteristic of DL inference as the non-linearity of execution time, which is further discussed in Section 2.1.2 . Figure 1. An example of multiple model groups We propose a methodology for scheduling and deploying multiple DL networks on mobile devices equipped with heterogeneous processors. Our approach emphasizes practical considerations and evaluations. First, observing that DL models on mobile devices often operate periodically based on sensors like cameras or audio sensors, we introduce a Model Group — a set of models that operate based on the same input source in a synchronized manner. In addition, an application may contain multiple model groups. Figure 1 shows an application example that contains multiple model groups: One group may process visual data received from the camera for selfie segmentation ( Google 2025b ) , while another group parses the audio input data to run a speech-to-text pipeline ( Radford et al. 2023 ) . We employ a Genetic Algorithm (GA) to explore the vast search space across multiple dimensions, including graph partitioning, processor mapping, data types, and backend implementations. To effectively explore this complex space, we design three types of chromosomes for graph partition, processor mapping, and priority. Additionally, we address the non-linear execution time by measuring objectives under a user-defined scenario on the target device. To avoid the search from becoming excessively slow, we employ a simple simulator that estimates the objectives with communication cost model and measured execution times of subgraphs. We also develop a runtime equipped with two optimizations, tensor pool and shared buffer, and evaluate GA solutions from various perspective. Based on this methodology, we develop an