Taming Asynchronous CPU-GPU Coupling for Frequency-aware Latency Estimation on Mobile Edge

Jiesong Chen, Jun You, Zhidan Liu, Zhenjiang Li · 2026

Read Taming Asynchronous CPU-GPU Coupling for Frequency-aware Latency Estimation on Mobile Edge by Jiesong Chen, Jun You, Zhidan Liu, Zhenjiang Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-15357.txt.

Introduction

Chen and three colleagues introduce FLAME, a method for estimating model inference latency on mobile edge devices under changing processor frequencies. Simple scaling fails because CPU launch work and GPU execution overlap asynchronously, so FLAME models each layer's overlap and pipeline bubbles, then aggregates them. Sparse profiling cuts DNN profiling from hours to minutes and small-language-model profiling from days to minutes, with small errors.

Catalog record

CREATORJiesong Chen, Jun You, Zhidan Liu, Zhenjiang Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60886
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.15357

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Taming Asynchronous CPU-GPU Coupling for Frequency-aware Latency Estimation on Mobile Edge Jiesong Chen, Jun You, Zhidan Liu, Zhenjiang Li Abstract Precise estimation of model inference latency is crucial for time-critical mobile edge applications, enabling devices to calculate latency margins against deadlines and trade them for enhanced model performance or resource savings. However, the ubiquity of Dynamic Voltage and Frequency Scaling (DVFS) renders traditional static profiling invalid in real-world deployments, as inference latency fluctuates with varying processor (CPU and GPU) frequencies. While extensive profiling across frequency combinations is theoretically possible, it is prohibitively expensive, particularly for emerging Small Language Models (SLMs), where variable context lengths explode the profiling up to days. We observe that simple analytic scaling fails to predict these fluctuations due to the complex asynchronous coupling between CPU (kernel launching) and GPU (execution). In this paper, we introduce FLAME to accurately estimate inference latency across frequency combinations. It features a novel layer-wise modeling that quantifies the overlapping parallelism and then aggregates dynamic pipeline bubbles caused by asynchronous processor interactions when extending to the full model. This bottom-up approach ensures generalizability across diverse models from DNNs to SLMs, and its precise modeling allows for profiling a sparse subset of samples, cutting DNN profiling from hours to minutes and SLM profiling from days to mere minutes, while maintaining small estimation errors across frequencies. We further showcase FLAME’s utility in a deadline-aware DVFS, outperforming the state-of-the-art approach in both power efficiency and latency guarantees. I Introduction Deploying AI models on mobile edge devices, such as the NVIDIA Jetson series and other System-on-Chips (SoCs), requires a delicate balance between performance and resource efficiency [ 4 ] . These devices are increasingly crucial to enable innovative, next-generation technologies such as autonomous vehicles [ 21 ] , embodied robotics [ 11 ] , satellite computing [ 52 ] , unmanned aerial vehicles (UAVs) [ 22 ] , etc. For time-critical mobile edge applications within these domains, latency awareness is paramount. Specifically, accurately estimating AI model’s inference latency before execution is a fundamental capability [ 56 ] . It enables the system to calculate a residual latency margin against a reference deadline, serving as the currency to trade excess speed for improved inference quality (via model variant adaptation [ 29 ] ), reduced power consumption (via frequency scaling [ 24 ] ), or higher task priority (via preemptive scheduling [ 16 ] ), thereby optimizing the overall system or application performance. The main approach for achieving latency awareness is profiling-based estimation [ 29 , 16 ] , where a model is executed offline on the target device to measure its execution latency or other factors (see § VII ) as a prior needed for runtime estimation. While straightforward, existing profiling methods assume that runtime frequencies of CPU and GPU processors match their frequencies used during profiling, typically locking them at maximum levels for consistence [ 56 ] . However, real-world edge deployments rely on Dynamic Voltage and Frequency Scaling (DVFS) to adjust frequencies dynamically [ 9 , 15 , 28 ] , optimizing for thermal and power constraints without sacrificing essential device performance [ 24 ] . Thus, static profiling results become invalid once the device adjusts processors’ operating frequencies. A natural question arises: Can we simply extend profiling to cover all CPU-GPU frequency combinations? If feasible, it could seamlessly adapt to any DVFS strategy and provide the largest solution space for downstream latency-based optimizations in other resources. While theoretically possible, this brute-force strategy is prohibitively expensive. For classic Deep Neural Networks (DNNs) like ResNet, profiling takes tens of minutes to over an hour because low-frequency states greatly extend execution time, and multiple iterations are required to average out jitters [ 16 ] . The challenge intensifies drastically for emerging Small Language Models (SLMs) [ 34 ] . Unlike DNNs, SLMs introduce a third dimension to profiling: context length [ 48 ] , since workload increases with changes in context length, e.g. , profiling a Qwen2-7B only up to a 1k context length on a Jetson AGX Orin takes 10+ days (§ VI ). Such overhead renders exhaustive profiling extremely expensive for diverse AI model architectures in practice. To overcome this issue, an effective solution is to model the latency-frequency relationship analytically. Once this interplay is captured precisely, we can use only a sparse subset of profiling samples to fit latency-frequency modeling parameters and extrapolate to unseen configurations ( e.g. , frequency combinations, context lengths, etc.), thereby reducing profiling overhead dramatically. While latency scales inversely with frequency for isolated processors [ 19 ] , this simple linear relationship fails for model inference on mobile edge, because inference on such devices involves complex coupling of the CPU (for preprocessing and kernel launching) and the GPU (for massive parallel computation) [ 25 ] , which introduces asynchronous timing dynamics . The CPU must prepare and feed workloads to the GPU. As the frequencies of the two processors scale, the timing relationship between CPU submission and GPU execution shifts. A frequency configuration that allows perfect pipelining at high frequencies may introduce significant pipeline bubbles (idle waiting times) at lower frequencies. Prior work on single-processor behaviors [ 17 , 37 ] or coarse modeling/learning [ 35 ] fails to effectively capture this shifting overlap, leading to large errors when frequencies vary. In this paper, we introduce FLAME ( F requency-aware L atency A nalysis for M obile E dge) to capture the intricate coupling effects between heterogeneous mobile edge processors. It enables precise latency estimation under arbitrary frequency combinations with very small profiling overhead. To achieve this, we tackle two main challenges: Asynchronous timing dynamics. The interplay between CPU kernel launching and GPU execution introduces a dynamic “timing factor” that varies non-linearly with frequency. This factor represents either the overlapping parallelism or the blocking wait time between the two processors. We propose a dedicated layer-wise formulation that decouples these components, allowing us to build a frequency-dependent modeling to effectively quantify the CPU-GPU overlap under various frequency combinations. Asynchronous pipeline aggregation. Extending layer-wise estimation to the full AI model is not straightforward, as operating system (OS)-scheduled asynchrony allows the CPU to prepare future layers while the GPU executes current ones [ 1 ] , creating a complex dependency chain . Naive summation ignores these pipeline overlaps, leading to cumulative errors. We further introduce a dependency-aware aggregation method that simulates this asynchronous pipeline behavior, ensuring accurate full-model estimation. By building precise latency-frequency modeling from the bottom up (starting with individual layers and aggregating to the full model), in addition to accurate latency estimation , FLAME delivers other two advantages further: Low profiling overhead. FLAME only needs to profile unique layers in the model, avoiding redundant measurements of the full model with repeated layers across frequencies. Moreover, the precise latency modeling allows for the use of only sparse subsets of profiling samples, e.g. , frequency combinations, context lengths (for SLMs), etc. These innovations together dramatically reduce profiling overhead, e.g. , shrinking the full profiling time of SLM from days to mere minutes, while maintaining high accuracy. Broad generalizability. Because FLAME is grounded in layer-wise composition, it is compatible to diverse AI model architectures, ranging from DNNs to SLMs. In addition, by explicitly modeling

Read the complete text