Compiler-Assisted Speculative Sampling for Accelerated LLM Inference on Heterogeneous Edge Devices ∗ Work realized during internship at NXP Semiconductors. Alejandro Ruiz y Mesa ∗ 2 , Guilherme Korol 3 , Moritz Riesterer 3 , João Paulo C. de Lima 2 4 , Jeronimo Castrillon 2 4 Abstract LLM deployment on resource-constrained edge devices faces severe latency constraints, particularly in real-time applications where delayed responses can compromise safety or usability. Among many approaches to mitigate the inefficiencies of sequential token-by-token generation, Speculative Decoding (SD) has emerged as a promising technique. However, SD at the edge is hindered by two major challenges: (1) integrating SD into a compiler-based workflow without sacrificing performance or programmability, and (2) exploiting the heterogeneous compute resources of modern SoCs through carefully designed partitioning strategies. This work addresses these challenges by using an analytical cost model that explores heterogeneous hardware configurations and guides coarse-grained partitioning of LLM subgraphs, particularly with edge-typical short input sequence lengths. The cost model predicts when speculative sampling and heterogeneous execution are jointly beneficial and is validated on an edge device featuring a hexacore Cortex-A CPU and a Mali GPU, revealing up to 1.68 × \times speedup for translation tasks, closely matching analytic expectations. I Introduction Enabling LLM on edge devices is essential for applications requiring privacy, robustness, and low latency, such as offline health assistants or real-time translation [ 2 ] . However, deploying LLM on edge platforms faces severe challenges under the scarce compute, power, and memory available. Modern SoC ( SoC ) platforms integrate multiple heterogeneous PU , such as multi-core CPUs, mobile GPUs, and accelerators, creating opportunities for workload acceleration. Still, transformer-based LLM present a particularly challenging workload due to their autoregressive decoding, which generates one token at each forward pass and imposes hard sequential dependencies. Addressing these challenges, therefore, requires a combination of algorithmic-, system-, and hardware-level optimization techniques. SD ( SD ), in particular, has demonstrated remarkable improvements in the generative pipeline on high-end GPUs, achieving speedups up to 6.5 \times compared to traditional autoregressive decoding [ 3 , 4 ] . SD converts savings from traditional algorithmic optimizations (e.g., quantization, pruning, and distillation) into end-to-end latency reductions by relying on an inexpensive draft model to reduce the number of costly target-model decoding steps, with performance determined by the draft cost, verification cost, and acceptance rate. This approach naturally lends itself to migrating LLM inference from cloud to edge platforms. A key question is how to integrate SD into the software stack of an edge system in a way that preserves productivity for developers while enabling meaningful hardware-aware optimization. Without such careful integration, SD is not inherently beneficial: naive adoption or poor hardware mapping can negate its advantages and even increase end-to-end latency. In this work, we build upon IREE [ 6 ] , an open-source ML compiler to provide a unified representation of SD that supports explicit device affinities, coarse-grained heterogeneous mapping, and AOT ( AOT ) compilation across multiple backends. On top of that, we leverage the analytical cost model from [ 3 ] to guide LLM partitioning and mapping across heterogeneous PU . Concretely, our contributions are the following: • We apply an analytical cost model to (i) determine when to use speculative sampling and to (ii) map it onto heterogeneous PU for efficient LLM inference acceleration; • We present compiler-level abstractions of speculative sampling that enable device placement decisions for heterogeneous edge execution, comparing monolithic and modular compilation strategies; • We demonstrate that our optimized heterogeneous mappings can outperform homogeneous CPU execution by up to 1.68\times by reducing speculation overhead; • We validate the proposed cost model using a hexacore Cortex-A55 CPU and a Mali-G310 GPU integrated on an NXP i.MX95 SoC, for translation tasks, with a 4\% deviation from analytical expectations. II Background and Related Work This section builds on insights into computational bottlenecks inherent in LLM inference, SD as an algorithmic optimization technique, and the role of ML compilers in enabling efficient deployment across diverse hardware platforms. Toward the end, we review related work on accelerating neural network inference and highlight how our approach differs from existing techniques. II-A LLM Inference Bottlenecks Inference in transformer-based LLM proceeds in two phases, each presenting distinct computational challenges [ 7 ] . The prefill phase processes the entire input sequence in parallel, populating the KV ( KV ) cache across all transformer layers. In contrast, the decode phase generates tokens sequentially, with each token depending on the previous model state. As such, opportunities for parallelism are limited. Furthermore, the hidden dimension of the LLM , denoted as d , plays a pivotal role in shaping the computational characteristics of the model [ 1 ] . The sequence length ( S_{L} ) is often categorized relative to d : short sequences ( S_{L}\ll d ) are dominated by the linear layers, whereas the attention layers dominate in long sequences ( S_{L}\gg d ). This distinction is crucial for understanding performance bottlenecks. II-B Speculative Decoding SD accelerates autoregressive generation by reducing the number of full forward passes needed from the LLM (referred to as the target model). The process is conceptually divided into two phases. In the speculation phase , the system generates a sequence of candidate tokens using a computationally inexpensive mechanism. This may be a smaller transformer, a distilled or pruned version of the original network, or a specialized predictive module, and is designed to approximate the next-token distribution of the target model at a significantly lower computational cost. In the subsequent verification phase , the larger target model evaluates the drafted tokens in a single forward pass, in a parallel way similar to the prefill phase of the LLM inference. Speculated tokens are accepted or rejected according to a selection criterion. Fig. 1 illustrates standard incremental decoding (top) and two speculative strategies: sequence-based (middle) and tree-based (bottom). Orange segments indicate speculation by a lightweight model, while green segments denote verification by the target model. Fig. 1 : Three generative pipelines: standard sampling (top, in blue), and two variants of SD : sequential drafting (center) and tree-based drafting (bottom). Adapted from [ 15 ] . Speculative sampling [ 3 ] is a form of SD that avoids the training of a drafter mechanism, since it employs a smaller transformer model as the speculation strategy. The structural similarity between drafter and target yields sufficiently correlated logits to produce meaningful acceptance rates, enabling substantial speedups on server hardware. On edge devices, however, acceptance behavior is more complex. Quantization alters the relative probabilities of the vocabulary distribution, resulting in acceptance rates lower than those observed with full-precision models [ 8 ] . In speculative sampling, the achieved speedup depends not only on the alignment between the drafter and target model distributions, but also on the workload, the task, and the hardware-software environment in which the accelerated LLM is executed. A formalism modeling the hardware impact on the speculative sampling speedup is developed in [ 3 ] . It presents the speedup S as a function of the acceptance rate’s expected value ( \alpha , the mean proportion of accepted tokens), the speculated draft length ( \gamma , the sequence length of the speculated tokens), and the cost coefficient c=t_{\text{draft}}/t_{\text{target}} , a hardware- and software-dependent parameter representing the ratio between the latency of a single drafter forward pass ( t_{\text{draft}} ) and that of the target model ( t_{\text{target}} ). The speedup S is given by ( 1 ). S\left(\alpha,\gamma,c\right)=\frac{1-\alpha^{\gamma+1}}{(1-\alpha)(\gamma c+1)} (1) Moreover, the condition c<\alpha must hold to achieve any speedup at all. This guarantees that there is at least one value of \gamma that yields a speedup greater than 1. Furthermore, the optimal draft length \gamma^{*} that maximizes the speedup depends on both the hardware configuration and the quality of the drafting mechanism [ 3 ] . ((a)) Overview of the optimization flow ((b)) Example of design space Fig. 2 : Overview of heterogeneous mapping workflow for speculative sampling on edge devices. II-C Related Work Traditional algorithmic optimizations (e.g., quantization, pruning, and knowledge distillation) reduce compute and memory requirements, effectively producing a cheaper draft model and/or improving draft–target agreement. Recent systems (e.g., Sequoia [ 16 ] , DuoDecoding [ 17 ] , and Dovetail [ 18 ] ) further show that the optimal draft/verifier configuration depends on hardware characteristics and device heterogeneity, motivating joint algorithm–system co-design. However, these approaches primarily target high-end GPU platforms rather than edge-grade SoCs. Complementary work on mapping neural workloads onto heterogeneous compute resources, including polyhedral optimization and design-space exploration, appears in frameworks like Tiramisu [ 19 ] , ZigZag [ 20 ] , and FlexInfer [ 21 ] . Related efforts on the edge explore algorithm-hardware co-design on reconfigurable platforms, combining algorithmic optimizations, such as pruning and early-exit [ 10 , 11 ] . Neural network multi-tenancy represents another relevant research direction. Frameworks like MAGMA [ 22 ] and Adyna [ 23 ] address how concurrent models contend for limited compute and memory resources. Such systems reveal structural similarities to speculative sampling pipelines, where drafter and target models form a multi-tenant workload sharing limited edge resources. However, these works focus on the orchestration of independent models, rather than coupled models within a pipeline. Furthermore, they evaluate against accelerator models or simulators rather than on silicon, as done in this work. Our work differs since (1) we target an edge SoC rather than high-end GPU platforms, addressing the unique constraints of resource-limited devices; (2) we provide the first exploration of coarse-grained heterogeneous partitioning strategies for speculative sampling, leveraging high-level compiler frontend abstractions to express device placement decisions; (3) we apply an analytical cost model that jointly optimizes algorithmic acceleration (speculative sampling) and hardware mapping decisions for heterogeneous edge execution; and (4) we validate our approach on real silicon rather than relying on simulators, providing empirical evidence of the method accuracy. III Heterogeneous Mapping Framework for Speculative Sampling This section presents a systematic approach to evaluate mappings of LLM with speculative sampling on edge devices. It begins by providing an overview of the complete compilation and optimization workflow in Sec. III-A , followed by the formulation of the heterogeneous mapping problem as a DSE ( DSE ) in the Sec. III-B and III-C . Sec. III-D details how abstractions can be aligned with hardware-aware execution to preserve productivity while enabling good performance. III-A Overview of the Compilation and Optimization Workflow Our approach combines offline profiling, analytical modeling, and compiler-assisted code generation (Fig. 2(a) ). We evaluate speculative sampling [ 3 ] for its training-free nature, though the workflow is generalizable to other sequential SD techniques. Our workflow consists of the following steps: Offline quantization (Sec. III-C ) Model quantization is performed offline. In Fig. 2(a) , from ⓐ to ⓑ the target model and the drafter model are quantized with different schemes that match the available arithmetic on the target SoC . In ⓒ, the acceptance rate \alpha is measured for the different combinations of quantized target-drafter mechanisms. Profiling (Sec. III-B and III-C ) We compile forward passes of both models targeting all PU ①, profile them on hardware to measure t_{\text{draft}} and t_{\text{target}} ②, and calculate the cost coefficient c=t_{\text{draft}}/t_{\text{target}} for each design variant ③ (Fig. 2(b) ). Exploration (Sec. III-B ) With \alpha and c values in hand, the analytical cost model ④ is evaluated to determine the optimal draft length \gamma and device mapping for each hardware configuration (see ⑤). Compilation and Inference (Sec. III-D ) The complete SD pipeline is compiled and executed using IREE in ⑥. III-B Design Space Encoding, Exploration, and Evaluation We formulate the heterogeneous execution of an LLM with speculative sampling as a DSE problem over static spatial mappings of the computational graph: each subgraph is assigned ahead of time to a PU and no dynamic remapping is considered. Additionally, the search concentrates on coarse-grained partitions that separate the drafter (speculation mechanism model) from the target model ( LLM to be accelerated), a modeling choice motivated by the fact that the cost coefficient c (the cost of the speculation phase) must be lowered by, for example, decreasing the latency of the drafter model. We express the size of the design space as v\cdot N^{m} , where v is the number of design variants, N the number of distinct PU available for assignment, and m is the number of subgraph partitions. In this formulation, a design variant corresponds to a unique combination of cores, shaders, or PEs available across all PU , which we count by placing its unordered configurations in bijective correspondence with the Cartesian product v=\prod_{i}n_{i}, where n_{i} denotes the number of available cores, shaders, or PEs in \text{PU}_{i} . For example, Fig. 2(b) (top) depicts a CPU with six physical cores and a GPU with a single shader that yields v=6\times 1=6 distinct hardware configurations for mapping subgraphs. In practice, only a subset of these resources may be accessible at runtime, for example two active CPU cores as shown in gray. Moreover, for the configuration illustrated in Fig. 2(b) , assuming two PU available for mapping ( N=2 ) (top) and two graph partitions ( m=2 , one for the drafter and one for the target, bottom), the size of the design space becomes v=6\cdot 2^{2}=24 possible mappings. This space grows quickly if any of its factors increase: finer-grained partitioning (larger m ) increases the exponent, adding more types of PU (larger N ) raises the base, and incorporating more cores, shaders, and PEs (larger v ) multiplies the size of the space. In practice, therefore, exhaustive exploration becomes infeasible on richer platforms; for this reason, and to keep the study focused, we deliberately keep v , N , and m small and rely on the analytical cost model presented in ( 1 ) to guide the search. To evaluate each mapping, we find the \gamma value that yields the highest acceleration S , for a given cost coefficient c , that encodes the speculation computational cost of a given mapping, and a given acceptance rate \alpha , that encodes the drafter model speculation quality. Both \alpha and c are measured empirically as described next. III-C Measuring Acceptance Rates and Cost Coefficients We determine empirically the acceptance rate \alpha using a 16-core server-grade CPU. Although \alpha is a model-dependent value, reflecting how closely the drafter model approximates the larger target model, it remains hardware-independent 1 1 1 Slight deviations may occur when different devices handle rounding and precision differently, potentially leading to numerical discrepancies or precision divergence arising from quantization. . Concretely, to estimate \alpha , we employed the Spec-Bench, a benchmark designed to evaluate SD performance across multiple tasks [ 24 ] . The dataset comprises 480 samples distributed among thirteen tasks. In edge deployments, such task variability is lower because models are tailored to a single domain and typically benefit from knowledge distillation or task-specific fine-tuning. For this reason, we focus on the translation task. Additionally, due to the nature of translation, the length of the generated tokens tends to closely match the input sequence length, which is typically short (a few tens of tokens representing a sentence). As quantization is a key enabler of ML systems on edge devices, we evaluated the acceptance rate \alpha of multiple model configurations. These included: • fully quantized target-drafter pairs, • semi-quantized combinations, and • unquantized FP16 counterparts as reference models. All quantization schemes are static and implemented in w8a8 precision using the Intel Neural Compressor [ 25 ] . We compute c by compiling a single forward pass of both the drafter and target models using IREE 3.6.0, evaluating all combinations of the LLVM and SPIR-V backends independently, and benchmarking their respective runtimes. The ratio between these measured latencies directly yields c , thereby quantifying the relative computational cost of the speculation phase with respect to the target model for each homogeneous and heterogeneous configuration. Given the measured acceptance rate \alpha and c values, the cost model given by ( 1 ) is used to estimate the expected speedup and the optimal speculative length \gamma for each device pairing (homogeneous and heterogeneous). We then calculate the best-performing mapping and then validate the predicted speedup experimentally on hardware, comparing the resulting acceleration against a CPU-only, non-speculative baseline. III-D Matching Abstractions of Speculative Sampling and Heterogeneous Execution Selecting the abstraction level for spatial partitioning onto heterogeneous PU presents a tension: speculative sampling is most naturally expressed in high-level ML frameworks, whereas heterogeneous partitioning requires low-level control. We address this by rising hardware partitioning into the compiler frontend and integrating it directly within the speculative sampling pipeline. This unified, high-level formulation preserves the programmability of PyTorch-level code while enabling targeted acceleration (e.g., mapping the speculative phase onto an appropriate PU ). IREE, backed by MLIR ( MLIR ), enables mixing high-level dialects (e.g., Torch IR) with lower-level dialects (e.g., Flow), allowing spatial partitioning to be expressed at the PyTorch level using custom operators resolved to physical devices during lowering. This enables capturing the entire pipeline within a single optimized module (Fig. 3 ), reducing glue code while supporting end-to-end optimization. However, monolithic AOT compilation increases complexity, particularly for procedural logic with data-dependent control flow. Fig. 3 : Monolithic approach: single IREE module with target, drafter, and control flow subgraphs with heterogeneous device affinities. Fig. 4 : Modular approach: separate IREE modules for model arithmetic with control flow in the serving platform. Alternatively, Fig. 4 shows a modular design where target and drafter are compiled independently while procedural logic executes in the serving mechanism. This simplifies heterogeneous execuiton but introduces stricter boundaries between modules, reflected by additional API calls (thick black arrows), which add runtime overhead. We evaluate both alternatives in the following section. IV Experimental Setup and Evaluation Tab. I summarizes the experimental configuration adopted in this work. It outlines the workload characteristics, task definition, optimization objective, SD strategy, and the heterogeneous hardware resources considered during evaluation. In our setup, greedy sampling is used across all experiments and no KV cache is enabled. We employ the Llama 3.2 family, specifically Llama 3.2 1B as drafter and Llama 3.2 3B as target, following prior work demonstrating that alignment of the training data distributions between drafter and target is beneficial for draft acceptance [ 3 ] . TABLE I : Experimental settings Setting Description Workload Short sequence lengths ( S_{L}\ll d ) Task Translation (from Spec-Bench benchmark [ 24 ] ) Optimization objective Minimize latency SD technique Speculative sampling with target: Llama 3.2 3B and drafter: Llama 3.2 1B. Heterogeneous edge hardware Mali-G310 GPU and Hexacore Cortex-A55 CPU on NXP i.MX95 SoC IV-A Empirical Estimation of Acceptance Rate The acceptance rate \alpha serves as a required input parameter for the analytical cost model given by ( 1 ). While improving \alpha through better quantization schemes [ 8 ] or training quantized drafters directly [ 4 ] falls outside the scope of this work, we must characterize its behavior to validate our heterogeneous mapping methodology on real hardware. Evaluating these techniques on edge hardware represents a direction for future work; our contributions lies in how to exploit \alpha effectively. Fig. 5 shows the impact of quantization on \alpha for the translation task (Fig. 5(a) ) and the full Spec-Bench dataset (Fig. 5(b) ). The vertical axes represent the acceptance rate \alpha , while each box represents the distribution of \alpha for three pairs of models: from left to right: an unquantized FP16 drafter and target, a partially quantized configuration, where just the target model is quantized 2 2 2 Note that the fully unquantized configuration, as well as the partially quantized variant in which only the drafter is quantized, were excluded from the experiments due to memory limitations during initialization on our hardware-software setup. , and a fully quantized INT8 setup. As the level of quantization increases, the boxes shift downward, showing a consistent reduction in the median value for \alpha . In particular, the fully quantized pairing median collapses toward \alpha\approx 0 , indicating that almost no drafted tokens are accepted. In contrast, the unquantized configuration achieves a median value of \alpha=0.58 . ((a)) Translation acceptance rates comparison ((b)) Entire dataset acceptance rates comparison Fig. 5 : Acceptance rate \alpha distribution for different quantization schemes: FP (FP32), T (target), D (drafter). These results confirm prior work [ 8 ] : as quantization increases, it introduces a distributional mismatch between drafter and target models, and the median \alpha decreases substantially for speculative sampling on edge devices. This pattern is consistent across the entire Spec-Bench dataset, as shown in Fig. 5(b) , where the median falls along with increased quantization. The subsequent analysis concentrates on the translation task using the semiquantized configuration, for two reasons: (1) it provides a realistic edge deployment scenario balancing memory constraints with model accuracy, and (2) its broad distribution of acceptance rates, with \alpha values spanning from 0\% to 100\% , enables an exploration of the full range of speculative sampling behavior. IV-B Computation of the Cost Coefficient We individually measured the latency of one forward pass for both the target and drafter models as a function of the input sequence length and then calculated the cost coefficient c , summarized in Fig. 6 . The horizontal axes represent the input sequence length, and each curve depicts one design variant. Fig. 6(a) shows c values for homogeneous mappings on CPU, whereas Fig. 6(b) shows heterogeneous mappings where the drafter is executed on the GPU 3 3 3 We do not map the quantized target model onto the GPU because it does not support INT8 datatype; any INT8 tensor is promoted to FP32 , adding overhead and diminishing the benefits of quantization. . The regions with c>1 are shaded in red to indicate infeasible configurations in which one forward pass of the drafter is slower than the target model. These infeasible cases arise mainly when the system uses three to six CPU cores in the heterogeneous mapping (i.e., drafter model mapped on the GPU). ((a)) Measured cost coefficient (homogeneous execution) ((b)) Measured cost coefficient (heterogeneous execution) Fig. 6 : Cost coefficients c for homogeneous ( 6(a) ) and heterogeneous ( 6(b) ) mappings as a function of input sequence length. The black vertical line indicates S_{L}=63 , the average input sequence length for the translation task in the Spec-Bench C‑A55 nC: Cortex‑A55 CPU, n cores. A notable case appears in the design variant with only a single CPU core available (purple curves in both plots). When performing a heterogeneous mapping, the cost coefficient c decreases substantially, from approximately 0.80 (purple curve at the top for an input sequence length of 63 ) 4 4 4 This input sequence length corresponds to the average length of the translation task in the Spec-Bench dataset. to 0.41 at the same sequence length, shown at the bottom. This improvement arises because the GPU executes the drafter model roughly three times faster than a single CPU core, significantly reducing the computational cost of the speculative phase. As a result, this configuration emerges as the most favorable candidate for speculative sampling on the evaluated SoC using the Spec-Bench dataset. IV-C Estimation of the Speedup with the Cost Model The expected speedup is estimated using Eq. ( 1 ) for two acceptance rates: a high value of \alpha=0.90 corresponding to the 90th percentile and a low value of \alpha=0.17 corresponding to the median. The results are reported in Tab. II and Tab. III , respectively. For \alpha=0.90 (Tab. II ), speculative sampling combined with heterogeneous execution yields substantial improvements, particularly in design variant 1, which corresponds to a configuration with one CPU core available for mapping and the GPU (recall that a design variant represents a unique combination of cores, shaders, or PEs across all the PU ). This variant reaches a speedup of 1.68\times using \gamma=5 . The correct selection of the draft sequence length ( \gamma ) is crucial: each design variant achieves its best performance with a different \gamma value, ranging from 0 (no speculative sampling) to 5. TABLE II : Estimated speedup for \alpha=0.90 and S_{L}=63 Design Speculative Heterogeneous Speedup Variant Sampling Execution [ \times ] 1 Yes ( \gamma=5 ) Yes 1.68 2 Yes ( \gamma=2 ) Yes 1.10 3 No NA 1 4 No NA 1 5 Yes ( \gamma=1 ) No 1.02 6 No NA 1 TABLE III : Estimated speedup for \alpha=0.17 and S_{L}=63 Design Speculative Heterogeneous Speedup Variant Sampling Execution [ \times ] 1–6 No NA 1 In contrast, design variants with a higher number of available cores (i.e., 3, 4, and 6 cores upwards) are expected to have a performance decline when both speculative sampling and heterogeneous execution are applied. Therefore, these methods should be avoided in such configurations. For the design variant holding five CPU cores, there is a modest speedup, but heterogeneous mapping should not be applied. If the performance gain is too small, there is a risk that, in a real system, the improvement becomes negligible, especially when accounting for deployment overheads. As a result, we discourage the use of heterogeneous mapping in this scenario. However, for \alpha=0.17 (Tab. III ), even the shortest draft length \gamma=1 becomes detrimental and neither speculative sampling nor heterogeneous execution yields a speedup in any design variant. IV-D Validation and Discussion ((a)) Estimated acceleration ((b)) Measured acceleration Fig. 7 : Measured acceleration introduced by speculative sampling with different drafted token lengths \gamma executed heterogeneously: Target model mapped on one CPU core; drafter model, on GPU. The input sequence length is set to 63 and we report the average after ten runs. The underlying acceleration trend and its empirical validation are jointly illustrated in Fig. 7 , where the horizontal axes represent the acceptance rate \alpha and the vertical axes report the predicted or achieved acceleration S , in Figs. 7(a) and 7(b) , respectively. In both cases, we present the heterogeneous mapping in which the quantized target model is executed on a single CPU core, while the unquantized drafter model is assigned to the GPU. The results for an input sequence length of 63 are shown. The different curves correspond to different draft lengths \gamma , with longer drafts (orange and red curves) offering greater acceleration at sufficiently high \alpha , while becoming detrimental when \alpha falls. This explains why the semiquantized configuration with an acceptance rate of \alpha=0.17 in Tab. III yields no measurable speedup: the acceptance rate is simply too low for speculative sampling to be beneficial. Fig. 7(b) presents the measured acceleration, showing that the expected acceleration (Fig. 7(a) ) is achieved with an \alpha approximately 4% higher than estimated. Despite this shift, the empirical curve remains consistent with the analytic prediction. The evaluated heterogeneous configuration (drafter on GPU, target on single CPU core) arises from hardware-software constraints (e.g., IREE lacking INT8 support for Mali backends, quantization effects on \alpha leading to semiquantized setups). Alternative strategies (full-GPU execution, dynamic scheduling) either exceed the memory budget of the platform, damage drafter accuracy, or require unavailable runtime capabilities. The analytical cost model (Eq. 1 ) abstracts hardware characteristics via c to determine when heterogeneous execution is beneficial. The methodology generalizes to other platforms through profiling c and measuring task-specific \alpha . Runtime constraints in IREE 3.6.0 prevented deploying a monolithic graph with heterogeneous device affinities. Consequently, we compiled separate graphs (Fig. 4 ) orchestrated via IREE’s runtime API. This introduces interface overhead that may explain the measured 4\% deviation, but provided necessary flexibility for heterogeneous execution. V Conclusion and Outlook This work showed how compiler-assisted heterogeneous mapping can accelerate LLM inference with speculative sampling on resource-constrained edge devices. An analytical cost model provided a basis for deciding when speculative sampling and heterogeneous execution are worthwhile under fixed hardware budgets, and model validation showed a 4% deviation from predictions. Exposing low-level abstractions into the compilation flow facilitated combining speculative sampling with heterogeneous execution, improving productivity with minimal performance penalties. Under favorable conditions (with a predicted \alpha=0.90 and measured \alpha=0.94 ), Llama 3.2 3B achieved up to a 1.68 \times speedup. Future work should: (1) integrate improved quantization techniques to achieve higher \alpha values; (2) validate the cost model with additional edge SoCs and other SD techniques, as discussed in Sec. III-A ; (3) extend the model to incorporate finer-grained partitioning would allow mapping computation onto NPUs and other accelerators with restricted operation support; and (4) improve the runtime memory allocator for heterogeneous execution of monolithic graphs with mixed device affinities to allow shared GPU-CPU scheduling.