EasySpec: Layer-Parallel Speculative Decoding for Efficient Multi-GPU Utilization Yize Wu, Ke Gao, Ling Li, Yanjun Wu Abstract Speculative decoding is an effective and lossless method for Large Language Model (LLM) inference acceleration. It employs a smaller model to generate a draft token sequence, which is then verified by the original base model. In multi-GPU systems, inference latency can be further reduced through tensor parallelism (TP), while the optimal TP size of the draft model is typically smaller than that of the base model, leading to GPU idling during the drafting stage. We observe that such inefficiency stems from the sequential execution of layers, which is seemingly natural but actually unnecessary. Therefore, we propose EasySpec, a layer-parallel speculation strategy that optimizes the efficiency of multi-GPU utilization. EasySpec breaks the inter-layer data dependencies in the draft model, enabling multiple layers to run simultaneously across multiple devices as “fuzzy” speculation. After each drafting-and-verification iteration, the draft model’s key-value cache is calibrated in a single forward pass, preventing long-term fuzzy-error accumulation at minimal additional latency. EasySpec is a training-free and plug-in method. We evaluated EasySpec on several mainstream open-source LLMs, using smaller versions of models from the same series as drafters. The results demonstrate that EasySpec can achieve a peak speedup of 4.17x compared to vanilla decoding, while preserving the original distributions of the base LLMs. Specifically, the drafting stage can be accelerated by up to 1.62x with a maximum speculation accuracy drop of only 7%. The code is available at https://github.com/Yize-Wu/EasySpec. Figure 1: Overview of EasySpec: comparison between standard Speculative Decoding and EasySpec. The differences are mainly about: (1) Fuzzy Speculation uses layer parallelization for drafting acceleration. (2) The fuzzy KV items are discarded regardless of acceptance length, while Speculative Decoding preserves accepted items. (3) Speculative Decoding is 2-staged and EasySpec is 3-staged, with an additional stage for bonus calibration. Figure 2: GPU runtime of different decoding algorithms in the distributed system. Speculative Decoding (SD) enables token-level parallelism (fewer base-model blocks), while causing multi-GPU under-utilization during the drafting stage. EasySpec solves the problem by layer-parallel speculation. 1 Introduction Transformer-based Large Language Models (LLMs) have demonstrated remarkable problem-solving abilities across various domains [ 28 , 5 , 31 , 32 , 1 ] . However, as the parameter size continues to grow, the time-consuming process of auto-regressive decoding poses a significant barrier to deploying large models in latency-sensitive applications [ 16 , 2 ] . Various effective approaches [ 12 , 14 , 8 ] have been proposed to reduce inference latency, including speculative decoding [ 19 , 3 ] and tensor-parallel (TP) distribution [ 25 ] . Speculative decoding employs a smaller model to generate a draft token sequence, and uses token-level parallelism to conduct non-autoregressive verification, ensuring no shifting of the original model’s output distribution. In contrast, TP distribution leverages cross-device parallelism by partitioning computational workloads across multiple devices (usually GPUs) and synchronizing the results subsequently, which is also lossless. Combining speculative decoding with TP distribution achieves even greater acceleration ratio. However, integrating a draft model into a distributed system is not trivial [ 2 ] . Since the parameter size of the draft model is typically smaller than that of the base model, the optimal TP size (the number of segments of workload distribution) is correspondingly smaller [ 3 ] , meaning that the draft model would run fastest when dispatched on one or a subset of GPUs, leaving other GPUs idle (see Figure 2 and Table 6 ). Consequently, multi-GPU computational resources are under-utilized during the drafting stage. We identify the primary cause of such inefficiency as the lack of parallelism between the draft model’s layers: while tensor operations within one layer can be parallelized by TP, the layers themselves are restricted to be executed sequentially, one after another and from bottom to top, for generating a “precise” result of inference. However, the drafting result is never required to be precise, as it is only used for token parallelism and does not directly impact the final output—the verification result does (see Section 2.1 ). Therefore, strictly following the execution order is unnecessary, while a “fuzzy” but faster layer execution strategy could be preferable than the precise one, as long as it can sufficiently approximate the drafting result. Based on this insight, we propose EasySpec, a layer-parallel speculation strategy for optimizing the efficiency of drafting-stage multi-GPU utilization. EasySpec introduces fuzzy speculation , which breaks the data dependencies between some consecutive attention layers by running them with the most recent hidden state as input to all of them ( Figure 1 and Algorithm 2 ). As the data dependencies are eliminated, multiple layers of the draft model can execute simultaneously on separate devices, achieving layer-level parallelism, more efficient multi-GPU utilization and hence faster speculation inference. Meanwhile, the speculation outputs can be well-approximated, retaining a high speculation accuracy. The approximation errors in the key-value (KV) cache of the draft model may accumulate during the inference procedure. To prevent this, we perform bonus calibration after each iteration of drafting-and-verification. Firstly, the accepted tokens are concatenated to the bonus token to form a token sequence. This sequence is then re-input to the draft model, where a conventional layer-sequential forward pass is executed, and the KV cache is updated with the precise values. This calibration step applies token parallelism, which is typically used only in the verification stage, to the drafting stage as well. Therefore, it incurs minimal additional latency to the speculation procedure, while significantly enhancing accuracy—ultimately contributing to improved overall inference speed. We evaluate our method on several widely-used open-source LLMs and task-specific models. The draft models are selected from the same series as their corresponding large models. The evaluation results show that EasySpec can achieve a peak speedup of 4.17x over vanilla decoding. Specifically, the drafting stage can be accelerated by up to 1.62x with no greater than 7% drop of speculation accuracy, requiring no additional training or fine-tuning on the existing draft models. 2 Preliminary 2.1 Speculative decoding Speculative decoding is a two-stage non-autoregressive decoding method for inference acceleration. The two stages are namely drafting and verification, which are iteratively executed as the inference proceeds. At time step t , the input token sequence is X . In the drafting stage, a smaller and faster model M^{\prime} is employed as the draft model. M^{\prime} auto-regressively runs n times, generating a speculation token sequence X,x^{\prime}_{t+1},...,x^{\prime}_{t+n} and probability distributions p^{\prime}_{t+1},...,p^{\prime}_{t+n} . The original large model M then takes the whole sequence as input and conducts a single-forward verification, the outputs of which are p_{t+1},...,p_{t+n+1} . The acceptance probability of token x^{\prime}_{i} is \min(1,\frac{p_{i}(x^{\prime}_{i})}{p^{\prime}_{i}(x^{\prime}_{i})}) . After the verification sampling, the sequence of m accepted tokens x_{t+1},...,x_{t+m} (the same as x^{\prime}_{t+1},...,x^{\prime}_{t+m} ) will be appended to the final output, along with a “bonus” token x_{t+m+1} (called bonus because it is a by-product of the verification process) which can be sampled from the distribution p : p=\left\{\begin{aligned} &norm(\max(0,p_{t+m+1}-p^{\prime}_{t+m+1}))&(m