PicoSpec: A Pipelined Collaborative Speculative Decoding Framework for Efficient Edge-Cloud LLM Inference

Yida Zhang, Zhiyong Gao, Shuaibing Yue, Jie Li, Rui Wang · 2026

Read PicoSpec: A Pipelined Collaborative Speculative Decoding Framework for Efficient Edge-Cloud LLM Inference by Yida Zhang, Zhiyong Gao, Shuaibing Yue, Jie Li, Rui Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-19133.txt.

Introduction

Zhang and co-authors present PicoSpec, a way to speed up large language models split between edge devices and the cloud. A small model on the device drafts text while the large cloud model checks it, and the two run at the same time. A sparse compression step cuts the cost of sending word scores back and forth. The method needs no new training. In tests, the team reports speed gains of up to 2.9 times over prior methods.

Catalog record

CREATORYida Zhang, Zhiyong Gao, Shuaibing Yue, Jie Li, Rui Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33576
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.19133

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PicoSpec: A Pipelined Collaborative Speculative Decoding Framework for Efficient Edge-Cloud LLM Inference Yida Zhang, Zhiyong Gao, Shuaibing Yue, Jie Li, Rui Wang Abstract Recent advancements and widespread adoption of Large Language Models (LLMs) in both industry and academia have catalyzed significant demand for LLM serving. However, traditional cloud services incur high costs, while on-device inference alone faces challenges due to limited resources. Edge-cloud collaboration emerges as a key research direction to combine the strengths of both paradigms, yet efficiently utilizing limited network bandwidth while fully leveraging and balancing the computational capabilities of edge devices and the cloud remains an open problem. To address these challenges, we propose Pi pelined Co llaborative Spec ulative Decoding Framework (PicoSpec), a novel, general-purpose, and training-free speculative decoding framework for LLM edge-cloud collaborative inference. We design an asynchronous pipeline that resolves the mutual waiting problem inherent in vanilla speculative decoding within edge collaboration scenarios, which concurrently executes a Small Language Model (SLM) on the edge device and a LLM in the cloud. Meanwhile, to mitigate the significant communication latency caused by transmitting vocabulary distributions, we introduce separate rejection sampling with sparse compression, which completes the rejection sampling with only a one-time cost of transmitting the compressed vocabulary. Experimental results demonstrate that our solution outperforms baseline and existing methods, achieving up to 2.9× speedup. 1 Introduction Large Language Models (LLMs) have shown incredible potential in tasks like natural language processing, coding, and reasoning. However, their massive memory and compute requirements make it difficult to deploy on resource-constrained edge devices. Edge-cloud collaborative inference offers a way out by offloading heavy work to the cloud while keeping easy task local 21 . Speculative decoding has proven to be an effective technique for improving inference speed 3 . Its core idea is to use a small ”Draft” model to quickly generate draft tokens, then verified by a larger ”Target” model (Figure 1 b). Originally, this approach was designed to speed up the generation process on high-performance cloud servers, where data transfer is almost instantaneous. Someone tried to apply it to edge-cloud collaborative setups (Figure 1 c). However, in practice, this transition is far from simple. When moved from a high-speed local environment to a distributed network, researchers have found that the original design faces massive performance drops and practical hurdles that make it difficult to use in the real world 14 . Figure 1: Comparison between (a) Cloud autoregressive decoding, (b) Cloud speculative decoding, (c) Vanilla collaborative speculative decoding, and (d) PicoSpec. Despite the promise of edge-cloud collaborative speculative decoding, existing solutions still face severe ”communication-computation mismatch” and flexibility issues in the real-world. Existing solutions use a serial ”stop-and-wait” mechanism where the edge must pause for cloud verification before generating the next batch 18 ; 11 . In a Wide Area Network (WAN), the Round-Trip Time (RTT) is often high enough to cancel out any speedups, leaving the pipeline full of ”bubbles”. Other approaches suggest using early-exit points in the cloud to reduce verification latency 13 . However, these invasive methods require model retraining or fine-tuning, which undermines generality and increases operational costs 14 . To solve these problems, we propose PicoSpec (as in Figure 1 d), a plug-and-play framework designed for efficient distributed inference. PicoSpec is a system-level solution that works with standard models without any retraining. Its core consists of an asynchronous pipeline that decouples the edge-side drafting and cloud-side verification steps, as well as a separate rejection sampling algorithm optimized for communication overhead. This allows the edge device to keep speculating future tokens without waiting for the cloud’s feedback on previous ones, effectively hiding the network latency. Our main contributions are as follows: • PicoSpec is the first research to propose a training-free asynchronous framework for distributed speculative inference. We have comprehensively designed a generative inference framework for collaborative speculative decoding of LLMs, enabling faster inference without any modification or retraining of existing models. • PicoSpec introduces an asynchronous distributed speculative decoding pipeline. We incorporate Parallel Drafting, Fast Verification, and Overlapped Communication into the pipeline, which eliminates data dependencies between the edge device and the cloud, enabling genuine parallel decoding between edge and cloud. • PicoSpec designs Separate Rejection Sampling algorithm to minimize data transmission overhead. Since transmitting the full vocabulary distribution typically incurs substantial communication costs, we distribute the rejection sampling process between the edge and the cloud and apply sparse compression on distribute data, significantly reducing communication latency without altering sampling accuracy. 2 Related Work 2.1 Speculative Decoding Speculative decoding accelerates LLM inference by substituting serial generation with a ”draft-then-verify” paradigm. Chen et al. 3 demonstrated that a modified rejection sampling mechanism can guarantee the output remains mathematically identical to the target model while providing significant wall-clock speedups. To enhance draft quality and throughput, Medusa 2 employs multiple parallel decoding heads fine-tuned on a frozen backbone, while EAGLE 8 uses feature-layer extrapolation to generate high-quality drafts with minimal overhead. For recent parallel speculative decoding studies, PEARL 9 tackles the mutual waiting problem between the draft model and target model by pre-verify and post-verify mechanisms, SwiftSpec 20 hides drafting costs via parallel tree generation, CoSine 5 scales this efficiency to multi-GPU environments by routing requests across specialized draft models. While these methods perform well in data centers with high-speed interconnects (like NVLink). they struggle in high-latency Edge-Cloud environment. The main issues are limited computing power and the mismatch between communication costs and latency masking. There is still a lack of system-level design specifically for high-latency, low-bandwidth, and state-separated environments. 2.2 Edge-Cloud Collaborative Inference Edge-Cloud collaborative inference via model spliting and task offloading to overcome the resource constraints of edge nodes. Frameworks like EdgeShard 19 and Jupiter 17 employ dynamic programming and intra-sequence pipeline parallelism to optimize model splitting across distributed nodes. However, these methods do not alter the auto-regressive nature of LLMs, leaving inference speeds heavily dependent on network RTT. Other research focuses on hybrid routing: MMSL 10 uses multi-stage scheduling, while Hybrid LLM 4 and Hybrid SLM and LLM 6 route requests or low-confidence tokens to the cloud based on task difficulty. Despite reducing local load, these schemes still suffer from stop-and-wait bottlenecks – the edge must idle during remote processing.This idling ties inference speed to network RTT, canceling out offloading gains in high-latency environment. Consequently, system throughput is often capped by network physical limits rather than the actual hardware capacity. 2.3 Edge-Cloud Collaborative Speculative Decoding To optimize edge performance, EdgeLLM 16 employs local parallel tree generation but imposes heavy memory and computing burdens on limited hardware. Recent collaborative efforts like Venkatesha et al. 13 and HAT 15 introduce early exits or hidden-state exchanges, yet these often require invasive model modifications or suffer from high transmission overhead. While DSSD 11 reduces uplink data through local re-sampling, it remains a serial ”stop-and-wait” architecture strictly bound by network RTT. A more recent framework, DSD 18 extends speculative decoding to distributed environments using an Adaptive Window Control (AWC) policy to tune speculation windows dynamically.Nevertheless, DSD primarily follows a serial stop-and-wait pattern Similarly, SLED 7 focuses on multi-client throughput but fails to provide deep latency masking for individual users. These limitations underscore the need for an asynchronous, general-purpose system that decouples generation from verification while maintaining model generality

Read the complete text