StarSD: One-for-Many Speculative Decoding
Junhao He, Feiran You, Hongyang Du · 2026
Read StarSD: One-for-Many Speculative Decoding by Junhao He, Feiran You, Hongyang Du (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21622.txt.
Introduction
Junhao He, Feiran You, and Hongyang Du scale speculative decoding across clusters of accelerators. The trick drafts tokens with a small fast model, then has the big model verify them. Most designs assume one machine, while modern serving spreads models over many nodes.
Their framework, StarSD, uses one draft model to serve many target models at once. The star layout links the single drafter to targets on distributed nodes. Sharing the drafter cuts wasted hardware across the cluster. Faster, cheaper generation matters as chat and coding tools serve millions of requests from shared data centers.
Catalog record
| CREATOR | Junhao He, Feiran You, Hongyang Du |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-21622
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
StarSD: One-for-Many Speculative Decoding Junhao He, Feiran You, Hongyang Du Abstract Speculative decoding accelerates autoregressive generation by separating token proposal from verification, but most existing approaches are designed for single-node execution and do not scale well to multi-accelerator clusters used for serving modern Large Language Models (LLMs). We present StarSD, a one-for-many speculative decoding framework that uses a single draft model to serve multiple target models across distributed nodes via a star topology. StarSD decouples drafting and verification, enabling effective sharing of draft computation, and preventing distributed accelerators from remaining idle under bursty workloads. We provide a system-level analysis that characterizes when and why a single draft model can remain fully utilized by multiple verifiers, yielding predictable latency and utilization gains. Extensive experiments in real-world distributed inference settings demonstrate that StarSD simplifies deployment and supports flexible resource allocation across heterogeneous accelerators, while maintaining output quality. These results indicate that StarSD is a practical and scalable framework for bringing speculative decoding to modern cloud and edge inference infrastructures. 1 Introduction Large Language Models (LLMs) have become a central interface for a wide range of applications, and their deployment increasingly involves serving large volumes of concurrent inference requests in shared accelerator clusters. Under such workloads, inference systems face sustained pressure on latency, throughput, and hardware efficiency, often operating near the limits of memory and compute. A key source of this pressure is autoregressive decoding, where each output token requires a serial forward pass, directly constraining end-to-end latency and limiting achievable throughput. This bottleneck has motivated many decoding acceleration methods that restructure generation to reduce the cost of sequential decoding. Among them, speculative decoding offers a lossless acceleration approach by introducing a lightweight draft model to propose multiple tokens that a large target model verifies in parallel, preserving the target model’s output distribution while reducing decoding time ( 8 ; 2 ) . Unlike batching or approximate decoding, this separation mechanism preserves exact generation semantics while exposing additional parallelism, which makes speculative decoding suitable for latency-sensitive serving. Building on this advantage, several methods are proposed to improve draft quality and reduce verification costs, thereby further increasing acceptance rates and achieving decoding speedups ( 13 ; 16 ; 12 ) . However, most speculative decoding designs implicitly adopt a centralized execution mode, in which the draft and target models are co-located on a single accelerator, e.g., a Graphics Processing Unit (GPU). This pattern is increasingly misaligned with production inference systems, as modern LLMs continue to grow in parameter scale and deployment workloads impose sustained memory pressure through large Key-Value (KV) caches, causing the combined footprint to approach or even exceed the capacity of a single accelerator. For instance, co-locating a Vicuna-7B ( 24 ) target model with the lightweight EAGLE draft model can leave little memory headroom ( 13 ) for an RTX 4090 (24GB), restricting support for long-context serving. Moreover, as LLMs expand into vertical domains and are increasingly deployed on resource-constrained edge devices, e.g., PCs, mobile phones, or NVIDIA Jetson, co-location becomes even less practical due to tighter memory and compute budgets. Although substantial work has explored memory-efficient inference techniques ( 7 ; 19 ) , these methods do not remove the need to host the target-draft model pair on every accelerator, which constrains batching efficiency and scheduling flexibility. Consequently, LLM inference systems require a spatial degree of freedom, enabling distributed placement that allows drafting and verification to be placed independently while cooperating tightly to share accelerator resources. In principle, this flexibility supports higher utilization by decoupling compute-intensive drafting from request-driven verification, and applies both within a single server across multiple accelerators and distributed clusters across machines. However, simply separating the draft and target models reveals an accelerator-level inefficiency due to a mismatch between the model inference pattern and the operating characteristics of modern accelerators. Specifically, the draft-then-verify workflow alternates between drafting, verification, and communication, while the target model verifies proposed tokens, the draft model remains idle. This alternating execution introduces idle gaps that limit end-to-end throughput. In addition, disaggregation makes the draft execution bursty , with short compute phases separated by millisecond-scale gaps. Such bursty execution can prevent the draft model from staying in a warm compute state, increasing per-token latency and its variance compared to continuous execution. Addressing these challenges requires a novel speculative decoding framework that can coordinate drafting and verification across machines while sustaining steady execution on modern accelerators. We propose StarSD, a one-for-many distributed speculative decoding framework where a single draft model concurrently assists multiple target instances. StarSD multiplexes asynchronous returns through a global request buffer and executes requests in a work-conserving loop with per-instance state isolation, ensuring continuous utilization of the draft accelerator whenever work is available. To analyze performance under concurrency, we develop an analytical system model that decouples progress , measured as the expected number of accepted tokens per round, from time , measured as the round duration. The model characterizes under-loaded and fully-loaded regimes and identifies the transition point at which idle gaps vanish. StarSD yields two complementary benefits under multi-instance workloads. By amortizing verification and communication return latency across multiple targets, it reduces idle gaps and increases aggregate throughput. Furthermore, a continuous request stream keeps the draft side closer to a steady-state execution regime, improving draft-side per-token compute efficiency and stability. These effects make speculative decoding more effective in disaggregated and memory-constrained deployments without changing model weights or the output distribution. Our contributions are summarized as • We propose StarSD in the one-for-many distributed speculative decoding setting, a framework that isolates per-target state and enables work-conserving draft execution in the presence of asynchronous verification returns from targets. • We develop an analytical system model that separates per-iteration progress from per-iteration time, characterizes load regimes, and explains scaling behavior, saturation effects, and the tradeoff between per-target throughput and system-wide throughput. • We evaluate StarSD across various deployments, including intra-machine multi-GPU, inter-machine multi-GPU, and heterogeneous platforms such as a Computer-Jetson setup. The results show that StarSD scales throughput with the number of target instances until saturation, with continuous drafting reducing per-token compute time and latency variance compared with bursty execution. 2 Preliminary Speculative decoding speeds up autoregressive inference by splitting decoding into two roles, i.e., a lightweight draft model proposes candidate continuations, and a large target model verifies them. Following the original speculative decoding formulation ( 8 ) , we denote the target and draft model types by \mathcal{M}_{p} and \mathcal{M}_{q} , and use M_{p} and M_{q} to refer to their deployed instances, respectively. When StarSD serves N target instances concurrently, we index them as \{M_{p}^{(i)}\}_{i=1}^{N} . In the single-instance case, we drop the index and write M_{p}\equiv M_{p}^{(1)} for notational simplicity. Per-request primitive. Given a verified prefix, one speculative decoding step contains of two stages: (i) proposal , where the draft instance M_{q} generates candidate continuations from the prefix, and (ii) verification , where the target instance M_{p} verifies the proposals and returns an accepted prefix defined as the longest prefix that passes verification. Both models then advance to the accepted prefix and continue decoding. Tree-based proposals. SpecInfer ( 16 ) popularizes tree-based speculative proposals to better exploit accelerator parallelism. Rather than proposing a single token chain, M_{q} expands a token tree of