Quasar: Quantized Self-Speculative Acceleration for Rapid Inference via Memory-Efficient Verification

Guang Huang, Zeyi Wen · 2026

Read Quasar: Quantized Self-Speculative Acceleration for Rapid Inference via Memory-Efficient Verification by Guang Huang, Zeyi Wen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-01399.txt.

Introduction

Huang and Wen offer Quasar, a training-free plan using low-bit quantizing for the checking stage of speculative drafting, overcoming the memory wall limiting speedup. Speculative drafting splits token making into rapid drafting and side-by-side checking. Checking needs one complete run through the target model and stays memory-flow bound. Hard shape pruning hurts checking truth badly, while quantizing-based checking keeps the logit spread with high faith while cutting memory traffic in half.

Catalog record

CREATORGuang Huang, Zeyi Wen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37056
LICENSEhttp://creativecommons.org/publicdomain/zero/1.0/
RIGHTSCC0
SOURCEarXiv

Source: https://arxiv.org/abs/2603.01399

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Quasar: Quantized Self-Speculative Acceleration for Rapid Inference via Memory-Efficient Verification Guang Huang, Zeyi Wen Abstract Speculative Decoding (SD) has emerged as a premier technique for accelerating Large Language Model (LLM) inference by decoupling token generation into rapid drafting and parallel verification. While recent advancements in self-speculation and lookahead decoding have successfully minimized drafting overhead, they have shifted the primary performance bottleneck to the verification phase. Since verification requires a full forward pass of the target model, it remains strictly memory-bandwidth bound, fundamentally limiting the maximum achievable speedup.In this paper, we introduce Quasar ( Qua ntized S elf-speculative A cceleration for R apid Inference), a novel, training-free framework designed to overcome this ”memory wall” by employing low-bit quantization specifically for the verification stage. Our empirical analysis reveals that while aggressive structural pruning significantly degrades verification accuracy, quantization-based verification preserves the logit distribution with high fidelity while effectively halving memory traffic. Extensive experiments on state-of-the-art models (e.g., OpenPangu and Qwen3) demonstrate that Quasar maintains a speculative acceptance length comparable to full-precision methods while achieving a 1.28 × 1.28\times improvement in end-to-end throughput. Being orthogonal to existing drafting strategies, Quasar offers a generic and efficient pathway to accelerate the verification leg of speculative execution. Code is available at https://github.com/Tom-HG/Quasar . 1 Introduction Figure 1: Comparison of the verification bottleneck in Self-Speculative Decoding versus Quasar. The left panel illustrates standard full-precision verification, where the process is memory-bound and saturates the memory bandwidth bus, leading to significant latency. The right panel depicts Quasar’s approach using low-bit quantized verification, which effectively reduces bandwidth pressure and enables high-throughput validation. The rapid proliferation of Large Language Models (LLMs) has revolutionized natural language processing, enabling unprecedented capabilities in reasoning and generation ( 1 ; 31 ) . However, deploying these models in real-world applications presents a significant challenge: the prohibitive cost of inference. The standard decoding process for Transformer-based models is auto-regressive, generating one token at a time. This sequential nature renders the process memory-bandwidth bound rather than compute-bound, leading to high latency and suboptimal utilization of modern hardware accelerators ( 25 ) . Consequently, accelerating LLM inference without compromising generation quality has become a necessity for the scalability of AI services ( 12 ) . Speculative Decoding ( 16 ; 3 ) has emerged as a promising solution to this bottleneck. This paradigm decouples token generation from verification by utilizing a lightweight “draft model” to predict a short sequence of future tokens. These tokens are then verified in parallel by the target LLM. Recently, Self-Speculative Decoding ( 9 ; 7 ; 21 ) has gained traction, where the draft is generated by the target model itself (e.g., by skipping layers). This approach elegantly solves the problem of finding a compatible draft model. While self-speculative decoding effectively accelerates the drafting phase, it exposes a new bottleneck: Verification Latency (as shown in Figure 1 ). Although verification is performed in parallel, it necessitates a full forward pass of the target model for the candidate sequence. Since LLM inference is predominantly memory-bound, loading the full-precision weights for verification consumes substantial memory bandwidth. As the draft length increases to maximize potential speedup, the cost of this verification step becomes non-negligible, often diminishing the overall latency gains. Specifically, in bandwidth-constrained environments, the time taken to verify tokens can rival the time saved by speculative drafting. Therefore, a critical research question arises: How can we accelerate the parallel verification phase in self-speculative decoding without significantly degrading the generation quality? Figure 2: End-to-end speedup comparison across various benchmarks. We compare the acceleration achieved by Quasar against the Ngram baseline under two sampling settings: greedy decoding ( T=0 ) and stochastic sampling ( T=1 ). Quasar consistently outperforms the baseline across MT-bench, HumanEval, GSM8k, Alpaca, and CNN/DM, reaching up to a 1.6\times speedup on reasoning-heavy tasks like GSM8k. To address this bottleneck, we propose a novel Quantized Verification framework - Quasar ( Qua ntized S elf-speculative A cceleration for R apid Inference). Our key insight is that memory bandwidth pressure during verification can be drastically reduced by lowering the model’s weight precision. Instead of performing verification using the full-precision model, we employ a quantized version of the target model as the verifier. By compressing the weights, we significantly reduce the data movement requirements during the parallel forward pass, thereby accelerating the verification step. We posit that modern post-training quantization techniques ( 8 ; 20 ) have matured sufficiently that the divergence between the quantized and full-precision distributions is minimal, making quantized verification a robust and high-throughput alternative. Experiments demonstrate that our method achieves superior end-to-end throughput compared to standard verification baselines while maintaining generation quality. As shown in Figure 2 , Quasar delivers consistent speedup across various benchmarks. Contributions. In this paper, we present a framework to break the verification wall in speculative decoding. Our key contributions are summarized as follows: • We identify that the verification step is the primary bottleneck in existing self-speculative decoding systems due to memory bandwidth constraints. • We propose Quasar which uses low-bit-weight representations to accelerate parallel draft token validation. • Experiments demonstrate that Quasar achieves superior end-to-end throughput compared to standard verification baselines while maintaining generation quality. 2 Related Work 2.1 Speculative Decoding Paradigm Speculative Decoding (SD) fundamentally transforms the auto-regressive generation process by breaking the strict sequential dependency of token production. First introduced by ( 16 ) and ( 3 ) , SD employs a predictor-verifier architecture: a computationally efficient draft model proposes a sequence of candidate tokens (the draft), which are subsequently verified in parallel by the target model. Theoretical frameworks have since been established to ensure that the speculative distribution strictly aligns with the target distribution via rejection sampling or optimal transport schemes ( 28 ; 22 ; 13 ) . While early iterations relied on separate, smaller draft models (e.g., a 7B drafter for a 70B target), this introduces significant operational complexity, including the need to manage dual KV caches and coordinate distinct model weights in distributed environments. 2.2 Advancements in Efficient Drafting To mitigate the overhead of maintaining independent draft models, recent research has pivoted towards Self-Speculation, where the drafting mechanism is integrated into the target model ecosystem. These approaches broadly fall into three categories: Retrieval and Pattern Matching. Motivated by the inherent locality and repetition in natural language, methods like Prompt Lookup Decoding (PLD) ( 26 ) and REST ( 11 ) utilize n -gram matching against the pre-generated context or a datastore to construct drafts. These training-free approaches are highly effective for tasks with high repetition (e.g., code editing) but struggle with open-ended generation where context reuse is minimal. Parallel Fixed-Point Iteration. A separate stream of work reformulates decoding as a system of non-linear equations. Lookahead Decoding ( 9 ) and Jacobi-based methods ( 24 ) apply parallel fixed-point iteration (Jacobi iteration) to the target model’s trajectory. These methods refine multiple tokens simultaneously using the model’s internal attention mechanism, eliminating the need for explicit draft weights, though often yielding shorter acceptance lengths compared to model-based drafters. Auxiliary Verification Structures. Inspired by blockwise parallel decoding ( 27 ) , Medusa ( 2 ) augments the frozen target model with multiple lightweight decoding heads (MLPs) to predict future tokens in a single pass. State-of-the-art

Read the complete text