M A R S : Unleashing the Power of S peculative Decoding via M a r gin-Aware Verification
Jingwei Song, Xinyu Wang, Hanbin Wang, Xiaoxuan Lei, Bill Shi, Shixin Han, Eric Yang, Xiao-Wen Chang, Lynn Ai, [1mm] The University of Hong Kong, McGill University, Peking University, [2mm] Equal contribution Corresponding authors [email protected] , [email protected] , tianyu@grad · 2026
Read M A R S : Unleashing the Power of S peculative Decoding via M a r gin-Aware Verification by Jingwei Song, Xinyu Wang, Hanbin Wang, Xiaoxuan Lei, Bill Shi, Shixin Han, Eric Yang, Xiao-Wen Chang, Lynn Ai, [1mm] The University of Hong Kong, McGill University, Peking University, [2mm] Equal contribution Corresponding authors [email protected] , [email protected] , tianyu@grad (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-15498.txt.
Introduction
Jingwei Song and eight co-authors work on speeding up large language models in a 2026 arXiv paper. Their method is called MARS, and it targets speculative decoding. That technique speeds up text generation by splitting the work. A draft model proposes tokens, and the target model checks them. Recent work has tightened the link between drafter and target. Verification itself, though, has changed little.
Most systems still check drafts by strict token-level rejection sampling. MARS replaces that step with margin-aware verification. The name suggests checks that weigh how close a draft token is, not just right or wrong. Looser but sound checks could accept more draft text per pass. That would cut the time and cost of running large models.
Catalog record
| CREATOR | Jingwei Song, Xinyu Wang, Hanbin Wang, Xiaoxuan Lei, Bill Shi, Shixin Han, Eric Yang, Xiao-Wen Chang, Lynn Ai, [1mm] The University of Hong Kong, McGill University, Peking University, [2mm] Equal contribution Corresponding authors [email protected] , [email protected] , tianyu@grad |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-15498
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
M A R S : Unleashing the Power of S peculative Decoding via M a r gin-Aware Verification Jingwei Song, Xinyu Wang, Hanbin Wang, Xiaoxuan Lei, Bill Shi, Shixin Han, Eric Yang, Xiao-Wen Chang, Lynn Ai, [1mm] The University of Hong Kong, McGill University, Peking University, [2mm] Equal contribution Corresponding authors [email protected] , [email protected] , [email protected] , [email protected] Abstract Speculative Decoding (SD) accelerates autoregressive large language model (LLM) inference by decoupling generation and verification. While recent methods improve draft quality by tightly coupling the drafter with the target model, the verification mechanism itself remains largely unchanged, relying on strict token-level rejection sampling. In practice, modern LLMs frequently operate in low-margin regimes where the target model exhibits weak preference among top candidates. In such cases, rejecting plausible runner-up tokens yields negligible information gain while incurring substantial rollback cost, leading to a fundamental inefficiency in verification. We propose Margin-Aware Speculative Verification , a training-free and domain-agnostic verification strategy that adapts to the target model’s local decisiveness. Our method conditions verification on decision stability measured directly from the target logits and relaxes rejection only when strict verification provides minimal benefit. Importantly, the approach modifies only the verification rule and is fully compatible with existing target-coupled speculative decoding frameworks. Extensive experiments across model scales ranging from 8B to 235B demonstrate that our method delivers consistent and significant inference speedups over state-of-the-art baselines while preserving generation quality across diverse benchmarks. 1 Introduction Autoregressive Large Language Models (LLMs) suffer from high inference latency due to memory-bandwidth constraints ( 24 ) . Speculative Decoding (SD) addresses this by decoupling generation and verification: a lightweight draft model proposes a sequence of tokens, which are then verified in parallel by the target model ( 16 ) . State-of-the-art methods like Medusa ( 4 ) and EAGLE 2 ( 18 ) have significantly advanced this paradigm by integrating draft heads directly with target features, enabling high-fidelity drafting with minimal overhead. Figure 1: Comparison of Logit Ratio vs. Probability Ratio for Adaptive Verification (Qwen3-8B). Red points denote tokens accepted by our relaxation strategy (Logit Ratio >0.9 ). (a) Top-2 Logits: Relaxed tokens cluster along the diagonal ( z_{2}\approx z_{1} ), indicating our method captures candidates with similar raw scores regardless of scale. (b) Top-2 Probabilities: Unlike probability-based metrics, logit-based relaxation is not confined to high-entropy regions ( p_{1}\approx p_{2} ); it also captures candidates with relatively lower probability ratios. (c) Metric Decoupling: High logit ratios (red) do not imply high probability ratios. The red points span the full range of probability ratios (y-axis), demonstrating that our method recovers valid candidates that are otherwise suppressed by the exponential sensitivity of the softmax function. (d) Relaxation Zone Effectiveness: A zoom-in on the low-margin regime (Logit Ratio >0.9 ), where red points denote rejected tokens and purple points denote accepted ones. Despite these drafting improvements, the verification mechanism remains rigidly tied to exact-match rejection sampling. This standard approach implicitly assumes the target model holds a decisive preference at every step. In practice, however, LLMs frequently operate in low-margin regimes , where the likelihood difference between top candidates is statistically negligible. Strictly rejecting a plausible runner-up token in these cases yields negligible information gain while incurring substantial computational costs. While some recent works explore relaxed verification via learned semantic judges 3 ; 17 , they often introduce complexity through supervised training and auxiliary models, limiting their plug-and-play capability. In this work, we focus on the essence of the verification bottleneck: the mismatch between verification strictness and the target model’s intrinsic uncertainty. We propose a training-free paradigm that analyzes the target model’s logit margin —a direct proxy for decision stability. We observe that wasteful rejections cluster precisely in low-margin regions. By conditioning verification on the stability of the target logits, we adaptively relax requirements when the model itself is indifferent. We introduce Mar gin-Aware S peculative Verification (MARS), a simple strategy that performs stability-aware tie-breaking. Our method modifies only the verification rule, operating entirely at inference time without parameter updates. Extensive experiments demonstrate that this lightweight principle unlocks the full potential of high-quality drafters. Contributions. • We identify low-margin rejections as a fundamental inefficiency in strict verification, showing that adaptive tie-breaking can significantly reduce rollback costs. • We propose a training-free verification algorithm MARS that leverages the logit margin to dynamically adapt verification rigor, avoiding the overhead of auxiliary judges. • We demonstrate the scalability and robustness of MARS across a wide spectrum of model sizes (from 8B to 235B ) and diverse task settings. Extensive experiments confirm that our method consistently outperforms state-of-the-art baselines across chat, coding, and reasoning benchmarks, delivering significant acceleration with near-lossless accuracy preservation. 2 Preliminaries Figure 2: Overview of the MARS Verification Workflow. We illustrate the adaptive decision mechanism with a threshold \theta=0.9 . (Left) Low-Margin Regime: The target model exhibits weak preference between the top candidates “nice” and “lovely” (Logit Ratio r=0.911>\theta ). Identifying this local indifference, MARS accepts the draft token “lovely” (Top-2) as a valid tie-breaker, avoiding unnecessary rollback. (Right) High-Margin Regime: The target model decisively prefers “walk” over the runner-up “work” (Logit Ratio r=0.728<\theta ). MARS detects this stability and reverts to strict verification, rejecting the draft to preserve generation fidelity. 2.1 Autoregressive Generation Autoregressive Large Language Models (LLMs) generate text token-by-token. Given a prefix sequence x_{<t} , the model computes the probability distribution for the next token x_{t} as P(x_{t}\mid x_{<t}) . This serial dependency prevents parallelization across time steps, making inference memory-bandwidth bound on modern hardware (e.g., GPUs) and resulting in high latency. 2.2 Speculative Decoding Speculative Decoding (SD) ( 16 ; 6 ) alleviates this bottleneck by employing a lightweight approximation model, referred to as the draft model \mathcal{M}_{s} . At each step, \mathcal{M}_{s} tentatively generates a short chain of K candidate tokens (the draft). These tokens are then verified in parallel by the target model \mathcal{M}_{t} in a single forward pass, accepting those that match the target’s distribution and discarding the rest. Let x_{<t} denote the current context. The draft model \mathcal{M}_{s} autoregressively generates a sequence of speculative tokens \hat{V}=[\hat{v}_{1},\dots,\hat{v}_{K}] . Subsequently, the target model \mathcal{M}_{t} performs a parallel forward pass to compute the conditional distributions P(\cdot\mid x_{<t},\hat{v}_{<i}) for all positions i\in\{1,\dots,K\} . The core challenge lies in efficiently verifying these candidates while ensuring the final output distribution remains faithful to the target model \mathcal{M}_{t} . 2.3 Greedy Speculative Verification To ensure lossless acceleration in greedy decoding—where the goal is to reproduce the target model’s most likely predictions—standard Speculative Decoding employs an Exact Match verification scheme. Unlike sampling-based approaches that use probabilistic rejection, this mechanism enforces a strict equality constraint. For a drafted token \hat{v}_{i} at position i , the verification criterion checks if the draft matches the target model’s top prediction: \hat{v}_{i}=\arg\max_{v\in\mathcal{V}}P_{\mathcal{M}_{t}}(v\mid x_{<t},\hat{v}_{<i}). (1) If this condition holds, the token is accepted, and verification proceeds to the next token i+1 . Upon the first mismatch, the draft chain is truncated at index i . The rejected draft token \hat{v}_{i} is discarded, and the target model’s actual top prediction v^{(1)} is emitted as the corrected token. This strict verification guarantees that the final output sequence is mathematically identical to that produced by running \mathcal{M}_{t} in isolation. This formulation forms the basis of standard