Think Before You Accept: Semantic Reflective Verification for Faster Speculative Decoding
Yixuan Wang, Yijun Liu, Shiyu Ji, Yuzhuang Xu, Qingfu Zhu, Wanxiang Che · 2025
Read Think Before You Accept: Semantic Reflective Verification for Faster Speculative Decoding by Yixuan Wang, Yijun Liu, Shiyu Ji, Yuzhuang Xu, Qingfu Zhu, Wanxiang Che (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18629.txt.
Introduction
Yixuan Wang and co-authors present a 2025 arXiv paper on faster text generation through reflective verification. Large language models write one token at a time, which is slow. Speculative decoding drafts several tokens with a small model and checks them in parallel, but standard checks watch word statistics, not meaning. Their method asks the big model to reflect on draft tokens semantically, in the same pass.
Fusing the plain and reflective distributions judges both consistency and correctness. Across benchmarks and model sizes, acceptance runs lengthened without quality loss. The semantic check also stacks with statistical ones, adding five to fifteen percent more speed when combined.
Catalog record
| CREATOR | Yixuan Wang, Yijun Liu, Shiyu Ji, Yuzhuang Xu, Qingfu Zhu, Wanxiang Che |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-18629
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Think Before You Accept: Semantic Reflective Verification for Faster Speculative Decoding Yixuan Wang, Yijun Liu, Shiyu Ji, Yuzhuang Xu, Qingfu Zhu, Wanxiang Che Abstract Large language models (LLMs) suffer from high inference latency due to the auto-regressive decoding process. Speculative decoding accelerates inference by generating multiple draft tokens using a lightweight model and verifying them in parallel. However, existing verification methods rely heavily on distributional consistency while overlooking semantic correctness, thereby limiting the potential speedup of speculative decoding. While some methods employ additional models for relaxed verification of draft tokens, they often fail to generalize effectively to more diverse or open-domain settings. In this work, we propose Reflective Verification, a training-free and semantics-aware approach that achieves a better trade-off between correctness and efficiency. Specifically, we leverage the inherent reflective capacity of LLMs to semantically assess the correctness of draft tokens in parallel during verification. Using prompt-based probing, we obtain both the original and reflective distributions of draft tokens in a single forward pass. The fusion of these distributions enables semantic-level verification of draft tokens that incorporates both consistency and correctness. Experiments across multiple domain benchmarks and model scales demonstrate that our method significantly increases the acceptance length of draft tokens without compromising model performance. Furthermore, we find that the proposed Reflective Verification is orthogonal to existing statistical verification methods, and their combination yields additional 5 ∼ \sim 15% improvements in decoding speed. 1 Introduction Large language models (LLMs), such as ChatGPT ( Achiam et al. 2023 ) and Deepseek ( Liu et al. 2024a ) , have demonstrated remarkable performance across a wide range of domains. However, they also face numerous challenges ( Zhou et al. 2024b ) during the deployment phase. One major contributor to the high inference latency of LLMs is the auto-regressive decoding mechanism inherent in decoder-only architectures. To mitigate the memory access bottlenecks associated with token-by-token generation, speculative decoding ( Xia et al. 2024 ) has recently emerged as a promising approach for inference acceleration. This technique employs a lightweight draft model to propose multiple candidate tokens, which are then simultaneously verified by the target model. Compared to the drafting stage, the primary objective of the verification stage is to determine whether the current candidate tokens are accepted. Using exact match as the verification criterion ( Xia et al. 2023 ; Santilli et al. 2023 ) can indeed ensure the losslessness of the acceleration method. But it is often constrained in scenarios with high sampling temperatures. In order to further enhance the acceleration effectiveness of speculative decoding, several recent efforts ( Kim et al. 2023 ) have focused on developing more relaxed verification strategies that increase the acceptance length of candidate tokens while preserving output correctness. Various statistical metrics have been employed to enhance the reliability of verification strategies. Speculative sampling ( Leviathan et al. 2023 ; Chen et al. 2023 ) proposes an unbiased decoding strategy with respect to the original distribution of the target LLM, allowing flexible adjustment of the output based on the alignment between the draft and target distributions. In addition, several studies have explored the use of neural networks to decide whether candidate drafts should be accepted. Judge Decoding ( Bachmann et al. 2025 ) trains a classifier on human-annotated data to achieve longer acceptance lengths without compromising downstream task performance. Liao et al. 2025 introduces an auxiliary reward model to evaluate drafts, allowing for step-level discrimination of candidate tokens. While the above methods improve the performance of speculative decoding during verification, two key challenges remain unresolved. (1) The lack of semantic guidance. Current mainstream approaches primarily perform verification using statistical information between the draft and target distributions, which provides limited information. There is a need for a more efficient verification mechanism guided by semantic-level information, where acceptance decisions are made based on semantic correctness rather than distributional consistency. (2) Limited generalization. Although some existing methods leverage deep models for verification, they typically require additional human annotations and training procedures. Moreover, many of these methods are tailored specifically for reasoning tasks with step-level responses, and thus struggle to generalize to more general scenarios. In this paper, we propose Reflective Verification , a training-free draft verification method that operates at the semantic level to address the aforementioned challenges. Inspired by the observation in Figure 1 that self-reflection can effectively identify the semantic correctness of draft tokens, we leverage prompt-based probing to explicitly trigger reflection of the model within a single forward pass. The outcome of this reflection is then used to guide the verification of candidate draft tokens. Specifically, we exploit the unidirectional attention mechanism of LLMs by appending a reflection prompt and a copy of the draft tokens after original draft tokens during verification. This allows us to obtain the reflective judgment of target models on the current candidate tokens during the verification process. By integrating the original output representing consistency with the reflective output ensuring correctness, Reflective Verification can significantly extend the acceptance length of drafts while maintaining correctness, thereby improving the speedup of speculative decoding. In addition, the proposed method primarily calibrates the original output probabilities using reflective probabilities, which is orthogonal to existing statistical-based verification mechanisms. We conduct experiments across multiple configurations on benchmarks from various domains. The results demonstrate that the proposed method can bring orthogonal improvements to a wide range of existing verification strategies, achieving faster decoding without compromising task performance. Moreover, under low-quality draft settings, Reflective Verification helps mitigate the performance degradation of lossy verification methods and can even lead to overall performance improvements. Our main contributions are as follows: • We present a plug-and-play speculative decoding verification approach that incorporates semantic correctness by leveraging the reflective abilities of LLMs. • By fusing the original and reflective outputs, the proposed method can be adapted to nearly all existing draft models and verification strategies, demonstrating strong generalization capability. • Extensive results show that the proposed method can significantly extend draft acceptance length without degrading model performance, yielding a 5 \sim 15% orthogonal improvement in end-to-end throughput. 2 Observations In this section, we present several phenomena related to the verification of draft tokens that we have observed during the speculative decoding phase. Motivated by these observations, we further propose the Reflective Verification method. Figure 1: An illustration of draft tokens rejected by standard speculative decoding. Self-reflection enables the acceptance of semantically correct drafts that would otherwise be rejected. 2.1 Not All Rejected Drafts Are Incorrect As discussed above, an effective verification mechanism requires a careful trade-off between correctness and inference efficiency. With the continuous advancement of small language models ( Xiao et al. 2024 ) , the quality of their draft tokens increases accordingly. Relying solely on strict consistency for draft verification can significantly limit the upper bound of speedup achievable by speculative decoding. In order to investigate potential improvements in verification strategies, we conduct an analysis of the draft tokens rejected by the standard speculative decoding verification process. Figure 1 presents several examples of rejected draft tokens identified by the standard verification mechanism. We observe that some of these tokens, despite distributional inconsistencies, are semantically equivalent to the correct outputs. Accepting such tokens would not compromise the overall correctness of the response but can significantly improve