Accelerated Test-Time Scaling with Model-Free Speculative Sampling

Woomin Song, Saket Dingliwal, Sai Muralidhar Jayanthi, Bhavana Ganesh, Jinwoo Shin, Aram Galstyan, Sravan Babu Bodapati · 2025

Read Accelerated Test-Time Scaling with Model-Free Speculative Sampling by Woomin Song, Saket Dingliwal, Sai Muralidhar Jayanthi, Bhavana Ganesh, Jinwoo Shin, Aram Galstyan, Sravan Babu Bodapati (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-04708.txt.

Introduction

Woomin Song and co-authors speed up in 2025 the trick of letting a language model draft its own next words, no helper model required. Reasoning models get better by thinking longer, but every extra token costs time. Speculative decoding usually drafts tokens with a small side model; their method, STAND, drafts from the reasoning itself.

The insight is that reasoning paths repeat themselves: similar patterns of thought recur, so recent n-grams of tokens forecast what comes next surprisingly well. A compact module stores the needed probability information, and a tuned sampling scheme accepts drafts that fit. Across math, science, and code benchmarks the method cut latency by roughly two thirds against plain decoding, with accuracy intact. It also beat rival speculative methods across single runs, batches, and tree search. Model free drafting travels anywhere: any reasoning model can adopt it without training or a companion network.

Catalog record

CREATORWoomin Song, Saket Dingliwal, Sai Muralidhar Jayanthi, Bhavana Ganesh, Jinwoo Shin, Aram Galstyan, Sravan Babu Bodapati
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-04708

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Accelerated Test-Time Scaling with Model-Free Speculative Sampling Woomin Song, Saket Dingliwal, Sai Muralidhar Jayanthi, Bhavana Ganesh, Jinwoo Shin, Aram Galstyan, Sravan Babu Bodapati Abstract Language models have demonstrated remarkable capabilities in reasoning tasks through test-time scaling techniques like best-of-N sampling and tree search. However, these approaches often demand substantial computational resources, creating a critical trade-off between performance and efficiency. We introduce STAND (STochastic Adaptive N-gram Drafting), a novel model-free speculative decoding approach that leverages the inherent redundancy in reasoning trajectories to achieve significant acceleration without compromising accuracy. Our analysis reveals that reasoning paths frequently reuse similar reasoning patterns, enabling efficient model-free token prediction without requiring separate draft models. By introducing stochastic drafting and preserving probabilistic information through a memory-efficient logit-based N-gram module, combined with optimized Gumbel-Top-K sampling and data-driven tree construction, STAND significantly improves token acceptance rates. Extensive evaluations across multiple models and reasoning tasks (AIME-2024, GPQA-Diamond, and LiveCodeBench) demonstrate that STAND reduces inference latency by 60-65% compared to standard autoregressive decoding while maintaining accuracy. Furthermore, STAND outperforms state-of-the-art speculative decoding methods by 14-28% in throughput and shows strong performance even in single-trajectory scenarios, reducing inference latency by 48-58%. As a model-free approach, STAND can be applied to any existing language model without additional training, being a powerful plug-and-play solution for accelerating language model reasoning. 0 0 footnotetext: † Work done during an internship at Amazon. 1 Introduction Test-time scaling has emerged as a prominent paradigm for enhancing the performance of language models by allocating additional computational resources during inference ( Snell et al. 2024 ) . This includes generating long sequences of thoughts though Large Reasoning Models (LRMs) Muennighoff et al. 2025 , multi-sampling approaches like best-of-N sampling and majority voting that generate multiple independent outputs to select the most promising one ( Wang et al. 2022 ) , as well as iterative methods like tree search and sequential refinement that allow models to progressively improve their reasoning process ( Uesato et al. 2022 ) . While these methods demonstrate the potential for significant accuracy improvements through increased computation, they often demand substantial computational resources due to the large number of tokens that need to be generated. Recent research has focused on reducing the high computational costs of test-time scaling and reasoning approaches ( Sui et al. 2025 ) . Some work has explored training with length-based rewards to generate more concise outputs ( Aggarwal and Welleck 2025 ; Qu et al. 2025 ) , while other approaches use combinations of small and large models to distribute the workload efficiently ( Liao et al. 2025 ; Yang et al. 2025 ) . However, these efficiency-focused methods typically face a fundamental trade-off. While they reduce computational costs, they tend to sacrifice some accuracy compared to more exhaustive approaches, as using fewer samples or cutting short the exploration process often leads to lower performance. (a) AIME-2024 (b) GPQA-Diamond (c) LiveCodeBench Figure 1: Scaling curve with speculative decoding. We report the scaling curve describing how the task performance improves with respect to the total decoding time. Keeping simple auto-regressive decoding total time as 1, we also report the scaling curves for different model-free SD methods. We report the reward-weighted majority voting accuracy for AIME-2024 and GPQA-Diamond, and pass@k for LiveCodeBench, where k is the total number of generated sequences generated at a given point. All measurements are made on a single A100 GPU with DeepSeek-R1-Distill-Qwen-7B. This raises a crucial question: How can we improve the efficiency of test-time scaling and reasoning methods without compromising their accuracy? To address this challenge, we turn our attention to speculative decoding (SD), which offers a promising solution for lossless acceleration of language model inference. Speculative decoding accelerates language model inference by using a smaller "draft" model to predict tokens, which are then verified by the larger target model ( Leviathan et al. 2023 ) . With appropriate verification strategies ( Chen et al. 2023a ) , SD can speed up the auto-regressive decoding process of large language models while preserving their output distribution. A key observation in LRMs is the significant repetition of token sequences across different reasoning paths. When models are performing chain-of-thought reasoning ( Snell et al. 2024 ) or exploring multiple solutions ( Wang et al. 2022 ; Xie et al. 2024 ) , they frequently reuse similar expressions, logical deductions, and reasoning patterns. This redundancy presents an opportunity for model-free speculative decoding ( Saxena 2023 ; Ou et al. 2024 ) . Unlike model-based approaches that rely on neural networks as drafters ( Li et al. 2024c ; Cai et al. 2024 ) , model-free methods can leverage patterns from previous generations to construct drafts. This makes them particularly well-suited for exploiting cross-trajectory information. Our experiments confirm this approach’s effectiveness, demonstrating improved efficiency as the number of reasoning trajectories increases. To fully leverage the power of model-free speculative decoding for reasoning tasks, we propose STAND (STochastic Adaptive N-gram Drafting) . Our approach is motivated by two key observations: First, existing model-free approaches have primarily focused on greedy decoding, leaving the potential benefits of sampling largely unexplored. Second, our experimental analysis demonstrates that stochastic drafting (i.e. sampling draft tokens from the draft probability distribution) significantly improves token acceptance rates. Building on these insights, STAND introduces three key innovations: (1) a memory-efficient logit-based N-gram module that preserves probabilistic information for better stochastic drafting, (2) an optimized sampling strategy using Gumbel-Top-K for efficient token selection, and (3) a data-driven approach to draft tree construction that balances efficiency with effectiveness. Combined, these techniques significantly enhance the speculative decoding performance in the context of test-time scaling, where sampling and diverse trajectory exploration are crucial. Our extensive evaluations demonstrate STAND’s effectiveness in various reasoning tasks (math, science and coding) and different model scales. As highlighted in Figure 1 , our results show that STAND’s benefits become more pronounced as the number of reasoning trajectories increases. When using best-of-16 sampling to achieve optimal accuracy, STAND reduces inference latency by 60-65% compared to standard autoregressive decoding while maintaining the same performance level. Moreover, STAND outperforms state-of-the-art speculative decoding methods by 14-28% in throughput, establishing an efficient drafting strategy for reasoning tasks. Furthermore, we observe that STAND also shows the best throughput in single-trajectory evaluations, reducing the inference latency by 48-58% compared to standard autoregressive decoding, although it was primarily designed to leverage information across multiple reasoning trajectories. As a model-free speculative decoding approach, STAND accomplishes all these achievements without requiring any additional drafter model, or fine-tuning the target model, being able to be used in plug-and-play manner to any existing LRMs. 2 Related Works Test-time scaling and efficiency. Test-Time Scaling (TTS) has emerged as a prominent strategy to enhance problem-solving capabilities during inference without model retraining Snell et al. 2024 ; Muennighoff et al. 2025 ; Wang et al. 2022 ; Uesato et al. 2022 ; Xie et al. 2024 . Generating long chain-of-thoughts or sampling multiple sequences have consistently showcased higher accuracy in many complex tasks like math, science and coding Wei et al. 2022 ; Cobbe et al. 2021 ; Chen et al. 2023b . However, the computational cost of TTS remains a critical bottleneck for their practical use. Recent work has explored optimizing inference using adaptive thinking

Read the complete text