Faster and Better LLMs via Latency-Aware Test-Time Scaling

Zili Wang, Tianyu Zhang, Haoli Bai, Lu Hou, Xianzhi Yu, Wulong Liu, Shiming Xiang, Lei Zhu · 2025

Read Faster and Better LLMs via Latency-Aware Test-Time Scaling by Zili Wang, Tianyu Zhang, Haoli Bai, Lu Hou, Xianzhi Yu, Wulong Liu, Shiming Xiang, Lei Zhu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19634.txt.

Introduction

Zili Wang and co-authors present, in twenty twenty-five, a way to make large language models both faster and better at test time. Test-time scaling means spending more compute as the model answers, to raise quality. Past work chased compute gains but ignored delay. In uses where delay matters, the cheapest compute path is not the fastest. The team proposes two parallel tricks. One runs many answer branches at once. The other drafts tokens ahead and checks them, a method called speculative decoding.

By balancing both tricks, their method hit strong math scores fast. A large model reached a high score on a five hundred problem math set within a minute. A small model scored well within ten seconds.

Catalog record

CREATORZili Wang, Tianyu Zhang, Haoli Bai, Lu Hou, Xianzhi Yu, Wulong Liu, Shiming Xiang, Lei Zhu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-19634

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Faster and Better LLMs via Latency-Aware Test-Time Scaling Zili Wang, Tianyu Zhang, Haoli Bai, Lu Hou, Xianzhi Yu, Wulong Liu, Shiming Xiang, Lei Zhu Abstract Test-Time Scaling (TTS) has proven effective in improving the performance of Large Language Models (LLMs) during inference. However, existing research has overlooked the efficiency of TTS from a latency-sensitive perspective. Through a latency-aware evaluation of representative TTS methods, we demonstrate that a compute-optimal TTS does not always result in the lowest latency in scenarios where latency is critical. To address this gap and achieve latency-optimal TTS, we propose two key approaches by optimizing the concurrency configurations: (1) branch-wise parallelism, which leverages multiple concurrent inference branches, and (2) sequence-wise parallelism, enabled by speculative decoding. By integrating these two approaches and allocating computational resources properly to each, our latency-optimal TTS enables a 32B model to reach 82.3% accuracy on MATH-500 within 1 minute and a smaller 3B model to achieve 72.4% within 10 seconds. Our work emphasizes the importance of latency-aware TTS and demonstrates its ability to deliver both speed and accuracy in latency-sensitive scenarios. Figure 1: “Compute-optimal” does not necessarily translate to “latency-optimal” for test-time scaling. Left: Previous works measure test-time scaling by #token budget, indicating sequential scaling shows superior token efficiency than parallel scaling (majority voting for instance). Right: When considering latency as budget, parallel scaling can be 1.6x faster to achieve the same accuracy than sequential scaling. Experiments are performed using s1.1-32B Muennighoff et al. (2025) on MATH-500 Hendrycks et al. (2021) . The red curve corresponds to sequential scaling with sequence length varying in {1024, 2048, 4096, 8192}, via budget forcing. The blue curve represents parallel scaling by majority voting with a fixed sequence length. † † ∗ Equal Contributions. † † † Corresponding Author. Figure 2: Latency-aware test-time scaling on MATH-500 with different model types, with sequential scaling in red and parallel scaling in blue. 1 Introduction Test-Time Scaling (TTS) is an effective approach to improve the performance of Large Language Models (LLMs) at the cost of additional inference-time computations Snell et al. (2024) ; Brown et al. (2024) . TTS can be realized by two basic approaches: sequential scaling and parallel scaling. Sequential scaling requires the model to produce an extended reasoning process in a single pass Muennighoff et al. (2025) . In contrast, parallel scaling generates multiple solutions in parallel and selects the final answer, usually through majority voting Wang et al. (2022) ; Liu et al. (2025) . Hybrid approaches can be constructed on top of the two basic ones Guan et al. (2025) ; Wang et al. (2024b) . With the number of generated tokens (#tokens) as budget, many existing studies Snell et al. (2024) ; Setlur et al. (2025) ; Yang et al. (2025) ; Liu et al. (2025) ; Shi and Jin (2025) ; Zhang et al. (2024) have examined compute-optimal strategies that enhance average performance gain per token, a metric we refer to as token efficiency . However, in latency-sensitive scenarios where small batch sizes are employed, e.g . , personal computer, small-scale commercial deployment and edge device, “compute-optimal” does not necessarily translate to “latency-optimal” . This discrepancy is because the performance achieved within a limited time is determined by token efficiency as well as the throughput ( i.e . , average number of output tokens per second). As shown in Figure 1 , for s1.1-32B Muennighoff et al. (2025) model, although sequential scaling has a better token efficiency, achieving a similar performance with 9 \times fewer tokens compared to parallel scaling, it turns out that parallel scaling achieves 1.6 \times lower latency. In fact, under small batch sizes, the time of an LLM autoregressive decoding step is dominated by the memory access of the parameters. Therefore, a moderately increased number of parallel branches incurs little additional latency, allowing a much higher throughput almost for free, as shown in Figure 3 . Figure 3: System state, latency, and throughput of s1.1-32B. Left : The roofline model with s1.1-32B. Increasing computational demand shifts execution from memory-bound to compute-bound. Middle : Throughput scales linearly with batch size before saturating at peak FLOPS. Right : The latency per forward pass under varying batch sizes. We then ask how we can achieve latency-optimal test-time scaling . One lesson from the observation above is that, in addition to optimizing token efficiency, attention must be given to improving the generation concurrency to chase a throughput. There are two approaches to improving concurrency: (1) branch-wise parallelism, which increases the number of parallel branches B , and (2) sequence-wise parallelism, which generates multiple successive tokens for a sequence in a single forward pass with speculative decoding Leviathan et al. (2023) ; Chen et al. (2023) . However, current research lacks a thorough analysis of how to allocate computational power to these two resource-competing approaches and to what extent they can be improved. To bridge the gap, we examine the impact of concurrency configuration for latency-aware test-time scaling in this study. Specifically, we conduct experiments on representative datasets including MATH-500 Hendrycks et al. (2021) , AIME24 AoPS (2024) , AIME25 AoPS (2025) , and GPQA-Diamond Rein et al. (2024) across model sizes from 3B to 32B and under varied concurrency configurations. Revealed by our experiments and analyses, the latency-optimal concurrency depends on the comparative advantage of token efficiency in sequential scaling and parallel scaling. For models with higher token efficiency with parallel scaling ( e.g . , LLaMa-3.1-8B-Instruct Meta (2024) ), one should prioritize branch-wise parallelism. Otherwise ( e.g . , for QwQ-32B Team (2025) ) sequence-wise parallelism takes priority. To determine the latency-optimal configuration, we propose a simple yet effective greedy search algorithm with less searching steps. Under 1 minute latency constraint, an optimal concurrency configuration can provide up to 7.3% better performance than baselines where only a single parallelism is applied. Our contributions can be summarized as follows: 1. We introduce latency-aware test-time scaling, which considers the performance scaling under latency constraints. Unlike existing works that prominently only focus on token efficiency, we discover the necessity of taking systemic throughput into consideration for a better latency-performance trade-off. 2. We provide a unified view for parallel branches and speculative decoding from the perspective of generation concurrency. This allows us to frame latency-optimal test-time scaling as a resource allocation problem. A greedy search algorithm is proposed to search the latency-optimal configuration. 3. Through extensive experiments, we explore the optimal concurrency configuration for latency-aware test-time scaling. Our experiments reveal that for s1.1-32B, an optimal concurrency configuration can improve the accuracy by 7.3% while reducing latency by 1.7 \times , reaching 82.3% on MATH-500 in 1 minute. 2 Related Work Test-time scaling Scaling the compute at inference time has been proven to be a prominent approach to improve LLM performance. Generally, test-time scaling methods fall into two main categories: sequential and parallel scaling. For the former, sequential scaling methods, represented by OpenAI’s o1 OpenAI (September 2024) , DeepSeek-R1 Guo et al. (2025) , and Qwen QwQ Team (2025) , enforce the model to generate longer solutions with a detailed reasoning chain. Such an ability can be incentivized with supervised training Nye et al. (2021) ; Lee et al. (2025) ; Muennighoff et al. (2025) or reinforcement

Read the complete text