First Finish Search: Efficient Test-Time Scaling in Large Language Models
Aradhye Agarwal, Ayan Sengupta, Tanmoy Chakraborty · 2025
Read First Finish Search: Efficient Test-Time Scaling in Large Language Models by Aradhye Agarwal, Ayan Sengupta, Tanmoy Chakraborty (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18149.txt.
Introduction
Giving a language model more time to think usually means generating many long attempts, which costs tokens and patience. Aradhye Agarwal, Ayan Sengupta, and Tanmoy Chakraborty notice a pattern in 2025: for reasoning tasks, short traces are far more likely to be right than long ones. Their method, First Finish Search, starts several independent attempts at once. It simply returns the first one that finishes. No extra training is needed.
On hard math contest sets, the method lifted one model's accuracy to 82 percent, a gain of 15 points, close to a leading commercial system. The theory explains when stopping early helps and when it can mislead. The authors argue that simple test-time tricks still hold untapped power.
Catalog record
| CREATOR | Aradhye Agarwal, Ayan Sengupta, Tanmoy Chakraborty |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-18149
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
First Finish Search: Efficient Test-Time Scaling in Large Language Models Aradhye Agarwal, Ayan Sengupta, Tanmoy Chakraborty Abstract Test-time scaling (TTS), which involves dynamic allocation of compute during inference, offers a promising way to improve reasoning in large language models. While existing TTS methods work well, they often rely on long decoding paths or require a large number of samples to be generated, increasing the token usage and inference latency. We observe the surprising fact that for reasoning tasks, shorter traces are much more likely to be correct than longer ones . Motivated by this, we introduce First Finish Search (FFS) , a training-free parallel decoding strategy that launches n n independent samples and returns as soon as any one completes. We evaluate FFS alongside simple decoding, beam search, majority voting, and budget forcing on four reasoning models (DeepSeek-R1, R1-Distill-Qwen-32B, QwQ-32B and Phi-4-Reasoning-Plus) and across four datasets (AIME24, AIME25-I, AIME25-II and GPQA Diamond). With DeepSeek-R1, FFS achieves 82.23% accuracy on the AIME datasets, a 15% improvement over DeepSeek-R1’s standalone accuracy, nearly matching OpenAI’s o4-mini performance. Our theoretical analysis explains why stopping at the shortest trace is likely to yield a correct answer and identifies the conditions under which early stopping may be suboptimal. The elegance and simplicity of FFS demonstrate that straightforward TTS strategies can perform remarkably well, revealing the untapped potential of simple approaches at inference time. 1 Introduction Large language models (LLMs) [ 33 , 8 , 20 ] have achieved impressive performance on complex reasoning tasks [ 10 ] , predominantly due to larger model sizes, greater pretraining compute, and vast training corpora [ 15 , 3 ] . Further improvements, however, are proving increasingly difficult to achieve as the benefits of increasing model size diminish and the cost of further pretraining rises [ 35 ] . Recent research has, instead, started exploring the possibility of test-time intervention [ 29 ] for enhancing LLM performance. This approach, known as test-time scaling (TTS) [ 24 ] , dynamically allocates compute during decoding in order to improve accuracy and consistency without any additional training. TTS is especially effective on complex and reasoning-heavy benchmarks, where greater compute often translates into more reliable and precise outputs. OpenAI’s o1 [ 20 ] and DeepSeek-R1 [ 8 ] are notable examples of models which utilize this approach. According to the taxonomy proposed by Zhang et al. 2025 , TTS strategies fall into three main categories: parallel scaling [ 24 , 4 ] , sequential scaling [ 13 , 14 ] , and hybrid approaches [ 31 , 27 ] . Parallel scaling methods, such as beam search (BS), diverse beam search (DVBS) [ 26 ] , and majority voting (MV), involve generating multiple candidate outputs concurrently and selecting the best response using heuristics or scoring functions. While these approaches excel in throughput and leverage computational parallelism effectively, they suffer from high token usage and require sophisticated mechanisms for evaluating and comparing candidate responses. For instance, MV assumes that the generated outputs can be easily compared using string equality or semantic similarity, which is often not the case in open-ended tasks. Sequential scaling techniques, on the other hand, extend the reasoning path deliberately with methods like budget forcing (BF) [ 19 ] introducing artificial delay tokens such as “Wait” to encourage deeper reasoning, while others like thought switching penalty (TIP) [ 28 ] discouraging premature shifts in reasoning direction. While these methods are powerful in improving accuracy, especially in tasks where extended deliberation leads to better answers, they are computationally expensive and introduce latency, making them less suitable for real-time or API-constrained deployments. Hybrid strategies, another class of TTS methods, combine elements of both parallel and sequential scaling to adapt to task difficulty and budget constraints. While combining diverse strategies does allow test-time compute to be allocated more judiciously, hybrid methods, which mix sequential and parallel strategies, still suffer from the same weaknesses to a greater or lesser extent. Figure 1: Distribution of trace lengths for correct and incorrect responses generated by QwQ [ 32 ] and R1-Distill-Qwen models [ 8 ] . The distributions show that shorter generation length and correctness are correlated, providing empirical justification for FFS. A Welch statistic of 16.56 with a p-value of < 0.001 indicates the statistical significance of this behavior. To address these challenges, we introduce F irst F inish S earch (FFS) , a training-free test-time scaling method that launches n samples in parallel and selects the output trace that completes first as the final answer. FFS is motivated by a surprising empirical finding (Figure 1 ) – for reasoning tasks, shorter traces are significantly more likely to be correct . 1 1 1 Note that Figure 1 actually implies the reverse , i.e., correct traces are more likely to be short. We analyze in Appendix 8.1 as to why this also means that shorter traces are more likely to be correct. FFS, unlike MV, needs no equality check between generated outputs, and unlike BF or TIP, it requires no special tokens or branching during decoding. It also uses far fewer tokens, and requires lesser sequential computation. We validate FFS with experiments on four benchmarks (GPQA Diamond, AIME24 and AIME25-I and AIME25-II) using four models: DeepSeek-R1, its distilled variant R1-Distill-Qwen-32B, QwQ-32B and Phi-4-Reasoning-Plus. Across all settings, FFS matches or exceeds the accuracy of strong baselines such as MV and BF while reducing token usage by up to 45%. On AIME24 and AIME25 (AIME25-I and AIME25-II), with DeepSeek-R1, FFS attains an accuracy of 82.23% on average, nearly reaching OpenAI’s o4-mini (83.7%) accuracy. 2 2 2 o4-mini accuracy is obtained from https://www.vals.ai/benchmarks/aime-2025-05-09 Our experiments show that FFS gains more from increased model capacity than any competing method, enabling it to outperform every baseline on DeepSeek-R1. Our theoretical results show that FFS’s expected sequential cost decreases with increasing sample count, demonstrating its compute efficiency for inference-time scaling. These findings establish FFS as a scalable, efficient and effective alternative for boosting LLM reasoning at test-time. 3 3 3 The source code is available at https://github.com/Aradhye2002/reasoning_exps 2 Related Work Table 1: Comparison of test-time scaling strategies across five criteria. “Training-free” indicates no additional model training is required; “API-friendly” means the method can be implemented using standard inference APIs without logit edits or repeated calls; “scalable” denotes the ability to improve performance by allocating more compute; “T-parallelizable” means total compute cost decreases when decoding is parallelized across more workers; and “S-parallelizable” means sequential latency decreases as the number of parallel samples increases. DVBS: diverse beam search, BS: beam search, TIP: thought switching penalty, BF: budget forcing, SD: simple decoding, MV: majority voting, LFS: last finish search, FFS: first finish search. DVBS BS TIP BF SD MV LFS L1 FFS Training-free ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✓ API-friendly ✗ ✗ ✗ ✗ ✓ ✓ ✓ ✓ ✓ Scalable ✓ ✓ ✗ ✓ ✗ ✓ ✓ ✗ ✓ T-parallelizable ✓ ✓ ✗ ✗ ✗ ✓ ✓ ✗ ✓ S-parallelizable ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ TTS techniques aim to improve LLM reasoning by dynamically adjusting the compute budget [ 24 ] during inference, without needing expensive model fine-tuning. Existing TTS methods can be categorized into two broad categories: 1.