Rethinking Inference-Time Scaling: Efficiency Limits and Linguistic Signals
Junlin Wang, Shang Zhu, Jon Saad-Falcon, Ben Athiwaratkun, Qingyang Wu, Jue Wang, Shuaiwen Leon Song, Ce Zhang, Bhuwan Dhingra, James Zou · 2025
Read Rethinking Inference-Time Scaling: Efficiency Limits and Linguistic Signals by Junlin Wang, Shang Zhu, Jon Saad-Falcon, Ben Athiwaratkun, Qingyang Wu, Jue Wang, Shuaiwen Leon Song, Ce Zhang, Bhuwan Dhingra, James Zou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-14047.txt.
Introduction
Junlin Wang and co-authors probe the limits of giving language models more thinking time, in a 2025 paper. Extra test-time compute, through repeated sampling or revision, is widely touted as a substitute for bigger models. The team ran scaling methods on hard reasoning tasks for both plain and reasoning-tuned models. A floor appeared: plain models plateaued no matter how much compute they burned.
Even with ten times the compute, general models could not match reasoning-tuned ones. Among tuned models, simple majority voting beat fancier revision schemes. Correct answers also carried a linguistic signature: shorter, with less hedging. Those surface cues predicted quality at zero extra compute. The authors suggest future agents should read their own style to judge themselves.
Catalog record
| CREATOR | Junlin Wang, Shang Zhu, Jon Saad-Falcon, Ben Athiwaratkun, Qingyang Wu, Jue Wang, Shuaiwen Leon Song, Ce Zhang, Bhuwan Dhingra, James Zou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-14047
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Rethinking Inference-Time Scaling: Efficiency Limits and Linguistic Signals Junlin Wang, Shang Zhu, Jon Saad-Falcon, Ben Athiwaratkun, Qingyang Wu, Jue Wang, Shuaiwen Leon Song, Ce Zhang, Bhuwan Dhingra, James Zou Abstract There is intense interest in investigating how inference time compute (ITC) (e.g. repeated sampling, refinements, etc) can improve large language model (LLM) capabilities. While breakthroughs like DeepSeek-R1 highlight the power of reinforcement learning for reasoning, the interaction between ITC and reasoning-optimized weights remains poorly understood. This work conducts a comprehensive analysis of inference-time scaling methods for both reasoning and non-reasoning models on challenging reasoning tasks. While prior work suggests that scaling test-time compute can optimally substitute for model parameter scaling, we identify a fundamental limit to this compute-equivalence, the “reasoning floor”, a performance plateau that non-reasoning models cannot escape, no matter how much inference compute is spent. We demonstrate that general-purpose models fail to match the accuracy of reasoning-optimized models even with an order of magnitude more inference compute, suggesting that internalizing reasoning protocols is a prerequisite for effective test-time scaling. Within reasoning models, we find that the complexity of the scaling method often yields diminishing returns; simple majority voting consistently outperforms sophisticated sequential revision and mixture-of-agents frameworks. Crucially, we identify a “Linguistic Signal of Correctness”—correct responses are significantly more concise and exhibit a lower density of “hedging” and “thinking” markers. We demonstrate that these intrinsic linguistic features can serve as zero-compute proxies for response quality, providing a pathway to more efficient, self-diagnostic reasoning agents. 1 Introduction Language models have witnessed remarkable advancements in recent years, demonstrating increasingly sophisticated capabilities across various tasks ( OpenAI, 2023 ; Dubey et al., 2024 ; Bai et al., 2023 ) . Despite these improvements, complex reasoning remains challenging, often requiring additional computational resources and specialized techniques to achieve satisfactory performance ( Wang et al., 2024b ; Yao et al., 2023 ) . This challenge has motivated the development of inference-time compute (ITC) scaling methods, which allocate additional computational resources during inference to enhance model outputs. The landscape of language model reasoning has evolved along two primary dimensions. First, approaches like Chain-of-thought ( Wei et al., 2023 ) , self-consistency ( Wang et al., 2022 ) , tree-structured sampling ( Snell et al., 2024 ) , and mixture of agents ( Wang et al., 2025 ) have emerged as effective techniques for boosting reasoning capabilities during inference without requiring model parameter changes. Second, a new class of "reasoning models", explicitly post-trained to solve highly challenging problems, has been introduced, exemplified by models like o1 ( OpenAI et al., 2024 ) , Deepseek-R1 ( DeepSeek-AI et al., 2025 ) , and QwQ ( Team, 2024 ) . While both approaches show promise, they introduce significant computational overhead. Chain-of-thought Wei et al. (2023) prompting increases token generation, tree-structured sampling requires exploring multiple solution paths ( Liu et al., 2025 ) , and mixture of agents ( Wang et al., 2025 ) demands running several specialized agent configurations simultaneously. This computational burden raises critical questions about efficiency: how can we optimize the trade-off between computational resources and reasoning performance? Which inference-time scaling methods deliver the best results for different model architectures? How do reasoning models compare to conventional models under varying computational budgets? These questions remain largely unsolved in the current literature. Prior work has primarily focused on fine-tuning strategies ( Yu et al., 2025 ; Xie et al., 2025 ) or simple inference evaluations ( Brown et al., 2024a ) , with limited attention to systematic evaluation of inference-time compute scaling across model types. Furthermore, a line of works suggest that inference-time compute can be more effective than model parameter scaling ( Snell et al., 2024 ) —these findings were largely established prior to the widespread availability of specialized reasoning architectures like DeepSeek-R1. Consequently, there is a lack of systematic evidence regarding whether this equivalence holds across different model classes. Furthermore, existing research often relies on resource-heavy process-reward models (PRMs), leaving the potential of reward-model-free (RM-free) scaling methods relatively misunderstood. Our work bridges this gap by conducting a comprehensive analysis of RM-free inference-time scaling across both reasoning-optimized and general-purpose architectures. We specifically focus on trained-verifier-free approaches to investigate the upper bounds of intrinsic model reasoning without external supervision. Our results challenge the compute-equivalence narrative, revealing a fundamental “reasoning floor”: a performance plateau that non-reasoning models cannot escape, no matter how much inference compute is spent. On the other hand, for reasoning-optimized models, we find that increasing method complexity (e.g., sequential revision) often yields diminishing returns compared to simple majority voting. Our research provides the following key contributions: • We compare inference-time scaling methods across reasoning and non-reasoning models, establishing efficiency–performance trade-off curves. We find that complex inference-time strategies yield diminishing returns, with simple majority voting consistently outperforming more elaborate methods. • We provide empirical evidence that inference-time compute is not a universal substitute for specialized training. We show that non-reasoning models are fundamentally limited by a "reasoning floor," where even an order of magnitude more compute cannot match the performance of reasoning-optimized weights. • We analyze the correlation between response linguistic features (response length, linguistic markers) and task performances, providing practical guidance for improving existing inference-time methods without increasing computational costs. 2 Related Work 2.1 Inference-Time Scaling Recent work has demonstrated that scaling compute during inference offers a promising alternative to costly model pretraining. Language models ( Touvron et al., 2023 ; Jiang et al., 2023a ; Team et al., 2024 ) continue to improve with increased data and parameters, though at escalating development costs. Inference-time scaling approaches like Brown et al. (2024b) show a log-linear relationship between problem-solving coverage and sample count across reasoning tasks, offering cost-effective alternatives to model size scaling. Inference-time architectures combine techniques such as generation ensembling, sampling, ranking, and fusion to exceed individual model performance. Works including Mixture-of-Agents ( Wang et al., 2024c ) , LLM Blender ( Jiang et al., 2023b ) , and orchestration frameworks like DSPy ( Khattab et al., 2023 ) demonstrate these approaches’ effectiveness. Even with single models, techniques like chain-of-thought ( Wei et al., 2023 ) and branch-solve-merge ( Saha et al., 2024 ) enhance reasoning capabilities. Our work extends this literature by focusing specifically on trained-verifier-free inference-time scaling methods that don’t require additional reward models. Our findings align with and extend Wang et al. (2024c) , showing that majority voting—especially when weighted by reasoning length—achieves the best balance between computational cost and performance. By constructing comprehensive efficiency–performance trade-off curves, we reveal three key insights: (1) non-reasoning models consistently underperform reasoning-specialized models, even under large inference-time budgets; (2) more sophisticated methods such as sequential revisions offer minimal gains over simple majority voting for reasoning models; and (3) response features, including length and linguistic markers, provide reliable signals for identifying high-quality outputs. 2.2 Response-Level Signals of Correctness Beyond allocating additional inference-time compute, prior work has investigated whether a model’s output contains signals of its own correctness. Early calibration studies examined whether token probabilities align with answer accuracy ( Jiang et al., 2021 ; Tian et al., 2023 ) . Related work on linguistic calibration studies whether expressions of confidence and uncertainty accurately reflect the factual reliability of generated responses ( Mielke et al., 2022 ) .