Revisiting the Test-Time Scaling of o1-like Models: Do they Truly Possess Test-Time Scaling Capabilities?

Zhiyuan Zeng, Qinyuan Cheng, Zhangyue Yin, Yunhua Zhou, Xipeng Qiu · 2025

Read Revisiting the Test-Time Scaling of o1-like Models: Do they Truly Possess Test-Time Scaling Capabilities? by Zhiyuan Zeng, Qinyuan Cheng, Zhangyue Yin, Yunhua Zhou, Xipeng Qiu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12215.txt.

Introduction

Reasoning models like OpenAI's o1 are said to get better the longer they think, a property called test-time scaling. Zhiyuan Zeng and co-authors put that claim under pressure in a 2025 study of similar open models. They found that longer chains of thought did not reliably raise accuracy.

For the same question, correct answers were often shorter than wrong ones. Long chains held more self-revision, and those revisions often talked the model out of right answers. The team then compared thinking longer in sequence against generating several answers in parallel. Parallel attempts covered more ground and scaled better. Their proposed fix, Shortest Majority Vote, combines parallel sampling with a preference for shorter chains, and it beat plain majority voting on their tests.

Catalog record

CREATORZhiyuan Zeng, Qinyuan Cheng, Zhangyue Yin, Yunhua Zhou, Xipeng Qiu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-12215

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Revisiting the Test-Time Scaling of o1-like Models: Do they Truly Possess Test-Time Scaling Capabilities? Zhiyuan Zeng, Qinyuan Cheng, Zhangyue Yin, Yunhua Zhou, Xipeng Qiu Abstract The advent of test-time scaling in large language models (LLMs), exemplified by OpenAI’s o1 series, has advanced reasoning capabilities by scaling computational resource allocation during inference. While successors like QwQ, Deepseek-R1 (R1) and LIMO replicate these advancements, whether these models truly possess test-time scaling capabilities remains underexplored. This study found that longer CoTs of these o1-like models do not consistently enhance accuracy; in fact, correct solutions are often shorter than incorrect ones for the same questions. Further investigation shows this phenomenon is closely related to models’ self-revision capabilities - longer CoTs contain more self-revisions, which often lead to performance degradation. We then compare sequential and parallel scaling strategies on QwQ, R1 and LIMO, finding that parallel scaling achieves better coverage and scalability. Based on these insights, we propose “Shortest Majority Vote”, a method that combines parallel scaling strategies with CoT length characteristics, significantly improving models’ test-time scalability compared to conventional majority voting approaches. Our implementation is available at https://github.com/ZhiYuanZeng/test-time-scaling-eval . 1 Introduction The release of the OpenAI o1 series models ( OpenAI, 2024a ; OpenAI, 2024b ) marked a pivotal advancement in the reasoning capabilities of Large Language Models (LLMs), introducing a novel scaling paradigm, test-time scaling, which allocates more compute resources during test time. The test-time scaling have two dimensions, sequential and parallel ( Zeng et al., 2024 ) . Sequential scaling increase test-time compute by scaling the length of Chain-of-Thought (CoT) ( Wei et al., 2022 ) , while parallel scaling parallely samples multiple solutions and pick the best one. Following o1’s success, models such as QwQ ( Team, 2024 ) , Deepseek-R1 (R1) ( DeepSeek-AI et al., 2025 ) and LIMO ( Ye et al., 2025 ) have emerged as leading open-source successors, replicating o1’s achievements and demonstrating comparable reasoning abilities. Although both QwQ, R1 and LIMO demonstrate strong reasoning capabilities and the ability to generate lengthy CoT at test time, the existence of true test-time scaling where performance consistently improves with longer CoTs remains to be verified for these models. To explore this question, we systematically investigate the relationship between CoT length and reasoning performance in QwQ, R1 and LIMO, challenging the conventional assumption that extended reasoning chains inherently lead to improved accuracy. Contrary to expectations, our analysis reveals that longer CoTs do not consistently improve accuracy of these o1-like models. Notably, we found that the average length of correct solutions is shorter than that of incorrect ones for the same questions, which is shown in Figure 1 . This counterintuitive finding underscores the need for a deeper understanding of the test-time scaling of o1-like models. To understand why the longer CoTs do not lead to the better performance, we compared the difference between long CoTs and short CoTs, finding that long CoTs contain more self-revisions (“Wait”, “Alternatively”) than the short CoTs, which is shown in Appendix E . Inspired by that, we iteratively prompted QwQ, R1 and LIMO for more self-revisions. Our observations revealed that QwQ and R1-Distill-1.5b exhibited performance degradation as the length of reflection increased. In contrast, R1-Distill-14b, R1-Distill-32b, and LIMO demonstrated initial performance improvements during early revisions, followed by oscillatory behavior in subsequent iterations. To further understand the limitations of sequential scaling, we evaluated the models’ capacity to revise incorrect answers. Our findings indicate that QwQ, R1 and LIMO all demonstrated limited ability to convert incorrect answers to correct ones during the revision process. Most revisions retained the original answers, and more concerning, both QwQ and R1-Distill-1.5b showed a higher propensity to change correct answers to incorrect ones rather than vice versa. These results reveal that self-revision ability is a key factor in the effectiveness of sequential scaling for o1-like models . Given the limited effectiveness of sequential scaling, we explored an alternative test-time scaling strategie, parallel scaling. Our comparative analysis of sequential and parallel scaling revealed that parallel scaling not only achieves the better coverage (pass@k score) but also offers superior scalability compared to sequential scaling for QwQ and R1, which demonstrates that o1-like models have limited sequential-scaling capability, but strong parallel-scaling capability. Building on these findings, we propose a novel test-time scaling method, Shortest Majority Vote , which incorporate parallel scaling approaches with our insight on sequential scaling. In particular, this method leverages the observation that shorter solutions tend to lead to better performance compared to longer ones. Shortest Majority Vote improves majority vote by prioritizing clusters that have both more solutions and shorter solution lengths. Experimental results demonstrate that Shortest Majority Vote substantially outperforms conventional Majority Vote, significantly improving the test-time scalability of both QwQ and R1 models. Our contributions are as follows: 1) We systematically investigate the test-time scaling capabilities of o1-like models QwQ, R1 and LIMO, and find that their performance can not be continuously improved through increasing CoT length. 2) We reveal that insufficient self-revision capability of o1-like models is the primary reason for their failure in sequential scaling. 3) We find that parallel scaling achieves better coverage and scalability than sequential revision for o1-like models. 4) Based on our insights into sequential and parallel scaling, we propose Shortest Majority Vote, a test-time scaling method that enhances majority voting by considering solution length, significantly outperforming traditional methods. Figure 1: The average length of correct solutions versus incorrect solutions evaluated on the same questions. For each question, solution lengths were averaged separately for correct and incorrect responses, then averaged across all questions. 2 Related Work The success of o1 has ushered in a new scaling paradigm, test-time compute scaling , which enables continuous improvements in model performance by increasing computational expenditure during inference ( OpenAI, 2024a ; OpenAI, 2024b ) . Currently, scaling test-time compute can be approached in two dimensions: parallel scaling and sequential scaling ( Snell et al., 2024 ; Zeng et al., 2024 ) . Parallel Scaling Parallel scaling typicallly samples multiple solutions in parallel and pick one according to some guidence signal like reward. Notable examples of parallel scaling include Best-of-N Search ( Cobbe et al., 2021 ; Sun et al., 2024 ; Gui et al., 2024 ; Amini et al., 2024 ; Sessa et al., 2024 ) , which is based on a reward model ( Cobbe et al., 2021 ; Lightman et al., 2024 ) , and Majority Vote ( Wang et al., 2023 ) , which exploits model uncertainty. The primary distinction between these approaches lies in the method used to select the final solution or answer after sampling multiple candidates. Both Best-of-N Search and Majority Vote are parallel scaling techniques at the solution level, while Tree-Search algorithms can be viewed as parallel scaling at the token or step level. Beam-Search ( Qiu et al., 2024 ; Yu et al., 2024 ; Xie et al., 2023 ; Kool et al., 2019 ) and MCTS ( Hao et al., 2023 ; Wan et al., 2024 ; Chen et al., 2024a ; Zhang et al., 2023 ) are classic examples of Tree-Search algorithms. All parallel scaling methods rely on guidance signals to select the optimal token, step, or solution from a set of candidates. Sequential Scaling Sequential

Read the complete text