Rethinking the Role of Prompting Strategies in LLM Test-Time Scaling: A Perspective of Probability Theory
Yexiang Liu, Zekun Li, Zhi Fang, Nan Xu, Ran He, Tieniu Tan · 2025
Read Rethinking the Role of Prompting Strategies in LLM Test-Time Scaling: A Perspective of Probability Theory by Yexiang Liu, Zekun Li, Zhi Fang, Nan Xu, Ran He, Tieniu Tan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-10981.txt.
Introduction
Yexiang Liu and co-authors rethink how prompting strategies behave as language models get more test-time compute, in a 2025 paper. Much work scales compute at test time, but how prompting fares under that scaling was little studied. They focus on majority voting, a standard and realistic setting.
Experiments span six models, eight prompting strategies, and six benchmarks. As sampling and compute grow, complex strategies that start ahead gradually fall behind simple chain-of-thought. The authors explain the pattern with theory and proofs. They also propose a probability-based method to predict scaling performance and pick the best strategy without heavy testing.
Catalog record
| CREATOR | Yexiang Liu, Zekun Li, Zhi Fang, Nan Xu, Ran He, Tieniu Tan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-10981
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Rethinking the Role of Prompting Strategies in LLM Test-Time Scaling: A Perspective of Probability Theory Yexiang Liu, Zekun Li, Zhi Fang, Nan Xu, Ran He, Tieniu Tan Abstract Recently, scaling test-time compute on Large Language Models (LLM) has garnered wide attention. However, there has been limited investigation of how various reasoning prompting strategies perform as scaling. In this paper, we focus on a standard and realistic scaling setting: majority voting. We systematically conduct experiments on 6 LLMs × \times 8 prompting strategies × \times 6 benchmarks. Experiment results consistently show that as the sampling time and computational overhead increase, complicated prompting strategies with superior initial performance gradually fall behind simple Chain-of-Thought. We analyze this phenomenon and provide theoretical proofs. Additionally, we propose a probabilistic method to efficiently predict scaling performance and identify the best prompting strategy under large sampling times, eliminating the need for resource-intensive inference processes in practical applications. Furthermore, we introduce two ways derived from our theoretical analysis to significantly improve the scaling performance. We hope that our research can promote to re-examine the role of complicated prompting, unleash the potential of simple prompting strategies, and provide new insights for enhancing test-time scaling performance. Code is available at https://github.com/MraDonkey/rethinking_prompting . 1 Introduction Over the past few years, how to enhance the reasoning abilities of large language models (LLMs) has been a topic of widespread interest ( Dubey et al., 2024 ; Anil et al., 2023 ; Touvron et al., 2023 ; Open AI, 2024a ; Team et al., 2024 ) . Researchers have introduced various prompting strategies to improve the reasoning capacity of LLMs, such as Chain of Thought (CoT) ( Wei et al., 2022 ) and so on ( Zheng et al., 2024 ; Yasunaga et al., 2024 ; Madaan et al., 2023 ) . Recently, many studies have shown that scaling LLM test-time compute can also effectively improve reasoning ( Snell et al., 2025 ; Open AI, 2024b ; Ji et al., 2025 ; Bi et al., 2024 ) . However, how different prompting strategies behave when scaling test-time compute is less explored. In this paper, we focus on a standard and effective scaling setting: majority voting.We comprehensively evaluate the performance of 8 mainstream prompting strategies under equivalent sampling time or computation overhead. We test 4 open-sourced and 2 closed-sourced LLMs on 6 reasoning benchmarks, finding that simple CoT consistently performs best on all LLMs across benchmarks with given budgets as scaling increases, even if it falls behind at the beginning. This indicates that current LLMs can achieve remarkable reasoning capabilities by only relying on simple CoT without other complicated prompting strategies. It also reminds us to reflect on the necessity of complicated prompting for scaling and fairly compare different strategies under the same budget. We systematically analyze this phenomenon and provide theoretical and experimental proofs. We conclude that this is caused by two reasons. One is that there are more easy questions and fewer hard questions for CoT compared to other strategies. Easy questions are more likely to get right solutions, and the error possibility decreases until 0% as scaling. In comparison, hard questions are the opposite. The other is that CoT is less likely to be affected by wrong answers. Although CoT sometimes has lower pass@1 accuracy, its probability of obtaining the correct answer is more prominent in the result distribution. In contrast, other strategies have higher disturbed peaks in the distribution of incorrect answers. These two reasons enable CoT to improve reasoning performance more rapidly and gradually dominate as scaling. What’s more, we propose a method with the complexity O(1) according to probability theory to quickly predict the scaling performance, which can serve as the test-time scaling law for majority voting. Experiments show that our method can accurately estimate the scaling performance and select the best strategy with arbitrary sampling time. Furthermore, we explore two ways to significantly improve scaling performance with our theories. (1) Adaptively scaling according to the question difficulty. (2) Dynamically selecting the optimal prompting strategy. Extensive experiments verify their general effectiveness and superiority, e.g. , improving Majority@10 accuracy from 86.0% to 97.4% and 15.2% to 61.0% for LLaMA-3-8B-Instruct Dubey et al. (2024) on GSM8K ( Cobbe et al., 2021 ) and MATH-500 Hendrycks et al. (2021b) by combining (1) and (2), respectively. Our contributions can be summarized as follows: • We comprehensively study the test-time scaling performance on 6 LLMs \times 8 prompting strategies \times 6 benchmarks. (Section 2 ) • We find that CoT consistently performs best under the equivalent sampling time and computation overhead. (Section 3 ) • We analyze this phenomenon and provide theoretical and experimental proofs. (Section 4 ) • We propose a method to quickly predict the scaling performance and the best strategy under given sampling times. (Section 5 ) • Based on the above analysis, we introduce two ways to significantly improve the scaling performance. (Section 6 ) 2 Scaling System Designs We focus on a straight and effective setting of test-time scaling, majority voting, i.e. , Self-Consistency ( Wang et al., 2023b ) , which selects the most consistent answer among several samples. Our goal is to study what prompting strategy performs best under the equivalent scaling overhead, particularly when largely increasing the scaling extent. 2.1 Models We conduct experiments on 4 open-sourced LLMs including Qwen2.5-7B-Instruct ( Yang et al., 2024a ) , LLaMA-3-8B-Instruct ( Dubey et al., 2024 ) , GLM-4-9B-Chat ( GLM et al., 2024 ) and Phi-3.5-mini-Instruct, and 2 closed-sourced LLMs including Gemini-1.5-Flash ( Team et al., 2024 ) and GPT-4o-mini ( Open AI, 2024a ) . 2.2 Prompting Strategies We mainly focus on generalizable reasoning prompting strategies, excluding those individually designed for specific tasks or involving fine-tuning, training auxiliary models, or incorporating other models, tools, or human assistance. In this setting, the model’s performance is only related to the input prompt, thus making it fairly compare the scaling performance of those prompting strategies. The prompting strategies we test are listed as follows. Direct Prompting (DiP): Directly input the question to the model, without any additional instruction or restrictions to the output. Chain-of-Thought (CoT) ( Wei et al., 2022 ; Kojima et al., 2022 ) : Use the prompt “Let’s think step by step.” to solve the problem step by step. Least-to-Most (L2M) ( Zhou et al., 2023 ) : Break down the question into progressive sub-questions. Answer the sub-questions and get the final result according to them and their answers. Tree-of-Thoughts (ToT) ( Yao et al., 2023 ) : Explore multiple reasoning paths to get several solutions, then analyze each solution and decide which one is the most promising. Self-Refine (S-RF) ( Madaan et al., 2023 ) : First, answer the question to get an initial answer. Next, evaluate the previous answer and get feedback. Finally, refine the previous answer according to feedback. This will last for several rounds. Step-Back Prompting (SBP) ( Zheng et al., 2024 ) : First, extract the discipline concepts and principles involved in solving the problem. Then, solve the problem step by step by following the principles. Analogous Prompting (AnP) ( Yasunaga et al., 2024 ) : Recall relevant problems as examples. Afterward, solve the analogous problems and proceed to solve the initial