THINK-Bench: Evaluating Thinking Efficiency and Chain-of-Thought Quality of Large Reasoning Models

Zhiyuan Li, Yi Chang, Yuan Wu · 2025

Read THINK-Bench: Evaluating Thinking Efficiency and Chain-of-Thought Quality of Large Reasoning Models by Zhiyuan Li, Yi Chang, Yuan Wu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22113.txt.

Introduction

Zhiyuan Li, Yi Chang, and Yuan Wu built a benchmark in 2025 to catch large reasoning models wasting thought. Reasoning models often outperform ordinary language models on hard problems. But they suffer from overthinking, spinning out long chains of redundant tokens that add little accuracy, especially on easy questions. That waste costs real computing resources at scale.

Their benchmark, THINK-Bench, evaluates thinking efficiency directly rather than accuracy alone. They propose new efficiency metrics and score many reasoning models across the reasoning process, outcome quality, and chain-of-thought characteristics. The analysis shows how differently models spend their tokens for similar results. For builders of reasoning systems, the message is practical: a model that thinks twice as long for the same answer is not twice as smart. Measuring thrift is a first step toward fixing it.

Catalog record

CREATORZhiyuan Li, Yi Chang, Yuan Wu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-22113

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

THINK-Bench: Evaluating Thinking Efficiency and Chain-of-Thought Quality of Large Reasoning Models Zhiyuan Li, Yi Chang, Yuan Wu Abstract Large reasoning models (LRMs) have achieved impressive performance in complex tasks, often outperforming conventional large language models (LLMs). However, the prevalent issue of overthinking severely limits their computational efficiency. Overthinking occurs when models generate excessive and redundant tokens that contribute little to accurate outcomes, especially in simple tasks, resulting in a significant waste of computational resources. To systematically investigate this issue, we introduce Think-Bench, a benchmark designed to evaluate the reasoning efficiency of LRMs. We also propose novel efficiency metrics and conduct a comprehensive evaluation of various LRMs across multiple dimensions, including the reasoning process, outcome quality, and chain-of-thought (CoT) characteristics. Our analysis reveals that most LRMs exhibit overthinking in handling easy questions, generating unnecessarily lengthy reasoning chains. While many LRMs demonstrate high CoT quality, several suffer from low efficiency. We hope that Think-Bench can serve as a robust foundation for advancing research into LRMs. Figure 1: The performance of various LRMs on Think-Bench. The results suggest that these prominent LRMs face a challenge of overthinking. 1 Introduction In recent years, with the rapid advancement of artificial intelligence, LLMs have achieved remarkable success in the field of natural language processing (NLP), particularly excelling in tasks such as text generation and question answering ( Grattafiori et al., 2024 ; Guo et al., 2025 ; Yang et al., 2024 ) . However, the reasoning capabilities of these models in solving multidisciplinary problems still face significant challenges, such as insufficient integration of cross-disciplinary knowledge and weak logical chain reasoning ability ( Wang et al., 2024 ; Chen et al., 2025 ) . To gain a deeper understanding and effectively enhance the efficiency and accuracy of the reasoning capabilities of LLMs, constructing high-quality multidisciplinary datasets and conducting systematic evaluations has become critically important ( Chang et al., 2024 ; Xia et al., 2024 ) . Figure 2: Overview of Think-Bench. Our benchmark contains a comprehensive efficiency evaluation framework with curated datasets across three categories. Recent research have introduced several high-quality multidisciplinary benchmarks, such as MMLU ( Hendrycks et al., 2020a ) and GPQA ( Rein et al., 2024 ) , to evaluate model performance across diverse knowledge domains. However, these datasets predominantly rely on the correctness of the final answer as the sole evaluation metric. Such outcome-oriented evaluation protocols are not directly applicable to the assessment of large reasoning models (LRMs), as they neglect the accuracy and logical coherence of intermediate reasoning steps during answer generation ( Jiang et al., 2025 ; Zheng et al., 2024 ) . In reality, for complex tasks spanning logical reasoning, mathematical problem-solving, and multi-step decision-making, the quality of intermediate CoT processes not only determines final-answer accuracy but also serves as a critical indicator of LRMs’ intrinsic reasoning capabilities and operational reliability. Existing evaluation paradigms that ignore CoT quality risk masking systemic deficiencies, such as erroneous premises, logical fallacies, or context misinterpretations, which may propagate through the reasoning chain. Consequently, establishing a holistic evaluation framework that quantifies both intermediate-step correctness and final-output accuracy emerges as a pivotal yet understudied research direction. Recently, several studies have begun to explore the evaluation of the correctness of CoTs. Notable works such as MME-CoT ( Jiang et al., 2025 ) establish a comprehensive benchmark that systematically examines CoTs across three fundamental dimensions: reasoning quality, robustness, and efficiency. Meanwhile, MiCEval ( Zhou et al., 2024 ) conducts a fine-grained evaluation of Multimodal Chains of Thought (MCoT), focusing on aspects such as the accuracy of image descriptions and the correctness, relevance, and informativeness of reasoning steps. Collectively, these benchmarks represent a significant shift from solely outcome-oriented evaluations to more process-aware assessments that effectively capture the internal reasoning dynamics of large models. Despite these advances, existing benchmarks remain limited by their lack of a systematic framework for evaluating both the efficiency and reliability of reasoning processes in LRMs. This challenge is particularly prominent in multidisciplinary problem-solving scenarios ( Wang et al., 2025 ) . Prior studies have demonstrated that reasoning models structurally similar to OpenAI-o1 often allocate computational resources inefficiently when handling relatively simple problems, while achieving only negligible performance improvements in final outcomes ( Chen et al., 2024 ; Guo et al., 2025 ) . These findings suggest that LRMs may suffer from suboptimal resource distribution during inference, consequently constraining their overall performance optimization. To bridge this critical research gap, we introduce Think-Bench , a multidisciplinary dataset specifically designed for comprehensively evaluating the reasoning efficiency and accuracy of LRMs. Our benchmark features meticulously annotated key reasoning steps for each problem instance, enabling granular analysis of model-generated reasoning processes. Additionally, we propose an evaluation protocol that can measure the efficiency and rationality of the reasoning process by analysing the specific behaviours exhibited by LRMs during their thinking process. 2 Dataset Curation 2.1 Data Overview Figure 3: Category and Subcategory Distribution of Think-Bench. Statistic Number Total questions 1375 -Choice questions 929 -Free-form questions 446 -Math questions 293 -Physics questions 590 -Chemistry questions 492 Total key step annotation 13311 -Average inference step 9.68 Maximum question length 1893 Maximum answer length 372 Average question length 422.42 Average answer length 7.59 Table 1: Key Statistics of Think-Bench. As shown in Figure 2 , Think-Bench is a dataset specifically designed to evaluate the thinking efficiency and the quality of CoTs of LRMs in complex reasoning tasks. This dataset comprises 1,375 carefully selected and organized data samples, covering three core subjects: mathematics, physics, and chemistry. Within each subject, the number of simple questions is approximately equal to the number of difficult questions. The data sources are diverse, drawing from multiple academic datasets, including MMLU ( Hendrycks et al., 2020a ) , Math500 ( Hendrycks et al., 2020b ) , AGIEval ( Zhong et al., 2023 ) , AIME ( Veeraboina, 2023 ) , GPQA ( Rein et al., 2024 ) , SciKnowEval ( Feng et al., 2024 ) , and UGPhysics ( Xu et al., 2025b ) . 2.2 Data Collection During the construction of Think-Bench, we aggregated questions from multiple authoritative, publicly available datasets. The distribution of the Think-Bench across different disciplines is shown in Table 1 . To ensure fairness, all samples were selected randomly. After the selection process, we conducted a systematic data cleaning and verification procedure to remove duplicate and invalid entries. The final dataset consists of 1,375 data points, after which we carried out the data annotation work. This benchmark covers the core disciplines of mathematics, physics, and chemistry, which inherently require structured and multi-step reasoning. Therefore, it provides a robust and rigorous foundation for evaluating the performance of reasoning models. Detailed statistics regarding the data composition can be found in Figure 3 and Table 1 . 2.3 Data Annotation and Review To systematically evaluate the CoT reasoning capabilities of LRMs on reasoning tasks, we implement a fine-grained annotation framework for key reasoning steps across all questions. Key steps are defined as essential logical components that must be completed and cannot be omitted in the process of achieving the correct answer. The annotation process is implemented in the following steps: First, we leverage Claude 3.7 Sonnet to generate comprehensive reasoning chains for each question (see Prompt D in Appendix

Read the complete text